<div dir="ltr">Hi,<div><br></div><div>thanks for a write up James. I am adding a few notes/ideas inline...<br><div><br><div class="gmail_quote"><div dir="ltr">On Mon, Aug 20, 2018 at 10:48 PM James Slagle <<a href="mailto:james.slagle@gmail.com">james.slagle@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">As we start looking at how TripleO will address next generation deployment<br>
needs such as Edge, multi-site, and multi-cloud, I'd like to kick off a<br>
discussion around how TripleO can evolve and adapt to meet these new<br>
challenges.<br>
<br>
What are these challenges? I think the OpenStack Edge Whitepaper does a good<br>
job summarizing some of them:<br>
<br>
<a href="https://www.openstack.org/assets/edge/OpenStack-EdgeWhitepaper-v3-online.pdf" rel="noreferrer" target="_blank">https://www.openstack.org/assets/edge/OpenStack-EdgeWhitepaper-v3-online.pdf</a><br>
<br>
They include:<br>
<br>
- management of distributed infrastructure<br>
- massive scale (thousands instead of hundreds)<br>
- limited network connectivity<br>
- isolation of distributed sites<br>
- orchestration of federated services across multiple sites<br>
<br>
We already have a lot of ongoing work that directly or indirectly starts to<br>
address some of these challenges. That work includes things like<br>
config-download, split-controlplane, metalsmith integration, validations,<br>
all-in-one, and standalone.<br>
<br>
I laid out some initial ideas in a previous message:<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2018-July/132398.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2018-July/132398.html</a><br>
<br>
I'll be reviewing some of that here and going into a bit more detail.<br>
<br>
These are some of the high level ideas I'd like to see TripleO start to<br>
address:<br>
<br>
- More separation between planning and deploying (likely to be further defined<br>
in spec discussion). We've had these concepts for a while, but we need to do<br>
a better job of surfacing them to users as deployments grow in size and<br>
complexity.<br></blockquote><div><br></div><div>One of the focus points of ui/cli and workflows squads for Stein is getting GUI and CLI consolidated so</div><div>that both clients operate on deployment plan via Mistral workflows. We are currently working on identifying</div><div>missing CLI commands which would lead to adopting the same workflow as GUI uses. This will lead to</div><div>complete interoperability between the clients and would make a deployment plan the first-class citizen as</div><div>Ben mentioned in discussion linked above.</div><div><br></div><div>Existing plan import/export functionality makes the deployment plan easily portable and replicable as it is</div><div>possible to export the plan at any point of time and re-use it (with ability to still</div><div>apply some tweaks for each usage)</div><div><br></div><div>When Steven's work [1] introduces plan-types which adds ability to define multiple starting points for the</div><div>deployment plan.</div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/574753">https://review.openstack.org/#/c/574753</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
With config-download, we can more easily separate the phases of rendering,<br>
downloading, validating, and applying the configuration. As we increase in<br>
scale to managing many deployments, we should take advantage of what each of<br>
those phases offer.<br>
<br>
The separation also makes the deployment more portable, as we should<br>
eliminate any restrictions that force the undercloud to be the control node<br>
applying the configuration.<br>
<br>
- Management of multiple deployments from a single undercloud. This is of<br>
course already possible today, but we need better docs and polish and more<br>
testing to flush out any bugs.<br>
<br>
- Plan and template management in git.<br>
<br>
This could be an iterative step towards eliminating Swift in the undercloud.<br>
Swift seemed like a natural choice at the time because it was an existing<br>
OpenStack service. However, I think git would do a better job at tracking<br>
history and comparing changes and is much more lightweight than Swift. We've<br>
been managing the config-download directory as a git repo, and I like this<br>
direction. For now, we are just putting the whole git repo in Swift, but I<br>
wonder if it makes sense to consider eliminating Swift entirely. We need to<br>
consider the scale of managing thousands of plans for separate edge<br>
deployments.<br>
<br>
I also think this would be a step towards undercloud simplification.<br></blockquote><div><br></div><div>+1, we need to identify how much this affects the existing API and overall user experience<br></div><div>for managing deployment plans. Currentl plan management options we support are:</div><div>- create plan from default files (/usr/share/tht...)</div><div>- create/update plan from local directory</div><div>- create/update plan by providing tarball</div><div>- create/update plan from remote git repository</div><div><br></div><div>Ian has been working on similar efforts towards performance improvements [2], It</div><div>would be good to take this a step further and evaluate possibility to eliminate Swift entirely.</div><div><br></div><div>[2] <a href="https://review.openstack.org/#/c/581153/">https://review.openstack.org/#/c/581153/</a></div><div><br></div><div>-- Jirka</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
- Orchestration between plans. I think there's general agreement around scaling<br>
up the undercloud to be more effective at managing and deploying multiple<br>
plans.<br>
<br>
The plans could be different OpenStack deployments potentially sharing some<br>
resources. Or, they could be deployments of different software stacks<br>
(Kubernetes/OpenShift, Ceph, etc).<br>
<br>
We'll need to develop some common interfaces for some basic orchestration<br>
between plans. It could include dependencies, ordering, and sharing parameter<br>
data (such as passwords or connection info). There is already some ongoing<br>
discussion about some of this work:<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2018-August/133247.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2018-August/133247.html</a><br>
<br>
I would suspect this would start out as collecting specific use cases, and<br>
then figuring out the right generic interfaces.<br>
<br>
- Multiple deployments of a single plan. This could be useful for doing many<br>
deployments that are all the same. Of course some info might be different<br>
such as network IP's, hostnames, and node specific details. We could have<br>
some generic input interfaces for those sorts of things without having to<br>
create new Heat stacks, which would allow re-using the same plan/stack for<br>
multiple deployments. When scaling to hundreds/thousands of edge deployments<br>
this could be really effective at side-stepping managing hundreds/thousands<br>
of Heat stacks.<br>
<br>
We may also need further separation between a plan and it's deployment state<br>
to have this modularity.<br>
<br>
- Distributed management/application of configuration. Even though the<br>
configuration is portable (config-download), we may still want some<br>
automation around applying the deployment when not using the undercloud as a<br>
control node. I think things like ansible-runner or Ansible AWX could help<br>
here, or perhaps mistral-executor agents, or "mistral as a library". This<br>
would also make our workflows more portable.<br>
<br>
- New documentation highlighting some or all of the above features and how to<br>
take advantage of it for new use cases (thousands of edge deployments, etc).<br>
I see this as a sort of "TripleO Edge Deployment Guide" that would highlight<br>
how to take advantage of TripleO for Edge/multi-site use cases.<br>
<br>
Obviously all the ideas are a lot of work, and not something I think we'll<br>
complete in a single cycle.<br>
<br>
I'd like to pull a squad together focused on Edge/multi-site/multi-cloud and<br>
TripleO. On that note, this squad could also work together with other<br>
deployment projects that are looking at similar use cases and look to<br>
collaborate.<br>
<br>
If you're interested in working on this squad, I'd see our first tasks as<br>
being:<br>
<br>
- Brainstorming additional ideas to the above<br>
- Breaking down ideas into actionable specs/blueprints for stein (and possibly<br>
future releases).<br>
- Coming up with a consistent message around direction and vision for solving<br>
these deployment challenges.<br>
- Bringing together ongoing work that relates to these use cases together so<br>
that we're all collaborating with shared vision and purpose and we can help<br>
prioritize reviews/ci/etc.<br>
- Identifying any discussion items we need to work through in person at the<br>
upcoming Denver PTG.<br>
<br>
I'm happy to help facilitate the squad. If you have any feedback on these ideas<br>
or would like to join the squad, reply to the thread or sign up in the<br>
etherpad:<br>
<br>
<a href="https://etherpad.openstack.org/p/tripleo-edge-squad-status" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/tripleo-edge-squad-status</a><br>
<br>
I'm just referring to the squad as "Edge" for now, but we can also pick a<br>
cooler owl themed name :).<br>
<br>
-- <br>
-- James Slagle<br>
--<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div></div></div>