<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 22, 2016 at 6:01 PM, Fox, Kevin M <span dir="ltr"><<a href="mailto:Kevin.Fox@pnnl.gov" target="_blank">Kevin.Fox@pnnl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">+1 for TripleO taking a look at Kolla.<br>
<br>
Some random thoughts:<br>
<br>
I'm in the middle of deploying a new cloud and I couldn't use either TripleO or Kolla for various reasons. A few reasons for each:<br>
* TripeO - worries me for ever having to do a major upgrade of the software, or needing to do oddball configs like vxlans over ipoib.<br>
* Kolla - At the time it was still immature. No stable artefacts posted. database container recently broke, little documentation for disaster recovery. No upgrade strategy at the time.<br>
<br>
Kolla rearchitected recently to support oddball configs like we've had to do at times. They also recently gained upgrade support. I think they are on the right path. If I had to start fresh, I'd very seriously consider using it.<br>
<br>
I think Kolla can provide the missing pieces that TripleO needs. TripleO has bare metal deployment down solid. I really like the idea of using OpenStack to deploy OpenStack. Kolla is now OpenStack so should be considered.<br>
<br>
I'm also in favor of using Magnum to deploy a COE to manage Kolla. I'm much less thrilled about Mesos though. It feels heavy enough weight that it feels like your deploying an OpenStack like system just to deploy OpenStack. So, OpenStack On NotOpenStack On OpenStack. :/ I've had good luck with Kubernetes (much simpler) recently and am disappointed that it was too immature at the time Kolla originally considered it. It seems much more feasible to use now. I use net=host like features all the time which was a major sticking point before.<br></blockquote><div><br></div><div>Frankly speaking, I think that Kolla project is doing right thing keeping its component decoupled. There are two parts of Kolla-mesos approach: use micro-services approach inside containers (service discovery, self-configuration etc..) and use Mesos/Marathon definitions to control containers placement. If you want to use Kubernetes you still can use Kolla images and you will need to write simple PODs\ReplicationGroups definitions. OpenStack services will configure themselves by using central configuration storage (ZooKeeper for now but other storages are on their way in oslo.config <a href="https://blueprints.launchpad.net/oslo.config/+spec/oslo-config-db">BP</a> )</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'd be interested in seeing TripeO use the Ansible version for now since that's working, stable, and supports upgrades/oddball configs. Then in the future as Kubernetes support or maybe Mesos support matures, consider that. Kolla's going to have to have a migration path from one to the other eventually... I think this would allow TripeO to really come into its own as an end to end, production ready system sooner.<br>
<br>
Thanks,<br>
Kevin<br>
<br>
<br>
<br>
<br>
_______________________________________<br>
From: Emilien Macchi [<a href="mailto:emilien@redhat.com">emilien@redhat.com</a>]<br>
Sent: Tuesday, March 22, 2016 5:18 PM<br>
To: OpenStack Development Mailing List (not for usage questions)<br>
Subject: Re: [openstack-dev] [TripleO][Heat][Kolla][Magnum] The zen of Heat, containers, and the future of TripleO<br>
<div class="HOEnZb"><div class="h5"><br>
This is quite a long mail, I'll reply to some statements inline.<br>
<br>
On Mon, Mar 21, 2016 at 4:14 PM, Zane Bitter <<a href="mailto:zbitter@redhat.com">zbitter@redhat.com</a>> wrote:<br>
> tl;dr Containers represent a massive, and also mandatory, opportunity for<br>
> TripleO. Lets start thinking about ways that we can take maximum advantage<br>
> to achieve the goals of the project.<br>
><br>
> Now that you have the tl;dr I'm going to start from the beginning, so settle<br>
> in and grab yourself a cup of coffee or other poison of your choice.<br>
><br>
> After working on developing Heat from the very beginning of the project in<br>
> early 2012 and debugging a bunch of TripleO deployments in the field, it is<br>
> my considered opinion that Heat is a poor fit for the workloads that TripleO<br>
> is currently asking of it. To illustrate why, I need to explain what it is<br>
> that Heat is really designed to do.<br>
><br>
> Here's a theoretical example of how I've always imagined Heat software<br>
> deployments would make Heat users' lives better. For simplicity, I'm just<br>
> going to model two software components, a user-facing service that connects<br>
> to some back-end service:<br>
><br>
> resources:<br>
> backend_component:<br>
> type: OS::Heat::SoftwareComponent<br>
> properties:<br>
> configs:<br>
> - tool: script<br>
> actions:<br>
> - CREATE<br>
> - UPDATE<br>
> config: |<br>
> PORT=$(get_backend_port || random_port)<br>
> stop_backend<br>
> start_backend $DEPLOY_VERSION $PORT $CONFIG<br>
> addr="$(hostname):$(get_backend_port)"<br>
> printf '%s' "$addr" >${heat_outputs_path}.host_and_port<br>
> - tool: script<br>
> actions:<br>
> - DELETE<br>
> config: |<br>
> stop_backend<br>
> inputs:<br>
> - name: DEPLOY_VERSION<br>
> - name: CONFIG<br>
> outputs:<br>
> - name: host_and_port<br>
><br>
> frontend_component:<br>
> type: OS::Heat::SoftwareComponent<br>
> properties:<br>
> configs:<br>
> - tool: script<br>
> actions:<br>
> - CREATE<br>
> - UPDATE<br>
> config: |<br>
> stop_frontend<br>
> start_frontend $DEPLOY_VERSION $BACKEND_ADDR $CONFIG<br>
> - tool: script<br>
> actions:<br>
> - DELETE<br>
> config: |<br>
> stop_frontend<br>
> inputs:<br>
> - name: DEPLOY_VERSION<br>
> - name: BACKEND_ADDR<br>
> - name: CONFIG<br>
><br>
> backend:<br>
> type: OS::Heat::SoftwareDeployment<br>
> properties:<br>
> server: {get_resource: backend_server}<br>
> name: {get_param: backend_version} # Forces upgrade replacement<br>
> actions: [CREATE, UPDATE, DELETE]<br>
> config: {get_resource: backend_component}<br>
> input_values:<br>
> DEPLOY_VERSION: ${get_param: backend_version}<br>
> CONFIG: ${get_param: backend_config}<br>
><br>
> frontend:<br>
> type: OS::Heat::SoftwareDeployment<br>
> properties:<br>
> server: {get_resource: frontend_server}<br>
> name: {get_param: frontend_version} # Forces upgrade replacement<br>
> actions: [CREATE, UPDATE, DELETE]<br>
> config: {get_resource: frontend_component}<br>
> input_values:<br>
> DEPLOY_VERSION: ${get_param: frontend_version}<br>
> BACKEND_ADDR: {get_attr: [backend, host_and_port]}<br>
> CONFIG: ${get_param: frontend_config}<br>
><br>
><br>
> This is actually quite a beautiful system, if I may say so:<br>
><br>
> - Whenever a version changes, Heat knows to update that component, and the<br>
> components can be updated independently.<br>
> - If the backend in this example restarts on a different port, the frontend<br>
> is updated to point to the new port.<br>
> - Everything is completely agnostic as to which server it is running on.<br>
> They could be running on the same server or different servers.<br>
> - Everything is integrated with the infrastructure (not only the servers<br>
> you're deploying on and the networks and volumes connected to them, but also<br>
> things like load balancers), so everything is created at the right time, in<br>
> parallel where possible, and any errors are reported all in one place.<br>
> - If something requires e.g. a restart after changing another component, we<br>
> can encode that. And if it doesn't, we can encode that too.<br>
> - There's next to no downtime required: if e.g. we upgrade the backend, we<br>
> first deploy a new one listening on a new port, then update the frontend to<br>
> listen on the new port, then finally shut down the old backend. Again, we<br>
> can choose when we want this and when we just want to update in place and<br>
> reload.<br>
> - The application doesn't even need to worry about versioning the protocol<br>
> that its two constituent parts communicate over: as long as the<br>
> backend_version and frontend_version that we pass are always compatible,<br>
> only compatible versions of the two services ever talk to each other.<br>
> - If anything at all fails at any point before, during or after this part of<br>
> the template, Heat can automatically roll everything back into the exact<br>
> same state as it was in before, without any outside intervention. You can<br>
> insert test deployments that check everything is working and have them<br>
> automatically roll back if it's not, all with no downtime for users.<br>
><br>
> So you can use this to do something like a fancier version of blue-green<br>
> deployment,[1] where you're actually rolling out the (virtualised) hardware<br>
> and infrastructure in a blue-green fashion along with the software. Not only<br>
> that, you can choose to replace your whole stack or only parts of it. (Note:<br>
> the way I had to encode this in the example above, by changing the<br>
> deployment name so that it forces a resource replacement, is a hack. We<br>
> really need a feature to specify in a software config resource which inputs<br>
> should result in a replacement on change.)<br>
><br>
> It's worth noting that in practice you really, really want everything<br>
> deployed in containers to make this process work consistently, even though<br>
> *in theory* you could make this work (briefly) without them. In particular,<br>
> rollback without containers is a dicey proposition. When we first started<br>
> talking about implementing software deployments in Heat I half-seriously<br>
> suggested that maybe we should make containers the only allowed type of<br>
> software deployment, and I kind of wonder now if I shouldn't have pressed<br>
> harder on that point.<br>
><br>
><br>
> In any event, unfortunately as everyone involved in TripleO knows, the way<br>
> TripleO uses Heat looks nothing like this. It actually looks more like this:<br>
><br>
> resources:<br>
> install_all_the_things_on_one_server_config:<br>
> type: OS::Heat::SoftwareConfig<br>
> properties:<br>
> actions: [CREATE]<br>
> config: {get_file: install_all_the_things_on_one_server.sh}<br>
><br>
> update_all_the_things_on_one_server_config:<br>
> type: OS::Heat::SoftwareConfig<br>
> properties:<br>
> actions: [UPDATE]<br>
> config: {get_file: update_all_the_things_on_one_server.sh}<br>
> inputs:<br>
> - name: update_count<br>
><br>
> ...<br>
><br>
> (Filling in the rest is left as an exercise to the reader. You're welcome.)<br>
<br>
I have to be honest, I'm not OpenStack Heat expert and the last block<br>
is very complex to understand.<br>
Maybe a simple example with Keystone API would have been great.<br>
<br>
> Not illustrated are the multiple sources of truth that we have: puppet<br>
> modules (packaged on the server), puppet manifests and hieradata (delivered<br>
> via Heat), external package repositories. Heat is a dataflow language but<br>
> much of the data it should be operating on is actually hidden from it.<br>
> That's going about as well as you might expect.<br>
><br>
> Due to the impossibility of ever rolling back a deployment like one of<br>
> those, we just disable rollback for the overcloud templates, so if there's a<br>
> failure we end up stuck in whatever intermediate state we were in when the<br>
> script died. That can leave things in an state where recovery is not<br>
> automatic when 'earlier' deployments (like the package update) end up<br>
> depending on state set up by 'later' deployments (like the post- scripts,<br>
> which manipulate Pacemaker's state in Pacemaker-based deployments). Even<br>
> worse, many of the current scripts leave the machine in a state that<br>
> requires manual recovery should they fail part-way through.<br>
><br>
> Indeed, this has literally none of the benefits of the ideal Heat deployment<br>
> enumerated above save one: it may be entirely the wrong tool in every way<br>
> for the job it's being asked to do, but at least it is still well-integrated<br>
> with the rest of the infrastructure.<br>
><br>
> Now, at the Mitaka summit we discussed the idea of a 'split stack', where we<br>
> have one stack for the infrastructure and a separate one for the software<br>
> deployments, so that there is no longer any tight integration between<br>
> infrastructure and software. Although it makes me a bit sad in some ways, I<br>
> can certainly appreciate the merits of the idea as well. However, from the<br>
> argument above we can deduce that if this is the *only* thing we do then we<br>
> will end up in the very worst of all possible worlds: the wrong tool for the<br>
> job, poorly integrated. Every single advantage of using Heat to deploy<br>
> software will have evaporated, leaving only disadvantages.<br>
><br>
> So what would be a good alternative? And how would we evaluate the options?<br>
><br>
><br>
> To my mind, the purpose of the TripleO project is this: to ensure that there<br>
> is an OpenStack community collaborating around each part of the OpenStack<br>
> installation/management story. We don't care about TripleO "owning" that<br>
> part (all things being equal, we'd prefer not to), just that nobody should<br>
> have to go outside the OpenStack community and/or roll their own thing to<br>
> install OpenStack unless they want to. So I think the ability to sustain a<br>
> community around whatever solution we choose ought to be a primary<br>
> consideration.<br>
><br>
> The use of Ironic has been something of a success story here. There's only<br>
> one place to add hardware support to enable both installing OpenStack itself<br>
> on bare-metal via TripleO and the 'regular' bare-metal-to-tenant use case of<br>
> Ironic. This is a clear win/win.<br>
><br>
> Beyond getting the bare-metal machines marshalled, the other part of the<br>
> solution is configuration management and orchestration of the various<br>
> software services. When TripleO started there was nowhere in OpenStack that<br>
> was defining the relationships between services needed to orchestrate them.<br>
> To a large extent there still isn't. I think that one of the reasons we<br>
> adopted Puppet in TripleO was that it was supposed to provide this, at least<br>
> within a limited scope (i.e. on one machine - the puppet-deploying community<br>
> is largely using Ansible to orchestrate across boxes, and we are using<br>
> Heat). However, what we've discovered in the past few months is that Puppet<br>
> is actually not able to fulfil this role as long as we support<br>
> Pacemaker-based deployments as an option, because in that case Pacemaker<br>
> actually has control of starting and stopping all of the services. As a<br>
> result we are back to defining it all ourselves in the Pacemaker config plus<br>
> various hacky shell scripts, instead of relying on (and contributing to!) a<br>
> larger community. Even ignoring that, Puppet doesn't solve the problem of<br>
> orchestrating across multiple machines.<br>
<br>
That is True there was/is an overlap between Puppet & Pacemaker<br>
orchestration for services management, that's why we disable service<br>
management in Puppet.<br>
This is something we are working on by created a Puppet provider to<br>
control 'pcs'.<br>
<br>
<a href="https://review.openstack.org/#/c/286124/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/286124/</a><br>
<br>
Thanks to this work, deployments done by Puppet will manage resources<br>
using pcs CLI and we'll be able to cleanup a lot of hacks that made it<br>
possible in the past.<br>
I agree Pacemaker is not helping, it has nothing to do with the<br>
deployment tool: Pacemaker is complex to deploy and manage.<br>
On a positive way, we are working on improving the Pacemaker/Puppet<br>
collision and I think we will fix that during Newton cycle.<br>
<br>
> Clearly one option would be to encode everything in Heat along the lines of<br>
> the first example above. I think once we have containers this could actually<br>
> work really well for compute nodes and other types of scale-out nodes (e.g.<br>
> Swift nodes). The scale-out model of Heat scaling groups works really well<br>
> for this use case, and between the improvements we have put in place (like<br>
> batched updates and user hooks) and those still on the agenda (like<br>
> notifications + automatic Mistral workflow triggering on hooks) Heat could<br>
> provide a really good way of capturing things like migrating user workloads<br>
> on scale down and rolling updates in the templates, so that they can be<br>
> managed completely automatically by the undercloud with no client<br>
> involvement (and when the undercloud becomes HA, they'll get HA for free).<br>
> I'd be pretty excited to see this tried. The potential downside is that the<br>
> orchestration definitions are still trapped inside the TripleO templates, so<br>
> they're not being shared outside of the TripleO community. This is probably<br>
> justified though owing to its close ties to the underlying infrastructure.<br>
><br>
> An alternative out of left field: as far as I can gather the "completely new<br>
> way of orchestrating activities" used by the new Puppet Application<br>
> Orchestration thing[2] uses substantially the same model as I described for<br>
> Heat above. If we added Puppet Application Orchestration data to<br>
> openstack-puppet-modules then it may be possible to write a tool to generate<br>
> Heat templates from that data. However in talking with Emilien it sounds<br>
> like o-p-m is quite some time away from tackling PAO. So I don't think this<br>
> is really feasible.<br>
<br>
That is True, Puppet OpenStack modules do not support PAO and I<br>
haven't seen any efforts from Puppetlabs to help us doing that.<br>
I'm even not sure about how it would work, since it require a<br>
Puppetmaster and some pieces are not Open-Source. Beurk.<br>
<br>
> In any event, it's when we get to the controller nodes that the downsides<br>
> become more pronounced. We're no longer talking about one deployment per<br>
> service like I sketched above; each service is actually multiple deployments<br>
> forming an active-active cluster with virtual IPs and failover and all that<br>
> jazz. It may be that everything would just work the same way, but we would<br>
> be in uncharted territory and there would likely be unanticipated<br>
> subtleties. It's particularly unclear how we would handle stop-the-world<br>
> database migrations in this model, although we do have the option of hoping<br>
> that stop-the-world database migrations will have been completely phased out<br>
> by then.<br>
><br>
> To make it even more complicated, we ultimately want the services to<br>
> heterogeneously spread among controller nodes in a configurable way. I<br>
> believe that Dan's work on composable roles has already gone some way toward<br>
> this without even using containers, but it's likely to become increasingly<br>
> difficult to model in Heat without some sort of template generation. (I<br>
> personally think that template generation would be a Good Thing, but we've<br>
> chosen not to go down that path so far.) Quite possibly even just having<br>
> composable roles could make it untenable to continue maintaining separate<br>
> Pacemaker and non-Pacemaker deployment modes. It'd be really nice to have<br>
> the flexibility to do things like scale out different services at different<br>
> rates. What's more, we are going to need some way of redistributing services<br>
> when a machine in the cluster fails, and ultimately we would like that<br>
> process to be automated, which would *require* a template generation<br>
> service.<br>
><br>
> We certainly *could* build all of that. But we definitely shouldn't because<br>
> this is the kind of thing that services like Kubernetes and Apache Mesos are<br>
> designed to do already. And that raises another possibility: Angus & friends<br>
> are working on capturing the orchestration relationships for Mesos+Marathon<br>
> within the Kolla project (specifically, in the kolla-mesos repository). This<br>
> represents a tremendous opportunity for the TripleO project to further its<br>
> mission of having the same deployment tools available to everyone as an<br>
> official part of the OpenStack project without having to maintain them<br>
> separately.<br>
><br>
> As of the Liberty release, Magnum now supports provisioning Mesos clusters,<br>
> so TripleO wouldn't have to maintain the installer for that either. (The<br>
> choice of Mesos is somewhat unfortunate in our case, because Magnum's<br>
> Kubernetes support is much more mature than its Mesos support, and because<br>
> the reasons for the decision are about to be or have already been overtaken<br>
> by events - I've heard reports that the features that Kubernetes was missing<br>
> to allow it to be used for controller nodes, and maybe even compute nodes,<br>
> are now available. Nonetheless, I expect the level of Magnum support for<br>
> Mesos is likely workable.) This is where the TripleO strategy of using<br>
> OpenStack to deploy OpenStack can really pay dividends: because we use<br>
> Ironic all of our servers are accessible through the Nova API, so in theory<br>
> we can just run Magnum out of the box.<br>
<br>
This is something we already & partially discussed when I submitted<br>
the Containers blueprint, in September 2015:<br>
<a href="https://review.openstack.org/#/c/222002/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/222002/</a><br>
<br>
I was told to abandon this blueprint, so Container folks could create a new one:<br>
<a href="https://review.openstack.org/#/c/223182/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/223182/</a><br>
<br>
The "new" blueprint was never merged so I guess TripleO team accepted<br>
the design, moreoever discussed during Tokyo Summit.<br>
I am sad to see you were not involved in both blueprints, nor present<br>
during our discussions, but let's move forward.<br>
<br>
><br>
> The chances of me personally having time to prototype this are slim-to-zero,<br>
> but I think this is a path worth investigating.<br>
<br>
A lot of people are currently very busy, because of releases and other<br>
critical topics, like CI, split stack and composable roles, maybe<br>
people focusing on containers can also reply to this thread and try to<br>
understand why we're here now.<br>
<br>
I know it has currently no impact on TripleO, but I plan to gather<br>
feedback about Puppet & Containers usage during next Summit in Austin,<br>
I know TimeWarner is deploying OpenStack with docker & Puppet<br>
OpenStack modules at scale, and they have a pretty nice experience on<br>
that.<br>
--<br>
Emilien Macchi<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div class="gmail_signature"><div dir="ltr"><font color="#999999"><span style="background-color:rgb(255,255,255)">Georgy Okrokvertskhov<br>
Director of Performance Engineering,<br><span style="font-family:arial;font-size:small">OpenStack Platform Products,</span><br>
Mirantis</span><br>
<a href="http://www.mirantis.com/" target="_blank">http://www.mirantis.com</a><br>
Tel. +1 650 963 9828<br>
Mob. +1 650 996 3284</font><br></div></div></div></div>
</div></div>