Open Stack

This is quite a long mail, I'll reply to some statements inline.

On Mon, Mar 21, 2016 at 4:14 PM, Zane Bitter <zbitter at redhat.com> wrote:
> tl;dr Containers represent a massive, and also mandatory, opportunity for
> TripleO. Lets start thinking about ways that we can take maximum advantage
> to achieve the goals of the project.
>
> Now that you have the tl;dr I'm going to start from the beginning, so settle
> in and grab yourself a cup of coffee or other poison of your choice.
>
> After working on developing Heat from the very beginning of the project in
> early 2012 and debugging a bunch of TripleO deployments in the field, it is
> my considered opinion that Heat is a poor fit for the workloads that TripleO
> is currently asking of it. To illustrate why, I need to explain what it is
> that Heat is really designed to do.
>
> Here's a theoretical example of how I've always imagined Heat software
> deployments would make Heat users' lives better. For simplicity, I'm just
> going to model two software components, a user-facing service that connects
> to some back-end service:
>
>   resources:
>     backend_component:
>       type: OS::Heat::SoftwareComponent
>       properties:
>         configs:
>           - tool: script
>             actions:
>               - CREATE
>               - UPDATE
>             config: |
>               PORT=$(get_backend_port || random_port)
>               stop_backend
>               start_backend $DEPLOY_VERSION $PORT $CONFIG
>               addr="$(hostname):$(get_backend_port)"
>               printf '%s' "$addr" >${heat_outputs_path}.host_and_port
>           - tool: script
>             actions:
>               - DELETE
>             config: |
>                stop_backend
>          inputs:
>            - name: DEPLOY_VERSION
>            - name: CONFIG
>          outputs:
>            - name: host_and_port
>
>     frontend_component:
>       type: OS::Heat::SoftwareComponent
>       properties:
>         configs:
>           - tool: script
>             actions:
>               - CREATE
>               - UPDATE
>             config: |
>               stop_frontend
>               start_frontend $DEPLOY_VERSION $BACKEND_ADDR $CONFIG
>           - tool: script
>             actions:
>               - DELETE
>             config: |
>               stop_frontend
>         inputs:
>           - name: DEPLOY_VERSION
>           - name: BACKEND_ADDR
>           - name: CONFIG
>
>     backend:
>       type: OS::Heat::SoftwareDeployment
>       properties:
>         server: {get_resource: backend_server}
>         name: {get_param: backend_version} # Forces upgrade replacement
>         actions: [CREATE, UPDATE, DELETE]
>         config: {get_resource: backend_component}
>         input_values:
>           DEPLOY_VERSION: ${get_param: backend_version}
>           CONFIG: ${get_param: backend_config}
>
>     frontend:
>       type: OS::Heat::SoftwareDeployment
>       properties:
>         server: {get_resource: frontend_server}
>         name: {get_param: frontend_version} # Forces upgrade replacement
>         actions: [CREATE, UPDATE, DELETE]
>         config: {get_resource: frontend_component}
>         input_values:
>           DEPLOY_VERSION: ${get_param: frontend_version}
>           BACKEND_ADDR: {get_attr: [backend, host_and_port]}
>           CONFIG: ${get_param: frontend_config}
>
>
> This is actually quite a beautiful system, if I may say so:
>
> - Whenever a version changes, Heat knows to update that component, and the
> components can be updated independently.
> - If the backend in this example restarts on a different port, the frontend
> is updated to point to the new port.
> - Everything is completely agnostic as to which server it is running on.
> They could be running on the same server or different servers.
> - Everything is integrated with the infrastructure (not only the servers
> you're deploying on and the networks and volumes connected to them, but also
> things like load balancers), so everything is created at the right time, in
> parallel where possible, and any errors are reported all in one place.
> - If something requires e.g. a restart after changing another component, we
> can encode that. And if it doesn't, we can encode that too.
> - There's next to no downtime required: if e.g. we upgrade the backend, we
> first deploy a new one listening on a new port, then update the frontend to
> listen on the new port, then finally shut down the old backend. Again, we
> can choose when we want this and when we just want to update in place and
> reload.
> - The application doesn't even need to worry about versioning the protocol
> that its two constituent parts communicate over: as long as the
> backend_version and frontend_version that we pass are always compatible,
> only compatible versions of the two services ever talk to each other.
> - If anything at all fails at any point before, during or after this part of
> the template, Heat can automatically roll everything back into the exact
> same state as it was in before, without any outside intervention. You can
> insert test deployments that check everything is working and have them
> automatically roll back if it's not, all with no downtime for users.
>
> So you can use this to do something like a fancier version of blue-green
> deployment,[1] where you're actually rolling out the (virtualised) hardware
> and infrastructure in a blue-green fashion along with the software. Not only
> that, you can choose to replace your whole stack or only parts of it. (Note:
> the way I had to encode this in the example above, by changing the
> deployment name so that it forces a resource replacement, is a hack. We
> really need a feature to specify in a software config resource which inputs
> should result in a replacement on change.)
>
> It's worth noting that in practice you really, really want everything
> deployed in containers to make this process work consistently, even though
> *in theory* you could make this work (briefly) without them. In particular,
> rollback without containers is a dicey proposition. When we first started
> talking about implementing software deployments in Heat I half-seriously
> suggested that maybe we should make containers the only allowed type of
> software deployment, and I kind of wonder now if I shouldn't have pressed
> harder on that point.
>
>
> In any event, unfortunately as everyone involved in TripleO knows, the way
> TripleO uses Heat looks nothing like this. It actually looks more like this:
>
>   resources:
>     install_all_the_things_on_one_server_config:
>       type: OS::Heat::SoftwareConfig
>       properties:
>         actions: [CREATE]
>         config: {get_file: install_all_the_things_on_one_server.sh}
>
>     update_all_the_things_on_one_server_config:
>       type: OS::Heat::SoftwareConfig
>       properties:
>         actions: [UPDATE]
>         config: {get_file: update_all_the_things_on_one_server.sh}
>         inputs:
>           - name: update_count
>
>     ...
>
> (Filling in the rest is left as an exercise to the reader. You're welcome.)

I have to be honest, I'm not OpenStack Heat expert and the last block
is very complex to understand.
Maybe a simple example with Keystone API would have been great.

> Not illustrated are the multiple sources of truth that we have: puppet
> modules (packaged on the server), puppet manifests and hieradata (delivered
> via Heat), external package repositories. Heat is a dataflow language but
> much of the data it should be operating on is actually hidden from it.
> That's going about as well as you might expect.
>
> Due to the impossibility of ever rolling back a deployment like one of
> those, we just disable rollback for the overcloud templates, so if there's a
> failure we end up stuck in whatever intermediate state we were in when the
> script died. That can leave things in an state where recovery is not
> automatic when 'earlier' deployments (like the package update) end up
> depending on state set up by 'later' deployments (like the post- scripts,
> which manipulate Pacemaker's state in Pacemaker-based deployments). Even
> worse, many of the current scripts leave the machine in a state that
> requires manual recovery should they fail part-way through.
>
> Indeed, this has literally none of the benefits of the ideal Heat deployment
> enumerated above save one: it may be entirely the wrong tool in every way
> for the job it's being asked to do, but at least it is still well-integrated
> with the rest of the infrastructure.
>
> Now, at the Mitaka summit we discussed the idea of a 'split stack', where we
> have one stack for the infrastructure and a separate one for the software
> deployments, so that there is no longer any tight integration between
> infrastructure and software. Although it makes me a bit sad in some ways, I
> can certainly appreciate the merits of the idea as well. However, from the
> argument above we can deduce that if this is the *only* thing we do then we
> will end up in the very worst of all possible worlds: the wrong tool for the
> job, poorly integrated. Every single advantage of using Heat to deploy
> software will have evaporated, leaving only disadvantages.
>
> So what would be a good alternative? And how would we evaluate the options?
>
>
> To my mind, the purpose of the TripleO project is this: to ensure that there
> is an OpenStack community collaborating around each part of the OpenStack
> installation/management story. We don't care about TripleO "owning" that
> part (all things being equal, we'd prefer not to), just that nobody should
> have to go outside the OpenStack community and/or roll their own thing to
> install OpenStack unless they want to. So I think the ability to sustain a
> community around whatever solution we choose ought to be a primary
> consideration.
>
> The use of Ironic has been something of a success story here. There's only
> one place to add hardware support to enable both installing OpenStack itself
> on bare-metal via TripleO and the 'regular' bare-metal-to-tenant use case of
> Ironic. This is a clear win/win.
>
> Beyond getting the bare-metal machines marshalled, the other part of the
> solution is configuration management and orchestration of the various
> software services. When TripleO started there was nowhere in OpenStack that
> was defining the relationships between services needed to orchestrate them.
> To a large extent there still isn't. I think that one of the reasons we
> adopted Puppet in TripleO was that it was supposed to provide this, at least
> within a limited scope (i.e. on one machine - the puppet-deploying community
> is largely using Ansible to orchestrate across boxes, and we are using
> Heat). However, what we've discovered in the past few months is that Puppet
> is actually not able to fulfil this role as long as we support
> Pacemaker-based deployments as an option, because in that case Pacemaker
> actually has control of starting and stopping all of the services. As a
> result we are back to defining it all ourselves in the Pacemaker config plus
> various hacky shell scripts, instead of relying on (and contributing to!) a
> larger community. Even ignoring that, Puppet doesn't solve the problem of
> orchestrating across multiple machines.

That is True there was/is an overlap between Puppet & Pacemaker
orchestration for services management, that's why we disable service
management in Puppet.
This is something we are working on by created a Puppet provider to
control 'pcs'.

https://review.openstack.org/#/c/286124/

Thanks to this work, deployments done by Puppet will manage resources
using pcs CLI and we'll be able to cleanup a lot of hacks that made it
possible in the past.
I agree Pacemaker is not helping, it has nothing to do with the
deployment tool: Pacemaker is complex to deploy and manage.
On a positive way, we are working on improving the Pacemaker/Puppet
collision and I think we will fix that during Newton cycle.

> Clearly one option would be to encode everything in Heat along the lines of
> the first example above. I think once we have containers this could actually
> work really well for compute nodes and other types of scale-out nodes (e.g.
> Swift nodes). The scale-out model of Heat scaling groups works really well
> for this use case, and between the improvements we have put in place (like
> batched updates and user hooks) and those still on the agenda (like
> notifications + automatic Mistral workflow triggering on hooks) Heat could
> provide a really good way of capturing things like migrating user workloads
> on scale down and rolling updates in the templates, so that they can be
> managed completely automatically by the undercloud with no client
> involvement (and when the undercloud becomes HA, they'll get HA for free).
> I'd be pretty excited to see this tried. The potential downside is that the
> orchestration definitions are still trapped inside the TripleO templates, so
> they're not being shared outside of the TripleO community. This is probably
> justified though owing to its close ties to the underlying infrastructure.
>
> An alternative out of left field: as far as I can gather the "completely new
> way of orchestrating activities" used by the new Puppet Application
> Orchestration thing[2] uses substantially the same model as I described for
> Heat above. If we added Puppet Application Orchestration data to
> openstack-puppet-modules then it may be possible to write a tool to generate
> Heat templates from that data. However in talking with Emilien it sounds
> like o-p-m is quite some time away from tackling PAO. So I don't think this
> is really feasible.

That is True, Puppet OpenStack modules do not support PAO and I
haven't seen any efforts from Puppetlabs to help us doing that.
I'm even not sure about how it would work, since it require a
Puppetmaster and some pieces are not Open-Source. Beurk.

> In any event, it's when we get to the controller nodes that the downsides
> become more pronounced. We're no longer talking about one deployment per
> service like I sketched above; each service is actually multiple deployments
> forming an active-active cluster with virtual IPs and failover and all that
> jazz. It may be that everything would just work the same way, but we would
> be in uncharted territory and there would likely be unanticipated
> subtleties. It's particularly unclear how we would handle stop-the-world
> database migrations in this model, although we do have the option of hoping
> that stop-the-world database migrations will have been completely phased out
> by then.
>
> To make it even more complicated, we ultimately want the services to
> heterogeneously spread among controller nodes in a configurable way. I
> believe that Dan's work on composable roles has already gone some way toward
> this without even using containers, but it's likely to become increasingly
> difficult to model in Heat without some sort of template generation. (I
> personally think that template generation would be a Good Thing, but we've
> chosen not to go down that path so far.) Quite possibly even just having
> composable roles could make it untenable to continue maintaining separate
> Pacemaker and non-Pacemaker deployment modes. It'd be really nice to have
> the flexibility to do things like scale out different services at different
> rates. What's more, we are going to need some way of redistributing services
> when a machine in the cluster fails, and ultimately we would like that
> process to be automated, which would *require* a template generation
> service.
>
> We certainly *could* build all of that. But we definitely shouldn't because
> this is the kind of thing that services like Kubernetes and Apache Mesos are
> designed to do already. And that raises another possibility: Angus & friends
> are working on capturing the orchestration relationships for Mesos+Marathon
> within the Kolla project (specifically, in the kolla-mesos repository). This
> represents a tremendous opportunity for the TripleO project to further its
> mission of having the same deployment tools available to everyone as an
> official part of the OpenStack project without having to maintain them
> separately.
>
> As of the Liberty release, Magnum now supports provisioning Mesos clusters,
> so TripleO wouldn't have to maintain the installer for that either. (The
> choice of Mesos is somewhat unfortunate in our case, because Magnum's
> Kubernetes support is much more mature than its Mesos support, and because
> the reasons for the decision are about to be or have already been overtaken
> by events - I've heard reports that the features that Kubernetes was missing
> to allow it to be used for controller nodes, and maybe even compute nodes,
> are now available. Nonetheless, I expect the level of Magnum support for
> Mesos is likely workable.) This is where the TripleO strategy of using
> OpenStack to deploy OpenStack can really pay dividends: because we use
> Ironic all of our servers are accessible through the Nova API, so in theory
> we can just run Magnum out of the box.

This is something we already & partially discussed when I submitted
the Containers blueprint, in September 2015:
https://review.openstack.org/#/c/222002/

I was told to abandon this blueprint, so Container folks could create a new one:
https://review.openstack.org/#/c/223182/

The "new" blueprint was never merged so I guess TripleO team accepted
the design, moreoever discussed during Tokyo Summit.
I am sad to see you were not involved in both blueprints, nor present
during our discussions, but let's move forward.

>
> The chances of me personally having time to prototype this are slim-to-zero,
> but I think this is a path worth investigating.

A lot of people are currently very busy, because of releases and other
critical topics, like CI, split stack and composable roles, maybe
people focusing on containers can also reply to this thread and try to
understand why we're here now.

I know it has currently no impact on TripleO, but I plan to gather
feedback about Puppet & Containers usage during next Summit in Austin,
I know TimeWarner is deploying OpenStack with docker & Puppet
OpenStack modules at scale, and they have a pretty nice experience on
that.
-- 
Emilien Macchi