Open Stack

Hi,

There is a lot of interest in getting support for container based
deployment within TripleO and many different ideas and opinions on how
to go about doing that.

One idea on the table is to use Heat to help orchestrate the deployment
of docker containers. This would work similar to our tripleo-heat
-templates implementation except that when using docker you would swap
in a nested stack template that would configure containers on
baremetal. We've even got a nice example that shows what a
containerized TripleO overcloud might look like here [1]. The approach
outlines how you might use kolla docker containers alongside of the
tripleo-heat-templates to do this sort of deployment.

This is all cool stuff but one area of concern is how we do the actual
configuration of the containers. The above implementation relies on
passing environment variables into kolla built docker containers which
then self configure all the required config files and start the
service. This sounds like a start... but creating (and maintaining)
another from scratch OpenStack configuration tool isn't high on my list
of things to spend time on. Sure there is already a kolla community
helping to build and maintain this configuration tooling (mostly
thinking config files here) but this sounds a bit like what tripleo
-image-elements initially tried to do and it turns out there are much
more capable configuration tools out there.

Since we are already using a good bit of Puppet in tripleo-heat
-templates the idea came up that we would try to configure Docker
containers using Puppet. Again, here there are several ideas in the
Puppet community with regards to how docker might best be configured
with Puppet. Keeping those in mind we've been throwing some ideas out
on an etherpad here [2] that describes using Heat for orchestration,
Puppet for configuration, and Kolla docker images for containers.

A quick outline of the approach is:

-Extend the heat-container-agent [3] that runs os-collect-config and
all the required hooks we require for deployment. This includes docker
-compute, bash scripts, and Puppet. NOTE: As described in the etherpad
I've taken to using DIB to build this container. I found this to be
faster from a TripleO development baseline.

-To create config files the heat-container-agent would run a puppet
manifest for a given role and generate a directory tree of config files
(/var/lib/etc-data for example).

-We then run a docker-compose software deployment that mounts those
configuration file(s) into a read only volume and uses them to start
the containerized service.

The approach could look something like this [4]. This nice thing about
this is that it requires no modification to OpenStack Puppet modules.
We can use those today, as-is. Additionally, although Puppet runs in
the agent container we've created a mechanism to set all the resources
to noop mode except for those that generate config files. And lastly,
we can use exactly the same role manifest for docker that we do for
baremetal. Lots of re-use here... and although we are disabling a lot
of Puppet functionality in setting all the non-config resources to noop
the Kolla containers already do some of that stuff for us (starting
services, etc.).

----

All that said (and trying to keep this short) we've still got a bit of
work to do around wiring up externally created config files to kolla
build docker containers. A couple of issues are:

-The external config file mechanism for Kolla containers only seems to
support a single config file. Some services (Neutron) can have multiple
files. Could we extend the external config support to use multiple
files?

-If a service has multiple files kolla may need to adjust its service
startup script to use multiple files. Perhaps a conf.d approach would
work here?

-We are missing published version of some key kolla containers. Namely
openvswitch and the neutron-openvswitch-agent for starters but I'd also
like to have a Ceilometer agent and SNMP agent container as well so we
have feature parity with the non-docker compute role.

Once we have solutions for the above I think we'll be very close to a
fully dockerized compute role with TripleO heat templates. From there
we can expand the idea to cover other roles within the tripleo-heat
-templates too.

I'll stop there for now. Any ideas and thoughts appreciated.

Dan

-----

[1] https://review.openstack.org/#/c/178840/ (Containerized TripleO
Overcloud.)
[2] https://etherpad.openstack.org/p/tripleo-docker-puppet
[3] http://git.openstack.org/cgit/openstack/heat
-templates/log/hot/software-config/heat-container-agent
[4] https://review.openstack.org/#/c/209505/  (Docker compute role
configured via Puppet)