Open Stack

On 8/5/15, 11:33 AM, "Ryan Hallisey" <rhallise at redhat.com> wrote:

>Tagging kolla so the kolla community also sees it.
>Pardon the top posting.
>
>-Ryan
Ryan,

Super appreciated I may have missed this in my deluge of mail - please
keep doing that - or if it involves kolla please tag as kolla as I read
that folder often :)

Dan,

Thanks for the detailed posting.

Comments inline.
>
>----- Original Message -----
>From: "Dan Prince" <dprince at redhat.com>
>To: "openstack-dev" <openstack-dev at lists.openstack.org>
>Sent: Wednesday, August 5, 2015 2:29:13 PM
>Subject: [openstack-dev] [TripleO] [Puppet] Deploying OpenStack with
>Puppet modules on Docker with Heat
>
>Hi,
>
>There is a lot of interest in getting support for container based
>deployment within TripleO and many different ideas and opinions on how
>to go about doing that.
>
>One idea on the table is to use Heat to help orchestrate the deployment
>of docker containers. This would work similar to our tripleo-heat
>-templates implementation except that when using docker you would swap
>in a nested stack template that would configure containers on
>baremetal. We've even got a nice example that shows what a
>containerized TripleO overcloud might look like here [1]. The approach
>outlines how you might use kolla docker containers alongside of the
>tripleo-heat-templates to do this sort of deployment.
>
>This is all cool stuff but one area of concern is how we do the actual
>configuration of the containers. The above implementation relies on
>passing environment variables into kolla built docker containers which
>then self configure all the required config files and start the
>service. This sounds like a start... but creating (and maintaining)
>another from scratch OpenStack configuration tool isn't high on my list

Agree we came to the exact same conclusion which is why we changed to the
config-external config strategy in the ansible-multi spec at:

https://review.openstack.org/#/c/189157/


>of things to spend time on. Sure there is already a kolla community
>helping to build and maintain this configuration tooling (mostly
>thinking config files here) but this sounds a bit like what tripleo
>-image-elements initially tried to do and it turns out there are much
>more capable configuration tools out there.

We would prefer to drop config-internal if TripleO doesn¹t plan to use it
and focus on config-external.  The reason we left it intact is we didn¹t
want to leave TripleO in the lurch.

>
>Since we are already using a good bit of Puppet in tripleo-heat
>-templates the idea came up that we would try to configure Docker
>containers using Puppet. Again, here there are several ideas in the
>Puppet community with regards to how docker might best be configured
>with Puppet. Keeping those in mind we've been throwing some ideas out
>on an etherpad here [2] that describes using Heat for orchestration,
>Puppet for configuration, and Kolla docker images for containers.
>
>A quick outline of the approach is:
>
>-Extend the heat-container-agent [3] that runs os-collect-config and
>all the required hooks we require for deployment. This includes docker
>-compute, bash scripts, and Puppet. NOTE: As described in the etherpad
>I've taken to using DIB to build this container. I found this to be
>faster from a TripleO development baseline.
>
>-To create config files the heat-container-agent would run a puppet
>manifest for a given role and generate a directory tree of config files
>(/var/lib/etc-data for example).

Would prefer per file mounting for config-external. (see more below)

>
>-We then run a docker-compose software deployment that mounts those
>configuration file(s) into a read only volume and uses them to start
>the containerized service.

Sounds good.

>
>The approach could look something like this [4]. This nice thing about
>this is that it requires no modification to OpenStack Puppet modules.
>We can use those today, as-is. Additionally, although Puppet runs in
>the agent container we've created a mechanism to set all the resources
>to noop mode except for those that generate config files. And lastly,
>we can use exactly the same role manifest for docker that we do for
>baremetal. Lots of re-use here... and although we are disabling a lot
>of Puppet functionality in setting all the non-config resources to noop
>the Kolla containers already do some of that stuff for us (starting
>services, etc.).
>
>----
>
>All that said (and trying to keep this short) we've still got a bit of
>work to do around wiring up externally created config files to kolla
>build docker containers. A couple of issues are:
>
>-The external config file mechanism for Kolla containers only seems to
>support a single config file. Some services (Neutron) can have multiple
>files. Could we extend the external config support to use multiple
>files?

Kolla supports multiple configuration files.  An example:

https://github.com/stackforge/kolla/blob/master/docker/common/neutron/neutr
on-server/config-external.sh


We will extend to new config files as it makes sense and I don¹t really
feel all that terrible about importing even proprietary config file
formats for specific proprietary drivers that neutron (or cinder or
whatever) may need if there was an actual interest and the file format
were published upstream somewhere so we could process it properly in
Ansible if we choose to do so at a later time.

>
>-If a service has multiple files kolla may need to adjust its service
>startup script to use multiple files. Perhaps a conf.d approach would
>work here?

A conf.d approach could work, but I¹d prefer to use the design pattern in
Neutron above.  The downside of the neutron pattern is each file must be
separately specified.  The reason I¹d prefer this approach is it maintains
container immutability without pollution of files into the container.
With a directory, new files could be loaded into the etc directory without
directly restarting the container with a :ro flag.  This would break the
immutability of the container, and I¹m not sure how to protect against
that happening with a directory import.  This is why we import only files.

>
>-We are missing published version of some key kolla containers. Namely
>openvswitch and the neutron-openvswitch-agent for starters but I'd also
>like to have a Ceilometer agent and SNMP agent container as well so we
>have feature parity with the non-docker compute role.

Openvswitch works fantastically today and was our first Ansible deployment
mechanism.  We later added linuxbridge support and plan nova-networking
for l3.  We don¹t plan to port this work to config-internal config
strategy, but as stated in our ansible-multi review, we won¹t turn away
patches that implement this work in the config-internal model.

Our ovs work is here:

https://github.com/stackforge/kolla/tree/master/docker/common/openvswitch
https://github.com/stackforge/kolla/tree/master/docker/common/neutron/neutr
on-openvswitch-agent


Ceilometer agent - agreed this is needed and high on my list for l3.  Not
familiar with an SNMP agent but it doesn¹t sound too difficult to take on.
 If you can provide more details on what to configure, I¹ll get a
blueprint filed and we can get this work prioritized for l3.  Our
ceilometer container in general needs some love to be tackled in l3.

It is likely you looked at the repo a couple weeks ago when this stuff
wasn¹t fully baked.  I realize we are moving pretty fast so no harm on the
lag :)

Regards
-steve

>
>Once we have solutions for the above I think we'll be very close to a
>fully dockerized compute role with TripleO heat templates. From there
>we can expand the idea to cover other roles within the tripleo-heat
>-templates too.
>
>I'll stop there for now. Any ideas and thoughts appreciated.
>
>Dan
>
>-----
>
>[1] https://review.openstack.org/#/c/178840/ (Containerized TripleO
>Overcloud.)
>[2] https://etherpad.openstack.org/p/tripleo-docker-puppet
>[3] http://git.openstack.org/cgit/openstack/heat
>-templates/log/hot/software-config/heat-container-agent
>[4] https://review.openstack.org/#/c/209505/  (Docker compute role
>configured via Puppet)
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev