Open Stack

Apologies for top post, but I just wanted to point out in the
config-external examples I am aware /opt/kolla is the wrong directory to
configure from (it should be /var/lib/kolla or something similar) and we
will fix this during l3.

Regards
-steve


On 8/5/15, 6:53 PM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:

>
>
>On 8/5/15, 11:33 AM, "Ryan Hallisey" <rhallise at redhat.com> wrote:
>
>>Tagging kolla so the kolla community also sees it.
>>Pardon the top posting.
>>
>>-Ryan
>Ryan,
>
>Super appreciated I may have missed this in my deluge of mail - please
>keep doing that - or if it involves kolla please tag as kolla as I read
>that folder often :)
>
>Dan,
>
>Thanks for the detailed posting.
>
>Comments inline.
>>
>>----- Original Message -----
>>From: "Dan Prince" <dprince at redhat.com>
>>To: "openstack-dev" <openstack-dev at lists.openstack.org>
>>Sent: Wednesday, August 5, 2015 2:29:13 PM
>>Subject: [openstack-dev] [TripleO] [Puppet] Deploying OpenStack with
>>Puppet modules on Docker with Heat
>>
>>Hi,
>>
>>There is a lot of interest in getting support for container based
>>deployment within TripleO and many different ideas and opinions on how
>>to go about doing that.
>>
>>One idea on the table is to use Heat to help orchestrate the deployment
>>of docker containers. This would work similar to our tripleo-heat
>>-templates implementation except that when using docker you would swap
>>in a nested stack template that would configure containers on
>>baremetal. We've even got a nice example that shows what a
>>containerized TripleO overcloud might look like here [1]. The approach
>>outlines how you might use kolla docker containers alongside of the
>>tripleo-heat-templates to do this sort of deployment.
>>
>>This is all cool stuff but one area of concern is how we do the actual
>>configuration of the containers. The above implementation relies on
>>passing environment variables into kolla built docker containers which
>>then self configure all the required config files and start the
>>service. This sounds like a start... but creating (and maintaining)
>>another from scratch OpenStack configuration tool isn't high on my list
>
>Agree we came to the exact same conclusion which is why we changed to the
>config-external config strategy in the ansible-multi spec at:
>
>https://review.openstack.org/#/c/189157/
>
>
>>of things to spend time on. Sure there is already a kolla community
>>helping to build and maintain this configuration tooling (mostly
>>thinking config files here) but this sounds a bit like what tripleo
>>-image-elements initially tried to do and it turns out there are much
>>more capable configuration tools out there.
>
>We would prefer to drop config-internal if TripleO doesn¹t plan to use it
>and focus on config-external.  The reason we left it intact is we didn¹t
>want to leave TripleO in the lurch.
>
>>
>>Since we are already using a good bit of Puppet in tripleo-heat
>>-templates the idea came up that we would try to configure Docker
>>containers using Puppet. Again, here there are several ideas in the
>>Puppet community with regards to how docker might best be configured
>>with Puppet. Keeping those in mind we've been throwing some ideas out
>>on an etherpad here [2] that describes using Heat for orchestration,
>>Puppet for configuration, and Kolla docker images for containers.
>>
>>A quick outline of the approach is:
>>
>>-Extend the heat-container-agent [3] that runs os-collect-config and
>>all the required hooks we require for deployment. This includes docker
>>-compute, bash scripts, and Puppet. NOTE: As described in the etherpad
>>I've taken to using DIB to build this container. I found this to be
>>faster from a TripleO development baseline.
>>
>>-To create config files the heat-container-agent would run a puppet
>>manifest for a given role and generate a directory tree of config files
>>(/var/lib/etc-data for example).
>
>Would prefer per file mounting for config-external. (see more below)
>
>>
>>-We then run a docker-compose software deployment that mounts those
>>configuration file(s) into a read only volume and uses them to start
>>the containerized service.
>
>Sounds good.
>
>>
>>The approach could look something like this [4]. This nice thing about
>>this is that it requires no modification to OpenStack Puppet modules.
>>We can use those today, as-is. Additionally, although Puppet runs in
>>the agent container we've created a mechanism to set all the resources
>>to noop mode except for those that generate config files. And lastly,
>>we can use exactly the same role manifest for docker that we do for
>>baremetal. Lots of re-use here... and although we are disabling a lot
>>of Puppet functionality in setting all the non-config resources to noop
>>the Kolla containers already do some of that stuff for us (starting
>>services, etc.).
>>
>>----
>>
>>All that said (and trying to keep this short) we've still got a bit of
>>work to do around wiring up externally created config files to kolla
>>build docker containers. A couple of issues are:
>>
>>-The external config file mechanism for Kolla containers only seems to
>>support a single config file. Some services (Neutron) can have multiple
>>files. Could we extend the external config support to use multiple
>>files?
>
>Kolla supports multiple configuration files.  An example:
>
>https://github.com/stackforge/kolla/blob/master/docker/common/neutron/neut
>r
>on-server/config-external.sh
>
>
>We will extend to new config files as it makes sense and I don¹t really
>feel all that terrible about importing even proprietary config file
>formats for specific proprietary drivers that neutron (or cinder or
>whatever) may need if there was an actual interest and the file format
>were published upstream somewhere so we could process it properly in
>Ansible if we choose to do so at a later time.
>
>>
>>-If a service has multiple files kolla may need to adjust its service
>>startup script to use multiple files. Perhaps a conf.d approach would
>>work here?
>
>A conf.d approach could work, but I¹d prefer to use the design pattern in
>Neutron above.  The downside of the neutron pattern is each file must be
>separately specified.  The reason I¹d prefer this approach is it maintains
>container immutability without pollution of files into the container.
>With a directory, new files could be loaded into the etc directory without
>directly restarting the container with a :ro flag.  This would break the
>immutability of the container, and I¹m not sure how to protect against
>that happening with a directory import.  This is why we import only files.
>
>>
>>-We are missing published version of some key kolla containers. Namely
>>openvswitch and the neutron-openvswitch-agent for starters but I'd also
>>like to have a Ceilometer agent and SNMP agent container as well so we
>>have feature parity with the non-docker compute role.
>
>Openvswitch works fantastically today and was our first Ansible deployment
>mechanism.  We later added linuxbridge support and plan nova-networking
>for l3.  We don¹t plan to port this work to config-internal config
>strategy, but as stated in our ansible-multi review, we won¹t turn away
>patches that implement this work in the config-internal model.
>
>Our ovs work is here:
>
>https://github.com/stackforge/kolla/tree/master/docker/common/openvswitch
>https://github.com/stackforge/kolla/tree/master/docker/common/neutron/neut
>r
>on-openvswitch-agent
>
>
>Ceilometer agent - agreed this is needed and high on my list for l3.  Not
>familiar with an SNMP agent but it doesn¹t sound too difficult to take on.
> If you can provide more details on what to configure, I¹ll get a
>blueprint filed and we can get this work prioritized for l3.  Our
>ceilometer container in general needs some love to be tackled in l3.
>
>It is likely you looked at the repo a couple weeks ago when this stuff
>wasn¹t fully baked.  I realize we are moving pretty fast so no harm on the
>lag :)
>
>Regards
>-steve
>
>>
>>Once we have solutions for the above I think we'll be very close to a
>>fully dockerized compute role with TripleO heat templates. From there
>>we can expand the idea to cover other roles within the tripleo-heat
>>-templates too.
>>
>>I'll stop there for now. Any ideas and thoughts appreciated.
>>
>>Dan
>>
>>-----
>>
>>[1] https://review.openstack.org/#/c/178840/ (Containerized TripleO
>>Overcloud.)
>>[2] https://etherpad.openstack.org/p/tripleo-docker-puppet
>>[3] http://git.openstack.org/cgit/openstack/heat
>>-templates/log/hot/software-config/heat-container-agent
>>[4] https://review.openstack.org/#/c/209505/  (Docker compute role
>>configured via Puppet)
>>
>>_________________________________________________________________________
>>_
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>_________________________________________________________________________
>>_
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>