[openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes
Fox, Kevin M
Kevin.Fox at pnnl.gov
Mon Jul 17 23:40:01 UTC 2017
We do support some upstream charts but we started mariadb/rabbit before some of the upstream charts were written, so we duplicate a little bit of functionality at the moment. You can mix and match though. If an upstream chart doesn't work with kolla-kubernetes, I consider that a bug we should fix. Likewise, you should be able to run noncontainerized stuff mixed in too. If it doesn't work, its likewise a bug. You should be able to run kolla-kubernetes with a baremetal db.
Some known working stuff: prometheus/grafana upstream charts start collecting data from the containers as soon as they are launched.
I have also tested a bit with the upstream fluent-bit chart and have a ps in the works to make it work much better.
From: Emilien Macchi [emilien at redhat.com]
Sent: Monday, July 17, 2017 10:13 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes
On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco <flavio at redhat.com> wrote:
> On 14/07/17 08:08 -0700, Emilien Macchi wrote:
>> On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco <flavio at redhat.com> wrote:
>>> As some of you know, I've been working on the second phase of TripleO's
>>> containerization effort. This phase if about migrating the docker based
>>> deployment onto Kubernetes.
>>> These phase requires work on several areas: Kubernetes deployment,
>>> deployment on Kubernetes, configuration management, etc. While I've been
>>> into all of these areas, this email is about the second point, OpenStack
>>> deployment on Kubernetes.
>>> There are several tools we could use for this task. kolla-kubernetes,
>>> openstack-helm, ansible roles, among others. I've looked into these tools
>>> I've come to the conclusion that TripleO would be better of by having
>>> roles that would allow for deploying OpenStack services on Kubernetes.
>>> The existing solutions in the OpenStack community require using Helm.
>>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>>> believe using any of them would add an extra layer of complexity to
>>> which is something the team has been fighting for years years -
>>> that the snowball is being chopped off.
>>> Adopting any of the existing projects in the OpenStack communty would
>>> TripleO to also write the logic to manage those projects. For example, in
>>> case of openstack-helm, the TripleO team would have to write either
>>> roles or heat templates to manage - install, remove, upgrade - the charts
>>> happy to discuss this point further but I'm keepping it at a high-level
>>> purpose for the sake of not writing a 10k-words-long email).
>>> James Slagle sent an email, a couple of days ago, to form TripleO
>>> around ansible. One take-away from this thread is that TripleO is
>>> ansible more and more, which is great and it fits perfectly with the
>>> I reached.
>>> Now, what this work means is that we would have to write an ansible role
>>> each service that will deploy the service on a Kubernetes cluster.
>>> roles will also generate the configuration files (removing the need of
>>> entirely) and they would manage the lifecycle. The roles would be
>>> this will reduce the need of TripleO Heat templates. Doing this would
>>> TripleO full control on the deployment process too.
>>> In addition, we could also write Ansible Playbook Bundles to contain
>>> and run them using the existing docker-cmd implementation that is coming
>>> Pike (you can find a PoC/example of this in this repo).
>>> Now, I do realize the amount of work this implies and that this is my
>>> opinion/conclusion. I'm sending this email out to kick-off the discussion
>>> gather thoughts and opinions from the rest of the community.
>>> Finally, what I really like about writing pure ansible roles is that
>>> a known, powerfull, tool that has been adopted by many operators already.
>>> provide the flexibility needed and, if structured correctly, it'll allow
>>> operators (and other teams) to just use the parts they need/want without
>>> depending on the full-stack. I like the idea of being able to separate
>>> in the deployment workflow and the idea of making it simple for users of
>>> to do the same at runtime. Unfortunately, going down this road means that
>>> hope of creating a field where we could collaborate even more with other
>>> deployment tools will be a bit limited but I'm confident the result would
>>> be useful for others and that we all will benefit from it... My hopes
>>> be a
>>> bit naive *shrugs*
>> Of course I'm biased since I've been (a little) involved in that work
>> but I like the idea of :
>> - Moving forward with our containerization. docker-cmd will help us
>> for sure for this transition (I insist on the fact TripleO is a
>> product that you can upgrade and we try to make it smooth for our
>> operators), so we can't just trash everything and switch to a new
>> tool. I think the approach that we're taking is great and made of baby
>> steps where we try to solve different problems.
>> - Using more Ansible - the right way - when it makes sense : with the
>> TripleO containerization, we only use Puppet for Configuration
>> Management, managing a few resources but not for orchestration (or not
>> all the features that Puppet provide) and for Data Binding (Hiera). To
>> me, it doesn't make sense for us to keep investing much in Puppet
>> modules if we go k8s & Ansible. That said, see the next point.
>> - Having a transition path between TripleO with Puppet and TripleO
>> with apbs and have some sort of binding between previous hieradata
>> generated by TripleO & a similar data binding within Ansible playbooks
>> would help. I saw your PoC Flavio, I found it great and I think we
>> should make
>> optional when running apbs, and allow to provide another format (more
>> Ansiblish) to let folks not using TripleO to use it. We also should
>> target this new format and switch service by service in TripleO to use
>> this new format, as long as apbs support both. I think that way we can
>> step by step migrate to use Ansible for configuration management.
>> There are some things to figure out:
>> - We kind of found out solutions for OpenStack services - great - now
>> what do we do for services like MySQL, Apache, etc. Do we have
>> "standard" and "community-supported" apbs? Do we need to create some?
> The question should be whether we have community maintained roles that
> third-party services on k8s. I'm sorry for nitpicking but I just want to
> sure we all keep in mind that the APB wrap is optional (although convenient
> The answer is not that I'm aware of. There are roles to deploy some of these
> services on baremetal.
*If* we're going to use Helm, I found the answer of my own question:
(though it's missing Galera support AFIK).
And some other services that we currently use in TripleO:
I tried to find if whether or not kolla-kubernetes was using these
Helms, and unless I fail to find them I think they don't. I would be
curious to know the reason if that's the case. (Please correct me with
>> - Where the apbs should live? IMO in OpenStack and IMO not in big tent
>> for now, under no umbrella.
> Again, roles!
> I feel like this question is forcing us to get a bit ahead of ourselves but,
> the sake of discussion, I think they could just exist in the openstack org
> no team for now. But the answer really depends on where the discussion about
> hosted repos goes.
> There are different ways we could structure this and I would be lying if I
> I haven't thought about it. We can talk about this point when the right time
> comes. I don't see ourselves creating repos under openstack just yet.
>> - Since we use Puppet modules which don't only make configuration
>> management but also some orchestration (like creating keystone
>> endpoints, etc) - where should we put this logic? +1 for apbs using
>> clean Ansible code (and not bash templating).
> Yeah, this logic would go into the ansible roles.
>> - How we can help our vendors to whom we asked them to write Puppet
>> modules to deploy their software (Contrail, Nuage, etc). We still
>> might need some sort or "running puppet from ansible" for some
>> software we wouldn't have apbs as quickly as we would need.
> Yeah, this would be one of the hard problems to solve. Running puppet from
> ansible is possible and I know openstack-infra does that. This is one option
> probably the one we would have to support in the transition period.
>> I hope I didn't divert the discussion but here's my feedback and food
>> for thoughts.
> Not at all, thanks for throwing all these questions out there. These are
> concerns and we'll have to address them. It's better to document them now.
> Flavio Percoco
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
More information about the OpenStack-dev