Open Stack

On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco <flavio at redhat.com> wrote:
> On 14/07/17 08:08 -0700, Emilien Macchi wrote:
>>
>> On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco <flavio at redhat.com> wrote:
>>>
>>>
>>> Greetings,
>>>
>>> As some of you know, I've been working on the second phase of TripleO's
>>> containerization effort. This phase if about migrating the docker based
>>> deployment onto Kubernetes.
>>>
>>> These phase requires work on several areas: Kubernetes deployment,
>>> OpenStack
>>> deployment on Kubernetes, configuration management, etc. While I've been
>>> diving
>>> into all of these areas, this email is about the second point, OpenStack
>>> deployment on Kubernetes.
>>>
>>> There are several tools we could use for this task. kolla-kubernetes,
>>> openstack-helm, ansible roles, among others. I've looked into these tools
>>> and
>>> I've come to the conclusion that TripleO would be better of by having
>>> ansible
>>> roles that would allow for deploying OpenStack services on Kubernetes.
>>>
>>> The existing solutions in the OpenStack community require using Helm.
>>> While
>>> I
>>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>>> projects,
>>> I
>>> believe using any of them would add an extra layer of complexity to
>>> TripleO,
>>> which is something the team has been fighting for years years -
>>> especially
>>> now
>>> that the snowball is being chopped off.
>>>
>>> Adopting any of the existing projects in the OpenStack communty would
>>> require
>>> TripleO to also write the logic to manage those projects. For example, in
>>> the
>>> case of openstack-helm, the TripleO team would have to write either
>>> ansible
>>> roles or heat templates to manage - install, remove, upgrade - the charts
>>> (I'm
>>> happy to discuss this point further but I'm keepping it at a high-level
>>> on
>>> purpose for the sake of not writing a 10k-words-long email).
>>>
>>> James Slagle sent an email[0], a couple of days ago, to form TripleO
>>> plans
>>> around ansible. One take-away from this thread is that TripleO is
>>> adopting
>>> ansible more and more, which is great and it fits perfectly with the
>>> conclusion
>>> I reached.
>>>
>>> Now, what this work means is that we would have to write an ansible role
>>> for
>>> each service that will deploy the service on a Kubernetes cluster.
>>> Ideally
>>> these
>>> roles will also generate the configuration files (removing the need of
>>> puppet
>>> entirely) and they would manage the lifecycle. The roles would be
>>> isolated
>>> and
>>> this will reduce the need of TripleO Heat templates. Doing this would
>>> give
>>> TripleO full control on the deployment process too.
>>>
>>> In addition, we could also write Ansible Playbook Bundles to contain
>>> these
>>> roles
>>> and run them using the existing docker-cmd implementation that is coming
>>> out
>>> in
>>> Pike (you can find a PoC/example of this in this repo[1]).
>>>
>>> Now, I do realize the amount of work this implies and that this is my
>>> opinion/conclusion. I'm sending this email out to kick-off the discussion
>>> and
>>> gather thoughts and opinions from the rest of the community.
>>>
>>> Finally, what I really like about writing pure ansible roles is that
>>> ansible
>>> is
>>> a known, powerfull, tool that has been adopted by many operators already.
>>> It'll
>>> provide the flexibility needed and, if structured correctly, it'll allow
>>> for
>>> operators (and other teams) to just use the parts they need/want without
>>> depending on the full-stack. I like the idea of being able to separate
>>> concerns
>>> in the deployment workflow and the idea of making it simple for users of
>>> TripleO
>>> to do the same at runtime. Unfortunately, going down this road means that
>>> my
>>> hope of creating a field where we could collaborate even more with other
>>> deployment tools will be a bit limited but I'm confident the result would
>>> also
>>> be useful for others and that we all will benefit from it... My hopes
>>> might
>>> be a
>>> bit naive *shrugs*
>>
>>
>> Of course I'm biased since I've been (a little) involved in that work
>> but I like the idea of :
>>
>> - Moving forward with our containerization. docker-cmd will help us
>> for sure for this transition (I insist on the fact TripleO is a
>> product that you can upgrade and we try to make it smooth for our
>> operators), so we can't just trash everything and switch to a new
>> tool. I think the approach that we're taking is great and made of baby
>> steps where we try to solve different problems.
>> - Using more Ansible - the right way - when it makes sense : with the
>> TripleO containerization, we only use Puppet for Configuration
>> Management, managing a few resources but not for orchestration (or not
>> all the features that Puppet provide) and for Data Binding (Hiera). To
>> me, it doesn't make sense for us to keep investing much in Puppet
>> modules if we go k8s & Ansible. That said, see the next point.
>> - Having a transition path between TripleO with Puppet and TripleO
>> with apbs and have some sort of binding between previous hieradata
>> generated by TripleO & a similar data binding within Ansible playbooks
>> would help. I saw your PoC Flavio, I found it great and I think we
>> should make
>> https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/hiera.yaml
>> optional when running apbs, and allow to provide another format (more
>> Ansiblish) to let folks not using TripleO to use it. We also should
>> target this new format and switch service by service in TripleO to use
>> this new format, as long as apbs support both. I think that way we can
>> step by step migrate to use Ansible for configuration management.
>>
>> There are some things to figure out:
>> - We kind of found out solutions for OpenStack services - great - now
>> what do we do for services like MySQL, Apache, etc. Do we have
>> "standard" and "community-supported" apbs? Do we need to create some?
>
>
> The question should be whether we have community maintained roles that
> deploy
> third-party services on k8s. I'm sorry for nitpicking but I just want to
> make
> sure we all keep in mind that the APB wrap is optional (although convenient
> for
> us).
>
> The answer is not that I'm aware of. There are roles to deploy some of these
> services on baremetal.

*If* we're going to use Helm, I found the answer of my own question:
https://github.com/kubernetes/charts/tree/master/stable/mariadb
(though it's missing Galera support AFIK).

And some other services that we currently use in TripleO:
https://github.com/kubernetes/charts/tree/master/stable

I tried to find if whether or not kolla-kubernetes was using these
Helms, and unless I fail to find them I think they don't. I would be
curious to know the reason if that's the case. (Please correct me with
links).

>> - Where the apbs should live? IMO in OpenStack and IMO not in big tent
>> for now, under no umbrella.
>
>
> Again, roles!
>
> I feel like this question is forcing us to get a bit ahead of ourselves but,
> for
> the sake of discussion, I think they could just exist in the openstack org
> under
> no team for now. But the answer really depends on where the discussion about
> hosted repos goes.
>
> There are different ways we could structure this and I would be lying if I
> said
> I haven't thought about it. We can talk about this point when the right time
> comes. I don't see ourselves creating repos under openstack just yet.
>
>> - Since we use Puppet modules which don't only make configuration
>> management but also some orchestration (like creating keystone
>> endpoints, etc) - where should we put this logic? +1 for apbs using
>> clean Ansible code (and not bash templating).
>
>
> Yeah, this logic would go into the ansible roles.
>
>> - How we can help our vendors to whom we asked them to write Puppet
>> modules to deploy their software (Contrail, Nuage, etc). We still
>> might need some sort or "running puppet from ansible" for some
>> software we wouldn't have apbs as quickly as we would need.
>
>
> Yeah, this would be one of the hard problems to solve. Running puppet from
> ansible is possible and I know openstack-infra does that. This is one option
> and
> probably the one we would have to support in the transition period.
>
>> I hope I didn't divert the discussion but here's my feedback and food
>> for thoughts.
>
>
> Not at all, thanks for throwing all these questions out there. These are
> valid
> concerns and we'll have to address them. It's better to document them now.
>
>
> Flavio
>
> --
> @flaper87
> Flavio Percoco
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Emilien Macchi