Open Stack

> I think this at some point might be single biggest benefit of using
> helm on the long run - leverage infrastructure charts that aren't
> openstack-centric. Things like etcd are already written and
> potentially we can just help support them.

I think the tools that are being discussion here are both very good
(helm & ansible), but I have a slightly different opinion about how
Helm should be used.

Helm is a *package manager*. It's scope is for simple applications
that need to bundle resources.  It's great at saving me time on doing
simple recipes like: kubectl create -f <my_pv> and kubectl create -f
<my_pod> over and over again. But, beyond a single app with a few
resources, Helm is going to struggle on it's own. Meaning, either Helm
would have to change or another tool would have to fill the gaps.

If Helm wants to change, it becomes less differentiated from what
Ansible already does. It's niche as a simple app package manager will
evaporate and Ansible already owns the orchestration space. Therefore,
I think long term Helm as an orchestration tool doesn't make sense
because it's limited to Kubernetes and Ansible adoption is wide
spread.

That doesn't mean that Helm is useless.  In fact, I think the Helm
charts are great when used as simple standalone recipes. However, for
a complex app like OpenStack, I think you need something like Ansible
to provide the orchestration. Underneath, you can use whatever you
want to create the Kubernetes resources. In the end, the difference
will be `helm create mariadb` vs `kubectl create -f mariadb-pod.yaml`.
Both solutions will work, but the Helm work seems to be much farther
along.

One other thing to mention. Maybe folks can speed up writing these
playbooks by using kolla-ansible's playbooks as a shell. Here's an
example: [1] Take lines 1-16 and replace it with helm install mariadb
or
kubectl create -f mariabd-pod.yaml and set inventory to localhost.
Just a thought.

There may be some other playbooks out there I don' know about that you
can use, but that could at least get some of the collaboration started
so folks don't have to start from scratch.

[1] - https://github.com/openstack/kolla-ansible/blob/afdd11b9a22ecca70962a4637d89ad50b7ded2e5/ansible/roles/mariadb/tasks/start.yml#L1-L16

Sincerely,
Ryan

On Mon, Jul 17, 2017 at 1:37 PM, Michał Jastrzębski <inc007 at gmail.com> wrote:
> On 17 July 2017 at 10:13, Emilien Macchi <emilien at redhat.com> wrote:
>> On Mon, Jul 17, 2017 at 5:32 AM, Flavio Percoco <flavio at redhat.com> wrote:
>>> On 14/07/17 08:08 -0700, Emilien Macchi wrote:
>>>>
>>>> On Fri, Jul 14, 2017 at 2:17 AM, Flavio Percoco <flavio at redhat.com> wrote:
>>>>>
>>>>>
>>>>> Greetings,
>>>>>
>>>>> As some of you know, I've been working on the second phase of TripleO's
>>>>> containerization effort. This phase if about migrating the docker based
>>>>> deployment onto Kubernetes.
>>>>>
>>>>> These phase requires work on several areas: Kubernetes deployment,
>>>>> OpenStack
>>>>> deployment on Kubernetes, configuration management, etc. While I've been
>>>>> diving
>>>>> into all of these areas, this email is about the second point, OpenStack
>>>>> deployment on Kubernetes.
>>>>>
>>>>> There are several tools we could use for this task. kolla-kubernetes,
>>>>> openstack-helm, ansible roles, among others. I've looked into these tools
>>>>> and
>>>>> I've come to the conclusion that TripleO would be better of by having
>>>>> ansible
>>>>> roles that would allow for deploying OpenStack services on Kubernetes.
>>>>>
>>>>> The existing solutions in the OpenStack community require using Helm.
>>>>> While
>>>>> I
>>>>> like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>>>>> projects,
>>>>> I
>>>>> believe using any of them would add an extra layer of complexity to
>>>>> TripleO,
>>>>> which is something the team has been fighting for years years -
>>>>> especially
>>>>> now
>>>>> that the snowball is being chopped off.
>>>>>
>>>>> Adopting any of the existing projects in the OpenStack communty would
>>>>> require
>>>>> TripleO to also write the logic to manage those projects. For example, in
>>>>> the
>>>>> case of openstack-helm, the TripleO team would have to write either
>>>>> ansible
>>>>> roles or heat templates to manage - install, remove, upgrade - the charts
>>>>> (I'm
>>>>> happy to discuss this point further but I'm keepping it at a high-level
>>>>> on
>>>>> purpose for the sake of not writing a 10k-words-long email).
>>>>>
>>>>> James Slagle sent an email[0], a couple of days ago, to form TripleO
>>>>> plans
>>>>> around ansible. One take-away from this thread is that TripleO is
>>>>> adopting
>>>>> ansible more and more, which is great and it fits perfectly with the
>>>>> conclusion
>>>>> I reached.
>>>>>
>>>>> Now, what this work means is that we would have to write an ansible role
>>>>> for
>>>>> each service that will deploy the service on a Kubernetes cluster.
>>>>> Ideally
>>>>> these
>>>>> roles will also generate the configuration files (removing the need of
>>>>> puppet
>>>>> entirely) and they would manage the lifecycle. The roles would be
>>>>> isolated
>>>>> and
>>>>> this will reduce the need of TripleO Heat templates. Doing this would
>>>>> give
>>>>> TripleO full control on the deployment process too.
>>>>>
>>>>> In addition, we could also write Ansible Playbook Bundles to contain
>>>>> these
>>>>> roles
>>>>> and run them using the existing docker-cmd implementation that is coming
>>>>> out
>>>>> in
>>>>> Pike (you can find a PoC/example of this in this repo[1]).
>>>>>
>>>>> Now, I do realize the amount of work this implies and that this is my
>>>>> opinion/conclusion. I'm sending this email out to kick-off the discussion
>>>>> and
>>>>> gather thoughts and opinions from the rest of the community.
>>>>>
>>>>> Finally, what I really like about writing pure ansible roles is that
>>>>> ansible
>>>>> is
>>>>> a known, powerfull, tool that has been adopted by many operators already.
>>>>> It'll
>>>>> provide the flexibility needed and, if structured correctly, it'll allow
>>>>> for
>>>>> operators (and other teams) to just use the parts they need/want without
>>>>> depending on the full-stack. I like the idea of being able to separate
>>>>> concerns
>>>>> in the deployment workflow and the idea of making it simple for users of
>>>>> TripleO
>>>>> to do the same at runtime. Unfortunately, going down this road means that
>>>>> my
>>>>> hope of creating a field where we could collaborate even more with other
>>>>> deployment tools will be a bit limited but I'm confident the result would
>>>>> also
>>>>> be useful for others and that we all will benefit from it... My hopes
>>>>> might
>>>>> be a
>>>>> bit naive *shrugs*
>>>>
>>>>
>>>> Of course I'm biased since I've been (a little) involved in that work
>>>> but I like the idea of :
>>>>
>>>> - Moving forward with our containerization. docker-cmd will help us
>>>> for sure for this transition (I insist on the fact TripleO is a
>>>> product that you can upgrade and we try to make it smooth for our
>>>> operators), so we can't just trash everything and switch to a new
>>>> tool. I think the approach that we're taking is great and made of baby
>>>> steps where we try to solve different problems.
>>>> - Using more Ansible - the right way - when it makes sense : with the
>>>> TripleO containerization, we only use Puppet for Configuration
>>>> Management, managing a few resources but not for orchestration (or not
>>>> all the features that Puppet provide) and for Data Binding (Hiera). To
>>>> me, it doesn't make sense for us to keep investing much in Puppet
>>>> modules if we go k8s & Ansible. That said, see the next point.
>>>> - Having a transition path between TripleO with Puppet and TripleO
>>>> with apbs and have some sort of binding between previous hieradata
>>>> generated by TripleO & a similar data binding within Ansible playbooks
>>>> would help. I saw your PoC Flavio, I found it great and I think we
>>>> should make
>>>> https://github.com/tripleo-apb/ansible-role-k8s-keystone/blob/331f405bd3f7ad346d99e964538b5b27447a0ebf/provision-keystone-apb/tasks/hiera.yaml
>>>> optional when running apbs, and allow to provide another format (more
>>>> Ansiblish) to let folks not using TripleO to use it. We also should
>>>> target this new format and switch service by service in TripleO to use
>>>> this new format, as long as apbs support both. I think that way we can
>>>> step by step migrate to use Ansible for configuration management.
>>>>
>>>> There are some things to figure out:
>>>> - We kind of found out solutions for OpenStack services - great - now
>>>> what do we do for services like MySQL, Apache, etc. Do we have
>>>> "standard" and "community-supported" apbs? Do we need to create some?
>>>
>>>
>>> The question should be whether we have community maintained roles that
>>> deploy
>>> third-party services on k8s. I'm sorry for nitpicking but I just want to
>>> make
>>> sure we all keep in mind that the APB wrap is optional (although convenient
>>> for
>>> us).
>>>
>>> The answer is not that I'm aware of. There are roles to deploy some of these
>>> services on baremetal.
>>
>> *If* we're going to use Helm, I found the answer of my own question:
>> https://github.com/kubernetes/charts/tree/master/stable/mariadb
>> (though it's missing Galera support AFIK).
>>
>> And some other services that we currently use in TripleO:
>> https://github.com/kubernetes/charts/tree/master/stable
>>
>> I tried to find if whether or not kolla-kubernetes was using these
>> Helms, and unless I fail to find them I think they don't. I would be
>> curious to know the reason if that's the case. (Please correct me with
>> links).
>
> We don't as of today. Reason is mostly because we had mariadb written
> for k8s before we started using helm and since we were porting
> everything...we just ported. That being said our mariadb is still not
> galeraised and it needs work to get there. If we could get high
> quality galera cluster in regular charts, I, for one, would be happy
> to start using it instead of homegrown. As you said either are not
> clustered so it's missing piece. I think that would be good approach
> to start supporting things like galera in native helm because there is
> value in it outside of openstack as well (sooo, you want your galera
> on OpenShift? You're in luck! Just run helm install ourrepo/galera).
>
> I think this at some point might be single biggest benefit of using
> helm on the long run - leverage infrastructure charts that aren't
> openstack-centric. Things like etcd are already written and
> potentially we can just help support them.
>
>>>> - Where the apbs should live? IMO in OpenStack and IMO not in big tent
>>>> for now, under no umbrella.
>>>
>>>
>>> Again, roles!
>>>
>>> I feel like this question is forcing us to get a bit ahead of ourselves but,
>>> for
>>> the sake of discussion, I think they could just exist in the openstack org
>>> under
>>> no team for now. But the answer really depends on where the discussion about
>>> hosted repos goes.
>>>
>>> There are different ways we could structure this and I would be lying if I
>>> said
>>> I haven't thought about it. We can talk about this point when the right time
>>> comes. I don't see ourselves creating repos under openstack just yet.
>>>
>>>> - Since we use Puppet modules which don't only make configuration
>>>> management but also some orchestration (like creating keystone
>>>> endpoints, etc) - where should we put this logic? +1 for apbs using
>>>> clean Ansible code (and not bash templating).
>>>
>>>
>>> Yeah, this logic would go into the ansible roles.
>>>
>>>> - How we can help our vendors to whom we asked them to write Puppet
>>>> modules to deploy their software (Contrail, Nuage, etc). We still
>>>> might need some sort or "running puppet from ansible" for some
>>>> software we wouldn't have apbs as quickly as we would need.
>>>
>>>
>>> Yeah, this would be one of the hard problems to solve. Running puppet from
>>> ansible is possible and I know openstack-infra does that. This is one option
>>> and
>>> probably the one we would have to support in the transition period.
>>>
>>>> I hope I didn't divert the discussion but here's my feedback and food
>>>> for thoughts.
>>>
>>>
>>> Not at all, thanks for throwing all these questions out there. These are
>>> valid
>>> concerns and we'll have to address them. It's better to document them now.
>>>
>>>
>>> Flavio
>>>
>>> --
>>> @flaper87
>>> Flavio Percoco
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Emilien Macchi
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev