On Sat, Sep 14, 2019 at 5:10 PM Alan Bishop <abishop@redhat.com> wrote:

On Fri, Sep 13, 2019 at 3:06 PM Emilien Macchi <emilien@redhat.com> wrote:

...

With our long-term goal to simplify TripleO and focus on what people actually deploy and how they operate their clouds, it appears that the Paunch CLI hasn't been a critical piece in our project and I propose that we deprecate it; create an Ansible module to call Paunch as a library only.

I've been playing with it a little today: https://review.opendev.org/#/c/682093/ https://review.opendev.org/#/c/682094/

Here is how you would call paunch: - name: Start containers for step {{ step }} paunch: config: "/var/lib/tripleo-config/hashed-container-startup-config-step_{{ step }}.json" config_id: "tripleo_step{{ step }}" action: apply container_cli: "{{ container_cli }}" managed_by: "tripleo-{{ tripleo_role_name }}"

A few benefits: - Deployment tasks in THT would call the new module instead of a shell command - More Pythonic and clean for Ansible, to interact with the actual task during the run - Removing some code in Paunch, make it easier to maintain for us

For now, the Ansible module only covers "paunch apply", we will probably cover "delete" and "cleanup" eventually.

The paunch cli's "print-cmd" action has been occasionally useful as a debug aid. Will this info be available through some other means?

I also rely on print-cmd and the other debug features. I sometimes use apply to reproduce issues but I suppose I could do without. -- -- James Slagle --

On Mon, Sep 16, 2019 at 4:22 AM Dave Holland <dh3@sanger.ac.uk> wrote:

We've found "paunch debug" useful in tracking down container issues that we've reported to RH and then fixed, e.g. when diagnosing a too-low file handle limit:

paunch debug --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_4.json --overrides '{ "ulimit": ["nofile=9999"] }' --container neutron_l3_agent --action run

Will there be a way to achieve this run-with-overrides functionality without the CLI?

Everything that will be "removed" or "replaced" in Paunch will be available elsewhere. Either in an Ansible module or still in Paunch itself. Our goal isn't to break our users but rather simplify our tooling. -- Emilien Macchi

On Mon, Sep 16, 2019 at 7:22 AM Rabi Mishra <ramishra@redhat.com> wrote:

Why not use ansible podman/docker modules (though I don't know how good they are atm) directly form ansible tasks?

Paunch is a tool for defining and running TripleO containers. Paunch consumes JSON files to configure the containers and the Paunch runs makes the abstraction between the stable configuration into the container api (compose or podman for now). There is quite a bunch of logic in Paunch that, imho would make take some time to convert to Ansible playbooks/modules, specially around resiliency. Not saying it's impossible, but I would rather be interested in having TripleO generating the container config, and the container tool consuming it and directly managing the containers without something like Paunch. This doesn't exist with Podman as far as I know. We have investigated the usage of Kubelet running on localhost, where TripleO would generate yaml files working with k8s API, it worked ok'ish for our containers but the major issue we encountered is that this solution isn't supported by Red Hat. So... it seems like we still need something like Paunch for now, and we can maybe investigate making the podman-ansible module more robust to sustain our needs in TripleO

Also, why deprecate the cli? As many others mentioned, lot of us use it for debugging.

Based on the answers so far, it's pretty clear we won't touch this command. As for the "paunch apply", we'll see, if the Ansible replacement works for everyone, then we might deprecate it in Paunch but not the debug command for sure. Thanks, -- Emilien Macchi

On Mon, Sep 16, 2019 at 11:47 AM Rabi Mishra <ramishra@redhat.com> wrote:

I'm not sure if podman as container tool would move in that direction, as it's meant to be a command line tool. If we really want to reduce the overhead of so many layers in TripleO and podman is the container tool for us (I'll ignore the k8s related discussions for the time being), I would think the logic of translating the JSON configs to podman calls should be be in ansible (we can even write a TripleO specific podman module).

I think we're both in strong agreement and say "let's convert paunch into ansible module". And make the module robust enough for our needs. Then we could replace paunch by calling the podman module directly. -- Emilien Macchi

On Mon, Sep 16, 2019 at 2:24 PM Sagi Shnaidman <sshnaidm@redhat.com> wrote:

wrt podman_container module, I have submitted podman_container module to ansible [1] that was reverted later, the main reason of revert was lack of idempotency. I planned to add it later, but ansible cores refused to wait. Maybe we can submit it to tripleo-ansible, finish idempotency and polish it, and then to submit to ansible upstream again. That way we'll have well tested with tripleo podman_container module. Meanwhile we can use paunch module, as it's easier to move from one ansible module to another.

Yes, it sounds like a plan. If you can patch: https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/ans... It would avoid duplication. Thanks, -- Emilien Macchi

This thread deserves an update: - tripleo-ansible has now a paunch module, calling openstack/paunch as a library. https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansi... And is called here for paunch apply: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/commo... In theory, we could deprecate "paunch apply" now as we don't need it anymore. I was working on porting "paunch cleanup" but it's still WIP. - I've been working on a new Ansible role which could totally replace Paunch, called "tripleo-container-manage", which has been enough for me to deploy an Undercloud: https://review.opendev.org/#/c/686196. It's being tested here: https://review.opendev.org/#/c/687651/ and as you can see the undercloud was successfully deployed without Paunch. Note that some container parameters haven't been ported and upgrade untested (this is a prototype). The second approach is a serious prototype I would like to continue further but before I would like some feedback. As for the feedback received in the previous answers, people would like to keep a "print-cmd" like, which makes total sense. I was thinking we could write a proper check mode for the podman_container module, which could output the podman commands that are run by the module. We could also extract the container management tasks to its own playbook so an operator who would usually run: $ paunch debug (...) --action print-cmd replaced by: $ ansible-playbook --check -i inventory.yaml containers.yaml A few benefits of this new role: - leverage ansible modules (we plan to upstream podman_container module) - could be easier to maintain and contribute (python vs ansible) - could potentially be faster. I want to investigate usage of async actions/polls in the role. Challenges: - no unit tests like in paunch, will need good testing with Molecule - we need to invest a lot in testing it, Paunch has a lot of edge cases that we carried over the cycles to manage containers. More feedback is very welcome and anyone interested to contribute please let me know. On Tue, Sep 17, 2019 at 5:03 AM Bogdan Dobrelya <bdobreli@redhat.com> wrote:

On 16.09.2019 18:07, Emilien Macchi wrote:

...

On Mon, Sep 16, 2019 at 11:47 AM Rabi Mishra <ramishra@redhat.com <mailto:ramishra@redhat.com>> wrote:

I'm not sure if podman as container tool would move in that direction, as it's meant to be a command line tool. If we really want to reduce the overhead of so many layers in TripleO and podman is the container tool for us (I'll ignore the k8s related discussions for the time being), I would think the logic of translating the JSON configs to podman calls should be be in ansible (we can even write a TripleO specific podman module).

I think we're both in strong agreement and say "let's convert paunch into ansible module".

I support the idea of calling paunch code as is from an ansible module. Although I'm strongly opposed against re-implementing the paunch code itself as ansible modules. That only brings maintenance burden (harder will be much to backport fixes into Queens and Train) and more place for potential regressions, without any functional improvements.

...

And make the module robust enough for our needs. Then we could replace paunch by calling the podman module directly. -- Emilien Macchi

-- Best regards, Bogdan Dobrelya, Irc #bogdando

-- Emilien Macchi

On Wed, Oct 9, 2019 at 7:05 PM Emilien Macchi <emilien@redhat.com> wrote:

This thread deserves an update:

- tripleo-ansible has now a paunch module, calling openstack/paunch as a library. https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansi...

And is called here for paunch apply: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/commo...

In theory, we could deprecate "paunch apply" now as we don't need it anymore. I was working on porting "paunch cleanup" but it's still WIP.

- I've been working on a new Ansible role which could totally replace Paunch, called "tripleo-container-manage", which has been enough for me to deploy an Undercloud: https://review.opendev.org/#/c/686196. It's being tested here: https://review.opendev.org/#/c/687651/ and as you can see the undercloud was successfully deployed without Paunch. Note that some container parameters haven't been ported and upgrade untested (this is a prototype).

The second approach is a serious prototype I would like to continue further but before I would like some feedback. As for the feedback received in the previous answers, people would like to keep a "print-cmd" like, which makes total sense. I was thinking we could write a proper check mode for the podman_container module, which could output the podman commands that are run by the module. We could also extract the container management tasks to its own playbook so an operator who would usually run: $ paunch debug (...) --action print-cmd

replaced by: $ ansible-playbook --check -i inventory.yaml containers.yaml

A few benefits of this new role: - leverage ansible modules (we plan to upstream podman_container module) - could be easier to maintain and contribute (python vs ansible) - could potentially be faster. I want to investigate usage of async actions/polls in the role.

Challenges: - no unit tests like in paunch, will need good testing with Molecule - we need to invest a lot in testing it, Paunch has a lot of edge cases that we carried over the cycles to manage containers.

More feedback is very welcome and anyone interested to contribute please let me know.

Nice work! I like the approach with the new ansible role. I do think there will be a balance between what makes sense to keep in a python module vs an ansible task. If/then branching logic and conditional tasks based on previous results is of course all possible with ansible tasks, but it starts to become complex and difficult to manage. A higher level language (python) is much better at that. Personally, I prefer to view ansible as just an execution engine and would look to keep the actual application and business logic in proper reusable/testable code modules (python). Finding that right balance is likely something we can figure out in review feedback, ad-hoc discussions, etc. An idea for a future improvement I would like to see as we move in this direction is to switch from reading the container startup configs from a single file per step (/var/lib/tripleo-config/container-startup-config-step_{{ step }}.json), to using a directory per step instead. It would look something like: /var/lib/tripleo-config/container-startup-config/step1 /var/lib/tripleo-config/container-startup-config/step1/keystone-init-tasks.json /var/lib/tripleo-config/container-startup-config/step1/pacemaker-init-tasks.json etc. That way each service template can be converted to a proper ansible role in tripleo-ansible that just drops its config into the right directory on the managed node. When the tripleo-container-manage role is then executed, it will operate on those files. This would also make it much more clear what container caused a failure, since we could log the results individually instead of just getting back the union of all logs per step. I think you're patches already address this to some degree since you are looping over the contents of the single file. The other feedback I would offer is perhaps continue to think about keeping the container implementation pluggable in some fashion. Right now you have a tasks/podman.yaml. What might it look like if we wanted to have a tasks/kubernetes.yaml in the future, and how would that be enabled? Thanks -- -- James Slagle --

Hey folks, +1 to the direction we're going with this. Like Emilien said, the skies the limit when using a flexible automation framework like Ansible. We're definitely going to need Molecule tests for the role and unit/integration tests for the podman_container module itself. I left a comment in the podman_container feature request <https://github.com/ansible/ansible/issues/46362> in Ansible to let the broader community know that we're working towards stabilizing that module. Hopefully that will get more contributors to help us fast track upstreaming it. It doesn't seem like Paunch is really used outside of TripleO so having it in Ansible, which has wider adoption, seems really ideal. As for backports, I think it's fair to say that Paunch for the most part "just works." When it does break it's a pain to fix. Which is even more reason to move away from it. Sincerely, Luke Short, RHCE Software Engineer, OpenStack Deployment Framework Red Hat, Inc. On Fri, Oct 11, 2019 at 11:13 AM Emilien Macchi <emilien@redhat.com> wrote:

On Fri, Oct 11, 2019 at 10:55 AM James Slagle <james.slagle@gmail.com> wrote: [snip]

...

Nice work! I like the approach with the new ansible role.

I do think there will be a balance between what makes sense to keep in a python module vs an ansible task. If/then branching logic and conditional tasks based on previous results is of course all possible with ansible tasks, but it starts to become complex and difficult to manage. A higher level language (python) is much better at that. Personally, I prefer to view ansible as just an execution engine and would look to keep the actual application and business logic in proper reusable/testable code modules (python). Finding that right balance is likely something we can figure out in review feedback, ad-hoc discussions, etc.

Ack & agreed on my side.

An idea for a future improvement I would like to see as we move in

...

this direction is to switch from reading the container startup configs from a single file per step (/var/lib/tripleo-config/container-startup-config-step_{{ step }}.json), to using a directory per step instead. It would look something like:

/var/lib/tripleo-config/container-startup-config/step1

/var/lib/tripleo-config/container-startup-config/step1/keystone-init-tasks.json

/var/lib/tripleo-config/container-startup-config/step1/pacemaker-init-tasks.json etc.

That way each service template can be converted to a proper ansible role in tripleo-ansible that just drops its config into the right directory on the managed node. When the tripleo-container-manage role is then executed, it will operate on those files. This would also make it much more clear what container caused a failure, since we could log the results individually instead of just getting back the union of all logs per step. I think you're patches already address this to some degree since you are looping over the contents of the single file.

This is an excellent idea. One of the feedback I've got from the Upgrade folks is the need to be able to easily upgrade one service, and the current structure doesn't easily allow it. Your proposal is I think exactly addressing it; and indeed it'll help when migrating container config into their individual roles in tripleo-ansible. I'll add that to the backlog.

The other feedback I would offer is perhaps continue to think about

...

keeping the container implementation pluggable in some fashion. Right now you have a tasks/podman.yaml. What might it look like if we wanted to have a tasks/kubernetes.yaml in the future, and how would that be enabled?

Yes, that's what I had in mind when starting the role. The podman.yaml is for Podman logic. We will probably have docker.yaml if we want to support Docker for FFU from Queens to Train. And we can easily add a playbook "kubernetes.yaml" which will read the container config data, generate k8s YAML and then consume it via https://docs.ansible.com/ansible/latest/modules/k8s_module.html . Really there is no limit if we can make it really pluggable.

Thanks for the input and the great feedback, -- Emilien Macchi