Open Stack

Update:

The underlying Ansible to support derived parameters within a TripleO
deployment has merged [1]. We now only need to call and write some new
Ansible modules to do the derivation. Then we can merge the THT change
[2] and this feature's migration from Mistral will be complete. Until
then anyone who wants this feature needs to enable and use Mistral on
the undercloud.

For anyone who wants to write new Ansible modules for derived params,
now that the merge is done, if you build an undercloud from TripleO
master and deploy with -p with the new plan-environment [2], the
derived_parameters will be in the deployment plan and whatever that
parameter contains will be applied as usual. The merged [1] patch has
a clear indication of where [3] to call the new modules.

Molecule tests the feature and has mock data so Saravanan, Jaganathan,
and I should be able to develop and test [4] the new modules. I've
documented how I extracted the mock data from my real deployment
(based on Kevin's example) and shrank them in a personal repos [5]
just in case the NFV team finds that information useful for the NFV
modules and need to bring in more mock data. I'm personally planning
to start the HCI derive params Ansible module after the PTG.

Let's keep this thread updated during the Victoria cycle so that we
can get the feature completely migrated from Mistral.

Thanks,
  John

[1] https://review.opendev.org/#/c/719466
[2] https://review.opendev.org/#/c/714217
[3] https://opendev.org/openstack/tripleo-ansible/src/commit/f99dea3b508f345e96a0e27e0250523e826eadbb/tripleo_ansible/roles/tripleo_derived_parameters/tasks/main.yml#L237
[4] cd tripleo-ansible; ./scripts/run-local-test tripleo_derived_param
[5] https://github.com/fultonj/ussuri/tree/master/derive/data (I think
a personal repos is sufficient but let me know if you want me to put
it in the actual project; seems too ad hoc for TripleO itself to me)

On Tue, Apr 28, 2020 at 8:10 PM John Fulton <johfulto at redhat.com> wrote:
>
> On Fri, Mar 20, 2020 at 6:07 PM John Fulton <johfulto at redhat.com> wrote:
> >
> > On Thu, Mar 19, 2020 at 5:37 AM Saravanan KR <skramaja at redhat.com> wrote:
> > >
> > > On Thu, Mar 19, 2020 at 1:02 AM John Fulton <johfulto at redhat.com> wrote:
> > > >
> > > > On Sat, Mar 14, 2020 at 8:06 AM Rabi Mishra <ramishra at redhat.com> wrote:
> > > > >
> > > > > On Sat, Mar 14, 2020 at 2:10 AM John Fulton <johfulto at redhat.com> wrote:
> > > > >>
> > > > >> On Fri, Mar 13, 2020 at 3:27 PM Kevin Carter <kecarter at redhat.com> wrote:
> > > > >> >
> > > > >> > Hello stackers,
> > > > >> >
> > > > >> > In the pursuit to remove Mistral from the TripleO undercloud, we've discovered an old capability that we need to figure out how best to handle. Currently, we provide the ability for an end-user (operator / deployer) to pass in "N" Mistral workflows as part of a given deployment plan which is processed by python-tripleoclient at runtime [0][1]. From what we have documented, and what we can find within the code-base, we're not using this feature by default. That said, we do not remove something if it is valuable in the field without an adequate replacement. The ability to run arbitrary Mistral workflows at deployment time was first created in 2017 [2] and while present all this time, its documented [3] and intra-code-base uses are still limited to samples [4].
> > > > >>
> > > > >> As it stands now, we're on track to making Mistral inert this cycle and if our progress holds over the next couple of weeks the capability to run arbitrary Mistral workflows will be the only thing left within our codebase that relies on Mistral running on the Undercloud.
> > > > >>
> > > > >> >
> > > > >> > So the question is what do we do with functionality. Do we remove this ability out right, do we convert the example workflow [5] into a stand-alone Ansible playbook and change the workflow runner to an arbitrary playbook runner, or do we simply leave everything as-is and deprecate it to be removed within the next two releases?
> > > > >
> > > > >
> > > > > Yeah, as John mentioned, tripleo.derive_params.v1.derive_parameters workflow is surely being used for NFV (DPDK/SR-IOV) and HCI use cases and  can't be deprecated or dropped. Though we've a generic interface in tripleoclient to run any workflow in plan-environment, I have not seen it being used for anything other than the mentioned workflow.
> > > > >
> > > > > In the scope of 'mistral-ansible' work, we seem to  have two options.
> > > > >
> > > > > 1. Convert the workflow to ansible playbook 'as-is' i.e calculate and merge the derived parameters in plan-environment and as you've mentioned, change tripleoclient code to call any playbook in plan-environment.yaml and the parameters/vars.
> > > >
> > > > Nice idea. I hadn't thought of that.
> > > >
> > > > If there's a "hello world" example of this (which results in a THT
> > > > param in the deployment plan being set to "hello world"), then I could
> > > > try writing an ansible module to derive the HCI parameters and set
> > > > them in place of the "hello world".
> > > >
> > > I am fine with the approach, but the only concern is, we have plans to
> > > remove Heat in the coming cycles. One of inputs for the Mistral derive
> > > parameters is fetched from the heat stack. If we are going to retain
> > > it, then it has to be re-worked during the Heat removal. Mistral to
> > > ansible could be the first step towards it.
> >
> > Hey Saravanan,
> >
> > That works for me. I'm glad we were able to come up with a way to do this.
> >
> > Kevin put some patches together today that will help a lot on this.
> >
> > 1. tht: https://review.opendev.org/#/c/714217/
> > 2. tripleo-ansible: https://review.opendev.org/#/c/714232/
> > 3. trilpeoclient: https://review.opendev.org/#/c/714198/
> >
> > If I put these on my undercloud, then I think I can run:
> >
> > 'openstack overcloud deploy ... -p plan-environment-derived-params.yaml'
> >
> > as usual and then the updated tripleoclient and tht patch should
> > trigger the new tripleo-ansible playbook in place of the Mistral
> > workbook.
> >
> > I think I can then just update that tripleo-ansible patch to have it
> > include a new derive_params_hci role and add a new derive_params_hci
> > module where I'll stick code from the original Python prototype I did
> > for it. I'll probably just shell to `openstack baremetal introspection
> > data save ID` from ansible to get the Ironic data. I'll give it a try
> > next week and update this thread. Even if Heat is not in the flow, at
> > least the Ansible role and module can be reused.
> >
> > Note that it uses the new tripleo_plan_parameters_update module that
> > Rabi wrote so that should make it easier to deal with the deployment
> > plan itself (https://review.opendev.org/712604).
>
> Kevin and Rabi have made a lot of progress and with their unmerged
> patches [1] 'openstack overcloud deploy -p
> plan-environment-derived-params.yaml' has Ansible is running a
> playbook [2] with place holders for us to use derive params which
> looks like it will push the changes back to the deployment plan as
> discussed above.
>
> As far as I can tell 'openstack overcloud deploy -p
> plan-environment-derived-params.yaml' from master won't work the old
> way as Mistral isn't in the picture anymore when 'openstack overcloud
> deploy -p' is run (someone please correct me if I'm wrong). Thus,
> derive params are not going to work with the U release unless we
> finish the above. I don't think we should undo the progress. We're now
> in the RC so time is short. As long as we still ship the Mistral
> container on the undercloud in U, a deployer could have it derive the
> params in theory if the workbook is run manually and the resultant
> params are applied as overrides. Do we GA with that as a known issue
> with workaround and then circle back to fix the above and backport it?
> I don't think we should delay the release for it. I think we should
> instead push harder on getting the playbook and roles that Kevin
> started [2] (they could still land in U).
>
>   John
>
> [1] https://review.opendev.org/#/q/topic:mistral_to_ansible+(status:open+OR+status:merged)
> [2] https://review.opendev.org/#/c/719466/13/tripleo_ansible/roles/tripleo_derived_parameters/tasks/main.yml@170
>
> >
> >   John
> >
> > > Regards,
> > > Saravanan KR
> > >
> > > >   John
> > > >
> > > > > 2. Move the functionality further down the component chain in TripleO to have the required ansible host/group_vars set for them to be used by config-download playbooks/ansible/puppet.
> > > > >
> > > > > I guess option 1 would be easier within the timelines. I've done some preliminary work to move some of the functionality in relevant mistral actions to utils modules[1], so that they can be called from ansible modules without depending on mistral/mistral-lib and use those in a playbook that kinda replicate the tasks in the mistral workflow.
> > > > >
> > > > > Having said that, it would be good to know what the DFG:NFV folks think, as they are the original authors/maintainers of that workflow.
> > > > >
> > > > > <snip>
> > > > >
> > > > >> The Mistral based workflow took advantage of the deployment plan which
> > > > >> was stored in Swift on the undercloud. My understanding is that too is
> > > > >> going away.
> > > > >
> > > > >
> > > > > I'm not sure that would be in the scope of 'mstral-to-ansible' work. Dropping swift would probably be a bit more complex, as we use it to store templates, plan-environment, plan backups (possibly missing few more) etc  and would require significant design rework (may be possible when we get rid of heat in undercloud). In spite of heat using the templates from swift and merging environments on the client side, we've had already bumped heat's REST API json body size limit (max_json_body_size) on the undercloud to 4MB[2] from the default 1MB and sending all required templates as part of API request would not be a good idea from undercloud scalability pov.
> > > > >
> > > > > [1] https://review.opendev.org/#/c/709546/
> > > > > [2] https://github.com/openstack/tripleo-heat-templates/blob/master/environments/undercloud.yaml#L109
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Rabi Mishra
> > > > >
> > > >
> > >