Open Stack

On Sat, May 23, 2020 at 8:37 PM John Fulton <johfulto at redhat.com> wrote:

> Update:
>
> The underlying Ansible to support derived parameters within a TripleO
> deployment has merged [1]. We now only need to call and write some new
> Ansible modules to do the derivation. Then we can merge the THT change
> [2] and this feature's migration from Mistral will be complete. Until
> then anyone who wants this feature needs to enable and use Mistral on
> the undercloud.
>
>
I would have liked the feature to have been migrated completely (rather
than placeholders)
before disabling mistral on the undercloud by default. If the NFV/HCI teams
don't have any
issues with the current plan, then it's probably ok:)

At Least we should document it somewhere so that it's not missed in upgrade.


> For anyone who wants to write new Ansible modules for derived params,
> now that the merge is done, if you build an undercloud from TripleO
> master and deploy with -p with the new plan-environment [2], the
> derived_parameters will be in the deployment plan and whatever that
> parameter contains will be applied as usual. The merged [1] patch has
> a clear indication of where [3] to call the new modules.
>
> Molecule tests the feature and has mock data so Saravanan, Jaganathan,
> and I should be able to develop and test [4] the new modules. I've
> documented how I extracted the mock data from my real deployment
> (based on Kevin's example) and shrank them in a personal repos [5]
> just in case the NFV team finds that information useful for the NFV
> modules and need to bring in more mock data. I'm personally planning
> to start the HCI derive params Ansible module after the PTG.
>
> Let's keep this thread updated during the Victoria cycle so that we
> can get the feature completely migrated from Mistral.
>
> Thanks,
>   John
>
> [1] https://review.opendev.org/#/c/719466
> [2] https://review.opendev.org/#/c/714217
> [3]
> https://opendev.org/openstack/tripleo-ansible/src/commit/f99dea3b508f345e96a0e27e0250523e826eadbb/tripleo_ansible/roles/tripleo_derived_parameters/tasks/main.yml#L237
> [4] cd tripleo-ansible; ./scripts/run-local-test tripleo_derived_param
> [5] https://github.com/fultonj/ussuri/tree/master/derive/data (I think
> a personal repos is sufficient but let me know if you want me to put
> it in the actual project; seems too ad hoc for TripleO itself to me)
>
> On Tue, Apr 28, 2020 at 8:10 PM John Fulton <johfulto at redhat.com> wrote:
> >
> > On Fri, Mar 20, 2020 at 6:07 PM John Fulton <johfulto at redhat.com> wrote:
> > >
> > > On Thu, Mar 19, 2020 at 5:37 AM Saravanan KR <skramaja at redhat.com>
> wrote:
> > > >
> > > > On Thu, Mar 19, 2020 at 1:02 AM John Fulton <johfulto at redhat.com>
> wrote:
> > > > >
> > > > > On Sat, Mar 14, 2020 at 8:06 AM Rabi Mishra <ramishra at redhat.com>
> wrote:
> > > > > >
> > > > > > On Sat, Mar 14, 2020 at 2:10 AM John Fulton <johfulto at redhat.com>
> wrote:
> > > > > >>
> > > > > >> On Fri, Mar 13, 2020 at 3:27 PM Kevin Carter <
> kecarter at redhat.com> wrote:
> > > > > >> >
> > > > > >> > Hello stackers,
> > > > > >> >
> > > > > >> > In the pursuit to remove Mistral from the TripleO undercloud,
> we've discovered an old capability that we need to figure out how best to
> handle. Currently, we provide the ability for an end-user (operator /
> deployer) to pass in "N" Mistral workflows as part of a given deployment
> plan which is processed by python-tripleoclient at runtime [0][1]. From
> what we have documented, and what we can find within the code-base, we're
> not using this feature by default. That said, we do not remove something if
> it is valuable in the field without an adequate replacement. The ability to
> run arbitrary Mistral workflows at deployment time was first created in
> 2017 [2] and while present all this time, its documented [3] and
> intra-code-base uses are still limited to samples [4].
> > > > > >>
> > > > > >> As it stands now, we're on track to making Mistral inert this
> cycle and if our progress holds over the next couple of weeks the
> capability to run arbitrary Mistral workflows will be the only thing left
> within our codebase that relies on Mistral running on the Undercloud.
> > > > > >>
> > > > > >> >
> > > > > >> > So the question is what do we do with functionality. Do we
> remove this ability out right, do we convert the example workflow [5] into
> a stand-alone Ansible playbook and change the workflow runner to an
> arbitrary playbook runner, or do we simply leave everything as-is and
> deprecate it to be removed within the next two releases?
> > > > > >
> > > > > >
> > > > > > Yeah, as John mentioned,
> tripleo.derive_params.v1.derive_parameters workflow is surely being used
> for NFV (DPDK/SR-IOV) and HCI use cases and  can't be deprecated or
> dropped. Though we've a generic interface in tripleoclient to run any
> workflow in plan-environment, I have not seen it being used for anything
> other than the mentioned workflow.
> > > > > >
> > > > > > In the scope of 'mistral-ansible' work, we seem to  have two
> options.
> > > > > >
> > > > > > 1. Convert the workflow to ansible playbook 'as-is' i.e
> calculate and merge the derived parameters in plan-environment and as
> you've mentioned, change tripleoclient code to call any playbook in
> plan-environment.yaml and the parameters/vars.
> > > > >
> > > > > Nice idea. I hadn't thought of that.
> > > > >
> > > > > If there's a "hello world" example of this (which results in a THT
> > > > > param in the deployment plan being set to "hello world"), then I
> could
> > > > > try writing an ansible module to derive the HCI parameters and set
> > > > > them in place of the "hello world".
> > > > >
> > > > I am fine with the approach, but the only concern is, we have plans
> to
> > > > remove Heat in the coming cycles. One of inputs for the Mistral
> derive
> > > > parameters is fetched from the heat stack. If we are going to retain
> > > > it, then it has to be re-worked during the Heat removal. Mistral to
> > > > ansible could be the first step towards it.
> > >
> > > Hey Saravanan,
> > >
> > > That works for me. I'm glad we were able to come up with a way to do
> this.
> > >
> > > Kevin put some patches together today that will help a lot on this.
> > >
> > > 1. tht: https://review.opendev.org/#/c/714217/
> > > 2. tripleo-ansible: https://review.opendev.org/#/c/714232/
> > > 3. trilpeoclient: https://review.opendev.org/#/c/714198/
> > >
> > > If I put these on my undercloud, then I think I can run:
> > >
> > > 'openstack overcloud deploy ... -p
> plan-environment-derived-params.yaml'
> > >
> > > as usual and then the updated tripleoclient and tht patch should
> > > trigger the new tripleo-ansible playbook in place of the Mistral
> > > workbook.
> > >
> > > I think I can then just update that tripleo-ansible patch to have it
> > > include a new derive_params_hci role and add a new derive_params_hci
> > > module where I'll stick code from the original Python prototype I did
> > > for it. I'll probably just shell to `openstack baremetal introspection
> > > data save ID` from ansible to get the Ironic data. I'll give it a try
> > > next week and update this thread. Even if Heat is not in the flow, at
> > > least the Ansible role and module can be reused.
> > >
> > > Note that it uses the new tripleo_plan_parameters_update module that
> > > Rabi wrote so that should make it easier to deal with the deployment
> > > plan itself (https://review.opendev.org/712604).
> >
> > Kevin and Rabi have made a lot of progress and with their unmerged
> > patches [1] 'openstack overcloud deploy -p
> > plan-environment-derived-params.yaml' has Ansible is running a
> > playbook [2] with place holders for us to use derive params which
> > looks like it will push the changes back to the deployment plan as
> > discussed above.
> >
> > As far as I can tell 'openstack overcloud deploy -p
> > plan-environment-derived-params.yaml' from master won't work the old
> > way as Mistral isn't in the picture anymore when 'openstack overcloud
> > deploy -p' is run (someone please correct me if I'm wrong). Thus,
> > derive params are not going to work with the U release unless we
> > finish the above. I don't think we should undo the progress. We're now
> > in the RC so time is short. As long as we still ship the Mistral
> > container on the undercloud in U, a deployer could have it derive the
> > params in theory if the workbook is run manually and the resultant
> > params are applied as overrides. Do we GA with that as a known issue
> > with workaround and then circle back to fix the above and backport it?
> > I don't think we should delay the release for it. I think we should
> > instead push harder on getting the playbook and roles that Kevin
> > started [2] (they could still land in U).
> >
> >   John
> >
> > [1]
> https://review.opendev.org/#/q/topic:mistral_to_ansible+(status:open+OR+status:merged)
> > [2]
> https://review.opendev.org/#/c/719466/13/tripleo_ansible/roles/tripleo_derived_parameters/tasks/main.yml@170
> >
> > >
> > >   John
> > >
> > > > Regards,
> > > > Saravanan KR
> > > >
> > > > >   John
> > > > >
> > > > > > 2. Move the functionality further down the component chain in
> TripleO to have the required ansible host/group_vars set for them to be
> used by config-download playbooks/ansible/puppet.
> > > > > >
> > > > > > I guess option 1 would be easier within the timelines. I've done
> some preliminary work to move some of the functionality in relevant mistral
> actions to utils modules[1], so that they can be called from ansible
> modules without depending on mistral/mistral-lib and use those in a
> playbook that kinda replicate the tasks in the mistral workflow.
> > > > > >
> > > > > > Having said that, it would be good to know what the DFG:NFV
> folks think, as they are the original authors/maintainers of that workflow.
> > > > > >
> > > > > > <snip>
> > > > > >
> > > > > >> The Mistral based workflow took advantage of the deployment
> plan which
> > > > > >> was stored in Swift on the undercloud. My understanding is that
> too is
> > > > > >> going away.
> > > > > >
> > > > > >
> > > > > > I'm not sure that would be in the scope of 'mstral-to-ansible'
> work. Dropping swift would probably be a bit more complex, as we use it to
> store templates, plan-environment, plan backups (possibly missing few more)
> etc  and would require significant design rework (may be possible when we
> get rid of heat in undercloud). In spite of heat using the templates from
> swift and merging environments on the client side, we've had already bumped
> heat's REST API json body size limit (max_json_body_size) on the undercloud
> to 4MB[2] from the default 1MB and sending all required templates as part
> of API request would not be a good idea from undercloud scalability pov.
> > > > > >
> > > > > > [1] https://review.opendev.org/#/c/709546/
> > > > > > [2]
> https://github.com/openstack/tripleo-heat-templates/blob/master/environments/undercloud.yaml#L109
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Rabi Mishra
> > > > > >
> > > > >
> > > >
>
>

-- 
Regards,
Rabi Mishra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200527/bd047c28/attachment-0001.html>