Open Stack

On Wed, May 24, 2017 at 4:09 AM, Graeme Gillies <ggillies at redhat.com> wrote:

> On 08/05/17 21:45, Marios Andreou wrote:
> > Hi folks, after some discussion locally with colleagues about improving
> > the upgrades experience, one of the items that came up was pre-upgrade
> > and update validations. I took an AI to look at the current status of
> > tripleo-validations [0] and posted a simple WIP [1] intended to be run
> > before an undercloud update/upgrade and which just checks service
> > status. It was pointed out by shardy that for such checks it is better
> > to instead continue to use the per-service  manifests where possible
> > like [2] for example where we check status before N..O major upgrade.
> > There may still be some undercloud specific validations that we can land
> > into the tripleo-validations repo (thinking about things like the
> > neutron networks/ports, validating the current nova nodes state etc?).
> >
> > So do folks have any thoughts about this subject - for example the kinds
> > of things we should be checking - Steve said he had some reviews in
> > progress for collecting the overcloud ansible puppet/docker config into
> > an ansible playbook that the operator can invoke for upgrade of the
> > 'manual' nodes (for example compute in the N..O workflow) - the point
> > being that we can add more per-service ansible validation tasks into the
> > service manifests for execution when the play is run by the operator -
> > but I'll let Steve point at and talk about those.
> >
> > cheers, marios
> >
> > [0] https://github.com/openstack/tripleo-validations
> > [1] https://review.openstack.org/#/c/462918/
> > [2]  https://github.com/openstack/tripleo-heat-templates/blob/
> stable/ocata/puppet/services/neutron-api.yaml#L197
> >
> >
> > ____________________________________________________________
> ______________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> Hi Marios,
>
> Forgive me if I misunderstand here, but it looks like part of this goal
> is to do things like ensure the overcloud is in a decent state before an
> upgrade/update is executed.
>
> How would this work in a situation where I have hit an openstack bug
> which causes my cinder service to stop working/fail, and I a fix has
> been created/packaged, ready for me to update my overcloud with, but the
> validations bomb out because cinder isn't running (and I can't update my
> overcloud to the newest package with the fix because the validation fails?)
>
>
o/ right... so there are roughly two groups of things here - validations
for the undercloud (of which we don't have much and we want to add some)
and validations for the overcloud. For the former we are targetting
tripleo-validations and for the latter adding to the existing service
checks in the tripleo-heat-template service manifests for execution during
the upgrade.

For both we need a way to disable them - one of the key concerns is the
scenario you describe. For the overcoud service checks we already have that
at least for the current simple "is service running " (grep
SkipUpgradeConfigTags at
https://docs.openstack.org/developer/tripleo-docs/post_deployment/upgrade.html).
For the tripleo-validations I believe there is a 'validations fatal' type
flag already that you can pass to the client.

hope it answers your concern



> Regards,
>
> Graeme
>
> --
> Graeme Gillies
> Principal Systems Administrator
> Openstack Infrastructure
> Red Hat Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170524/2a33e483/attachment.html>