[openstack-dev] [tripleo] Validations before upgrades and updates

Steven Hardy shardy at redhat.com
Mon May 15 16:27:38 UTC 2017


On Mon, May 08, 2017 at 02:45:08PM +0300, Marios Andreou wrote:
>    Hi folks, after some discussion locally with colleagues about improving
>    the upgrades experience, one of the items that came up was pre-upgrade and
>    update validations. I took an AI to look at the current status of
>    tripleo-validations [0] and posted a simple WIP [1] intended to be run
>    before an undercloud update/upgrade and which just checks service status.
>    It was pointed out by shardy that for such checks it is better to instead
>    continue to use the per-service  manifests where possible like [2] for
>    example where we check status before N..O major upgrade. There may still
>    be some undercloud specific validations that we can land into the
>    tripleo-validations repo (thinking about things like the neutron
>    networks/ports, validating the current nova nodes state etc?).
>    So do folks have any thoughts about this subject - for example the kinds
>    of things we should be checking - Steve said he had some reviews in
>    progress for collecting the overcloud ansible puppet/docker config into an
>    ansible playbook that the operator can invoke for upgrade of the 'manual'
>    nodes (for example compute in the N..O workflow) - the point being that we
>    can add more per-service ansible validation tasks into the service
>    manifests for execution when the play is run by the operator - but I'll
>    let Steve point at and talk about those. 

Thanks for starting this thread Marios, sorry for the slow reply due to
Summit etc.

As we discussed, I think adding validations is great, but I'd prefer we
kept any overcloud validations specific to services in t-h-t instead of
trying to manage service specific things over multiple repos.

This would also help with the idea of per-step validations I think, where
e.g you could have a "is service active" test and run it after the step
where we expect the service to start, a blueprint was raised a while back
asking for exactly that:

https://blueprints.launchpad.net/tripleo/+spec/step-by-step-validation

One way we could achive this is to add ansible tasks that perform some
validation after each step, where we combine the tasks for all services,
similar to how we already do upgrade_tasks and host_prep_tasks:

https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/database/redis.yaml#L92

With the benefit of hindsight using ansible tags for upgrade_tasks wasn't
the best approach, because you can't change the tags via SoftwareDeployment
(e.g you need a SoftwareConfig per step), it's better if we either generate
the list of tasks by merging maps e.g 

  validation_tasks:
    step3:
      - sometask

Or via ansible conditionals where we pass a step value in to each run of
the tasks:

  validation_tasks:
    - sometask
      when: step == 3

The latter approach is probably my preference, because it'll require less
complex merging in the heat layer.

As you mentioned, I've been working on ways to make the deployment steps
more ansible driven, so having these tasks integrated with the t-h-t model
would be well aligned with that I think:

https://review.openstack.org/#/c/454816/

https://review.openstack.org/#/c/462211/

Happy to discuss further when you're ready to start integrating some
overcloud validations.

Thanks!

Steve



More information about the OpenStack-dev mailing list