Open Stack

On Tue, May 9, 2017 at 11:46 AM, mathieu bultel <mbultel at redhat.com> wrote:

> On 05/08/2017 01:45 PM, Marios Andreou wrote:
>
> Hi folks, after some discussion locally with colleagues about improving
> the upgrades experience, one of the items that came up was pre-upgrade and
> update validations. I took an AI to look at the current status of
> tripleo-validations [0] and posted a simple WIP [1] intended to be run
> before an undercloud update/upgrade and which just checks service status.
> It was pointed out by shardy that for such checks it is better to instead
> continue to use the per-service  manifests where possible like [2] for
> example where we check status before N..O major upgrade. There may still be
> some undercloud specific validations that we can land into the
> tripleo-validations repo (thinking about things like the neutron
> networks/ports, validating the current nova nodes state etc?).
>
> Yes, I think a bunch of validation: db states, services states, network
> connectivity (external, internal)
>
>
> So do folks have any thoughts about this subject - for example the kinds
> of things we should be checking - Steve said he had some reviews in
> progress for collecting the overcloud ansible puppet/docker config into an
> ansible playbook that the operator can invoke for upgrade of the 'manual'
> nodes (for example compute in the N..O workflow) - the point being that we
> can add more per-service ansible validation tasks into the service
> manifests for execution when the play is run by the operator - but I'll let
> Steve point at and talk about those.
>
> I have a WIP review about that [1], but i need to revisit it a bit, to add
> a part into the mistral workflow (Im also writing a POC to create a mistral
> workbook for major upgrade and validate minor/major upgrade before starting
> [2], I have a third one in progress, not pushed yet, to implement the major
> upgrade option in the cli):
>
> [1] https://review.openstack.org/444224 <https://review.openstack.org/444224>
> [2] https://review.openstack.org/#/c/462961
>
>
>
ack I had a pass at those when you first sent this - will look again. I
think our discussion have highlighted a few things

* lack of tripleo-common/client support for upgrades workflow (e.g. so we
can kill the -e for every invocation)

* better - more pre-upgrade/update validations - especially undercloud
doesn't have much/any (already have some pre-upgrade validations) also
minor update doesn't have much/any converage .

* improving the 'manual node upgrade' - instead of running
upgrade-non-controller.sh use a playbook. that can be run by the operator
(or even automated?)

* better logging tracking of current upgrades step/progress - when failures
happen

* related to ^^^ when upgrade/update completes on current node writing a
/etc/tripleorelease or somesuch to say 'mitaka' or 'ocata' or whatever the
version just been upgraded to, is.

I recall Emilien suggesting blueprint for tracking we should discuss some
more and get these down as one or more blueprints probably - assuming folks
agree with that list and what we can add/remove/change  -  lets work
together to split the blueprints if you like but lets discuss them a bit
here first/or not more comments lets see :)

thanks



> cheers, marios
>
> [0] https://github.com/openstack/tripleo-validations
> [1] https://review.openstack.org/#/c/462918/
> [2]   <https://github.com/openstack/>https://github.com/openstack/
> tripleo-heat-templates/blob/stable/ocata/puppet/services/neutron-api.yaml#
> L197
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170517/f4a69bd1/attachment.html>