<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 16, 2017 at 4:28 PM, Florian Fuchs <span dir="ltr"><<a href="mailto:flfuchs@redhat.com" target="_blank">flfuchs@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Mon, May 15, 2017 at 6:27 PM, Steven Hardy <<a href="mailto:shardy@redhat.com">shardy@redhat.com</a>> wrote:<br>
> On Mon, May 08, 2017 at 02:45:08PM +0300, Marios Andreou wrote:<br>
>> Hi folks, after some discussion locally with colleagues about improving<br>
>> the upgrades experience, one of the items that came up was pre-upgrade and<br>
>> update validations. I took an AI to look at the current status of<br>
>> tripleo-validations [0] and posted a simple WIP [1] intended to be run<br>
>> before an undercloud update/upgrade and which just checks service status.<br>
>> It was pointed out by shardy that for such checks it is better to instead<br>
>> continue to use the per-service  manifests where possible like [2] for<br>
>> example where we check status before N..O major upgrade. There may still<br>
>> be some undercloud specific validations that we can land into the<br>
>> tripleo-validations repo (thinking about things like the neutron<br>
>> networks/ports, validating the current nova nodes state etc?).<br>
>> So do folks have any thoughts about this subject - for example the kinds<br>
>> of things we should be checking - Steve said he had some reviews in<br>
>> progress for collecting the overcloud ansible puppet/docker config into an<br>
>> ansible playbook that the operator can invoke for upgrade of the 'manual'<br>
>> nodes (for example compute in the N..O workflow) - the point being that we<br>
>> can add more per-service ansible validation tasks into the service<br>
>> manifests for execution when the play is run by the operator - but I'll<br>
>> let Steve point at and talk about those.Â<br>
><br>
> Thanks for starting this thread Marios, sorry for the slow reply due to<br>
> Summit etc.<br>
><br>
> As we discussed, I think adding validations is great, but I'd prefer we<br>
> kept any overcloud validations specific to services in t-h-t instead of<br>
> trying to manage service specific things over multiple repos.<br>
><br>
> This would also help with the idea of per-step validations I think, where<br>
> e.g you could have a "is service active" test and run it after the step<br>
> where we expect the service to start, a blueprint was raised a while back<br>
> asking for exactly that:<br>
><br>
> <a href="https://blueprints.launchpad.net/tripleo/+spec/step-by-step-validation" rel="noreferrer" target="_blank">https://blueprints.launchpad.<wbr>net/tripleo/+spec/step-by-<wbr>step-validation</a><br>
><br>
> One way we could achive this is to add ansible tasks that perform some<br>
> validation after each step, where we combine the tasks for all services,<br>
> similar to how we already do upgrade_tasks and host_prep_tasks:<br>
><br>
> <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/database/redis.yaml#L92" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>tripleo-heat-templates/blob/<wbr>master/docker/services/<wbr>database/redis.yaml#L92</a><br>
><br>
> With the benefit of hindsight using ansible tags for upgrade_tasks wasn't<br>
> the best approach, because you can't change the tags via SoftwareDeployment<br>
> (e.g you need a SoftwareConfig per step), it's better if we either generate<br>
> the list of tasks by merging maps e.g<br>
><br>
> validation_tasks:<br>
> step3:<br>
> - sometask<br>
><br>
> Or via ansible conditionals where we pass a step value in to each run of<br>
> the tasks:<br>
><br>
> validation_tasks:<br>
> - sometask<br>
> when: step == 3<br>
><br>
> The latter approach is probably my preference, because it'll require less<br>
> complex merging in the heat layer.<br>
><br>
> As you mentioned, I've been working on ways to make the deployment steps<br>
> more ansible driven, so having these tasks integrated with the t-h-t model<br>
> would be well aligned with that I think:<br>
><br>
> <a href="https://review.openstack.org/#/c/454816/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/454816/</a><br>
><br>
> <a href="https://review.openstack.org/#/c/462211/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/462211/</a><br>
><br>
> Happy to discuss further when you're ready to start integrating some<br>
> overcloud validations.<br>
<br>
</div></div>Maybe these are two kinds of pre-upgrade validations that serve<br>
different purposes.<br>
<br>
The more general validations (like checking connectivity, making sure<br>
the stack is in good shape, repos are available, etc.) should give<br>
operators a fair amount of confidence that all basic prerequisites to<br>
start an update are met *before* the upgrade is started. They could be<br>
run from the UI or CLI and would fit well into the tripleo-validations<br>
repo. Similar to the existing tripleo-validations, failures don't<br>
prevent operators from doing something.<br>
<br>
The service-specific validations otoh are closely tied to the upgrade<br>
process and will stop further progress when failing.</blockquote><div><br></div><div>yeah - you could also argue that the current overcloud service upgrade validations (which just check 'is this service running OK' at step0 of the upgrade) are also *pre* upgrade since we didn't do anything yet it is literally step0. Note that as of upgrade to stable/ocata you can disable these if you need to re-run the upgrade step for example so it doesn't fail on the service checks.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> They are<br>
fundamentally different to the tripleo-validations and could therefore<br>
live in t-h-t.<br></blockquote><div><br></div><div>ACK yeah this seems to be the general consensus forming here - tripleo-validations for checking things especially on the undercloud</div><div>and continue to use the tht for the overcloud service validations. For many reasons and especially since we get the benefit of 'auto' generated list of services currently deployed per node etc etc so per service validation runs only if service is deployed.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I personally don't see why we shouldn't have pre-upgrade validations<br>
both in tripleo-validations and in t-h-t, as long as we know which<br>
ones go where. If everything that's tied to a specific overcloud<br>
service or upgrade step goes into t-h-t, I could see these two groups<br>
(using the validations suggested earlier in this thread):<br>
<br>
tripleo-validations:<br>
- Undercloud service check<br>
- Verify that the stack is in a *_COMPLETE state<br>
- Verify undercloud disk space. For node replacement we recommended a<br>
<span class="">minimum of 10 GB free.<br>
</span>- Network/repo availability check (undercloud and overcloud)<br>
- Verify we're at the latest version of the current release<br>
- ...<br>
<br>
tripleo-heat-templates:<br>
<span class="">- Pacemaker cluster health<br>
- Ceph health<br>
</span>- APIs healthcheck (per overcloud service)<br>
- Check Galera and Rabbit clusters and verify all nodes are up.<br>
- Disabling stonith.<br>
- ...<br></blockquote><div><br></div><div>thanks these all seem like good things to be checking and the split seems reasonable to me,</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In theory I could imagine another variety of pre-upgrade validations:<br>
Ones that are general in nature (not tied to an overcloud service),<br>
</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">but are specific to a particular version jump (so they would be run<br>
before a N..O upgrade, but wouldn't make sense for an O..P jump).<br>
</blockquote><div><br></div><div>for sure, we have things like migrations for example service foo-api is deprecated and instead the foo service is now served by apache and that will happen only in a specific upgrade version</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">These could still live in the tripleo-validations repo, but would only<br>
exist as backports to the relevant "from"-version. But lacking a good<br>
example, this is probably a bit academic for now. :-)<br>
<br>
Any thoughts?<br>
<br>
Thanks<br>
<span class="HOEnZb"><font color="#888888">Florian<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>