<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 12, 2018 at 5:10 AM Sofer Athlan-Guyot <<a href="mailto:sathlang@redhat.com">sathlang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Testing and maintaining a green status for upgrade jobs within the 3h<br>
time limit has proven to be a very difficult job to say the least.<br></blockquote><div><br></div><div>Indeed </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The net result has been: we don't have anything even touching the<br>
upgrade code in the CI.<br>
<br>
So during Denver PTG it has been decided to give up on running a full<br>
upgrade job during the 3h time limit and instead to focus on two<br>
complementary approach to at least touch the upgrade code:<br>
1. run a standalone upgrade: this test the ansible upgrade playbook;<br>
2. run a N->N upgrade; this test the upgrade python code; </blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
And here there are, still not merged but seen working:<br>
- tripleo-ci-centos-7-standalone-upgrade:<br>
<a href="https://review.openstack.org/#/c/604706/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/604706/</a><br>
- tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades:<br>
<a href="https://review.openstack.org/#/c/607848/9" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/607848/9</a><br>
<br>
The first is good to merge (but other could disagree), the second could<br>
be as well (but I tend to disagree :))<br>
<br>
The first leverage the standalone deployment and execute an standalone<br>
upgrade just after it.<br>
<br>
The limitation is that it only tests non-HA services (sorry pidone,<br>
cannot test ha in standalone) and only the upgrade_tasks (ie not any<br>
workflow related to the upgrade cli)<br></blockquote><div><br></div><div>This can be augmented with 3rd party. The pidone team and the ci team are putting the final touches on a 3rd party job for HA services. Looking forward, I could see a 3rd party upgrade job that runs the pidone verification tests.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The main benefits here are:<br>
- ~2h to run the upgrade, still a bit long but far away from the 3h<br>
time limit;<br>
- we trigger a yum upgrade so that we can catch problems there as well;<br>
- we test the standalone upgrade which is good in itself;<br>
- composable role available (as in standalone/all-in-all deployment) so<br>
you can make a specific upgrade test for your project if it fits into<br>
the standalone constraint;<br></blockquote><div><br></div><div>These are all huge benefits over the previous implementation that have been made available to us via the standalone deployment!!!! </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
For this last point, if standalone specific role eventually goes into<br>
project testing (nova, neutron ...), they could have as well a way to<br>
test upgrade tasks. This would be a best case scenario.<br></blockquote><div><br></div><div>!!!!!!!!!!!!! woot !!!!!!!</div><div>This is a huge point that TripleO folks need to absorb!! </div><div><div>!!!!!!!!!!!!! woot !!!!!!!</div><br class="inbox-inbox-Apple-interchange-newline"></div><div>In the next several sprints the TripleO CI team will do our best to focus on the standalone deployments to convert TripleO's upstream jobs over and paving the way for other projects to start consuming it. IMHO I would think other projects would be *very* interested in testing an upgrade of their individual component w/o all the noise of unrelated services/components. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Now, for the second point, the N->N upgrade. Its "limitation" is that<br>
... well it doesn't run a yum upgrade at all. We start from master and<br>
run the upgrade to master.<br>
<br>
It's main benefit are:<br>
- it takes ~2h20 to run, so well under the 3h time;<br>
- tripleoclient upgrade code is run, which is one thing that the<br>
standalone ugprade cannot do.<br>
- It also tend to exercise idempotency of all the tasks as it runs them<br>
on an already "upgraded" node;<br>
- As added bonus, it could gate the tripleo-upgrade role as well as it<br>
definitively loads all of the role's tasks[1]<br>
<br>
For those that stayed with me to this point, I'm throwing another CI<br>
test that already proved useful already (caught errors), it's the<br>
ansible-lint test. After a standalone deployment we just run<br>
ansible-lint on all playbook generated[2].<br></blockquote><div><br></div><div>This is nice, thanks chem!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
It produces standalone_ansible_lint.log[3] in the working directory. It<br>
only takes a couple of minute to install ansible-lint and run it. It<br>
definitively gate against typos and the like. It touches hard to<br>
reach code as well, for instance the fast_forward tasks are linted.<br>
Still no pidone tasks in there but it could easily be added to a job<br>
that has HA tasks generated.<br>
<br>
Note that by default ansible-lint barks, as the generated playbooks hit<br>
several lintage problems, so only syntax errors and misnamed tasks or<br>
parameters are currently activated. But all the lint problems are<br>
logged in the above file and can be fixed later on. At which point we<br>
could activate full lint gating.<br>
<br>
Thanks for this long reading, any comments, shout of victory, cry of<br>
despair and reviews are welcomed.<br>
<br>
[1] but this has still to be investigated.<br>
[2] testing review <a href="https://review.openstack.org/#/c/604756/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/604756/</a> and main code <a href="https://review.openstack.org/#/c/604757/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/604757/</a><br>
[3] sample output <a href="http://paste.openstack.org/show/731960/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/731960/</a><br>
--<br>
Sofer Athlan-Guyot<br>
chem on #freenode<br>
Upgrade DFG.<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></blockquote><div><br></div><div>Very well done!! </div></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-family:sans-serif"><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="font-size:14px">Wes Hayutin</span></p><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="color:rgb(0,0,0);font-size:10px">Associate MANAGER</span><br></p><p style="color:rgb(153,153,153);font-family:overpass,sans-serif;margin:0px;font-size:10px"><a href="https://www.redhat.com/" target="_blank" style="color:rgb(0,136,206);margin:0px">Red Hat <br><br></a></p><span style="color:rgb(153,153,153);font-family:overpass,sans-serif;font-size:10px;margin:0px"><p style="margin:0px"><span style="color:rgb(153,153,153);margin:0px;padding:0px"><a href="mailto:whayutin@redhat.com">whayutin@redhat.com</a> </span><span style="color:rgb(153,153,153)"> T: </span><a href="tel:+19197544114" target="_blank" style="color:rgb(0,136,206);margin:0px">+1919</a>4232509<span style="color:rgb(153,153,153)"> IRC: </span>weshay<br></p></span><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img width="90" height="auto" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div><br style="font-family:sans-serif"><span style="font-family:sans-serif">View</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">calendar</span><span style="font-family:sans-serif"> and check </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> availability for meetings </span><span id="inbox-inbox-inbox-docs-internal-guid-37a711e5-1f4c-b3c1-221b-f589048200c7"><a href="https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">HERE</span></a></span><br></div></div>