<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
</span>Ah OK I see now.<br>
<br>
I guess what I'm driving at, is how do we get automated coverage for the<br>
full end-to-end upgrade, so that issues[1] that I uncovered during manual<br>
testing start getting uncovered by automation?<br>
<br>
[1]: <a href="https://review.openstack.org/#/c/400847/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/400847/</a></blockquote><div><br></div><div>Completely agree, the end goal is to have the full end-to-end testing in the openstack-ansible repository, along with the individual repo tests.</div><div>We have an experimental job setup in project-config: (gate-openstack-ansible-openstack-ansible-upgrade-ubuntu-xenial-nv) although no concrete work has gone into adding this to the repo - so that one is pending.</div><div><br></div><div>The idea would be to run the upgrade scripts against an AIO built with stable/newton, and run tests after the upgrade - I'm hoping to see that land this cycle.</div><div><br></div><div>Andy </div></div></div></div>