<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_extra">
<br><div class="gmail_quote">On Sun, Mar 12, 2017 at 7:40 AM, Matt Riedemann <span dir="ltr"><<a href="mailto:mriedemos@gmail.com" target="_blank">mriedemos@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 3/10/2017 3:02 PM, Andrea Frittoli wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
We had a couple of sessions related to this topic at the PTG [0][1].<br>
<br>
We agreed that we want to still maintain integration tests only in<br>
Tempest, which means that API micro versions that have no integration<br>
impact can be tested via functional tests.<br>
</blockquote>
<br>
To be clear, "integration" here means tests that span operations across multiple services, correct? Like a compute test that first creates a port in the networking service and a volume in the block storage service and then uses those to create a server and maybe take a snapshot of it which is then verified was uploaded to the image service.<br>
<br>
The non-integration things are self-contained in a single service, like if all you need to do is create an aggregate, show it's details and validate the response, at a particular microversion, we can just do that in nova functional tests, and it's not necessary in Tempest.<br>
<br>
It might be worth having a definition of this policy in the Tempest docs so when people ask this question again you can just point at the docs.<br>
<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Yes, that will be helpful to have those agreement in doc. I am adding those in existing microversion testing doc [0] and later will be refactoring to give them better and more visible shape.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Tempest maintain the info about what microversion tests are is implemented on Tempest side which will be helpful for projects to check the coverage [1]. Cinder one was missed which I added in [2]. </div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">As summary:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Tempest Scope:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">- Only Integration tests for microversion is allowed in Tempest</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">- Exception for non integration tests to fill the schema gap if exists.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Project Scope:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">- Remaining tests coverage of microversion should be on Projects side.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"> - Project can cover those as functional tests etc. </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"> - Nova currently have at least one functional tests per each microversion and plan to extend coverage like [3]</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"> - IMO, only running tests with 'latest' microversion is not enough, each microversion should be tested with positive and negative testing (w.r.t at least immediate previous microversion).<br></div></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
In terms of which versions we test in the gate, for nova we always run<br>
with min_microversion = None and max_microversion = latest, which means<br>
that all tests will be executed.<br>
Since micro versions are incremental, and each micro version usually<br>
involves no or one test on Tempest side, I think it will be a while<br>
before this becomes an issue for the common gate.<br>
</blockquote>
<br>
We test max_microversion=latest only on master. On the devstack stable branch we cap the max_microversion, e.g.:<br>
<br>
<a href="https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339" rel="noreferrer" target="_blank">https://github.com/openstack-d<wbr>ev/devstack/blob/stable/newton<wbr>/lib/tempest#L339</a><br>
<br>
--<br>
<br>
Thanks,<br>
<br>
Matt<br>
<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..0 <a href="https://review.openstack.org/#/c/444727/1">https://review.openstack.org/#/c/444727/1</a> </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..1 <a href="https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest">https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest</a> </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..2 <a href="https://review.openstack.org/#/c/444711/">https://review.openstack.org/#/c/444711/</a> </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..3 <a href="https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests">https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests</a> </div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Thanks</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">gmann</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</blockquote></div><br></div></div>