<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Mar 13, 2017 at 6:52 AM Ghanshyam Mann <<a href="mailto:ghanshyammann@gmail.com">ghanshyammann@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On Sun, Mar 12, 2017 at 7:40 AM, Matt Riedemann <span dir="ltr" class="gmail_msg"><<a href="mailto:mriedemos@gmail.com" class="gmail_msg" target="_blank">mriedemos@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 3/10/2017 3:02 PM, Andrea Frittoli wrote:<br class="gmail_msg">
<blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="gmail_msg">
We had a couple of sessions related to this topic at the PTG [0][1].<br class="gmail_msg">
<br class="gmail_msg">
We agreed that we want to still maintain integration tests only in<br class="gmail_msg">
Tempest, which means that API micro versions that have no integration<br class="gmail_msg">
impact can be tested via functional tests.<br class="gmail_msg">
</blockquote>
<br class="gmail_msg">
To be clear, "integration" here means tests that span operations across multiple services, correct? Like a compute test that first creates a port in the networking service and a volume in the block storage service and then uses those to create a server and maybe take a snapshot of it which is then verified was uploaded to the image service.<br class="gmail_msg">
<br class="gmail_msg">
The non-integration things are self-contained in a single service, like if all you need to do is create an aggregate, show it's details and validate the response, at a particular microversion, we can just do that in nova functional tests, and it's not necessary in Tempest.<br class="gmail_msg">
<br class="gmail_msg">
It might be worth having a definition of this policy in the Tempest docs so when people ask this question again you can just point at the docs.<br class="gmail_msg">
<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Yes, that will be helpful to have those agreement in doc. I am adding those in existing microversion testing doc [0] and later will be refactoring to give them better and more visible shape.</div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br class="gmail_msg"></div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Tempest maintain the info about what microversion tests are is implemented on Tempest side which will be helpful for projects to check the coverage [1]. Cinder one was missed which I added in [2]. </div><br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">As summary:</div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Tempest Scope:</div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">- Only Integration tests for microversion is allowed in Tempest</div></div></div></div></div></blockquote><div>If a microversion is important for interoperability it may end up in Tempest even if doesn't involve too much integration</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">- Exception for non integration tests to fill the schema gap if exists.</div></div></div></div></div></blockquote><div>Yeah when we fill schema gaps we need at least one test for the top version to verify that the latest implemented schema is valid</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br class="gmail_msg"></div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Project Scope:</div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">- Remaining tests coverage of microversion should be on Projects side.</div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"> - Project can cover those as functional tests etc. </div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"> - Nova currently have at least one functional tests per each microversion and plan to extend coverage like [3]</div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"> - IMO, only running tests with 'latest' microversion is not enough, each microversion should be tested with positive and negative testing (w.r.t at least immediate previous microversion).<br class="gmail_msg"></div></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="gmail_msg">
In terms of which versions we test in the gate, for nova we always run<br class="gmail_msg">
with min_microversion = None and max_microversion = latest, which means<br class="gmail_msg">
that all tests will be executed.<br class="gmail_msg">
Since micro versions are incremental, and each micro version usually<br class="gmail_msg">
involves no or one test on Tempest side, I think it will be a while<br class="gmail_msg">
before this becomes an issue for the common gate.<br class="gmail_msg">
</blockquote>
<br class="gmail_msg">
We test max_microversion=latest only on master. On the devstack stable branch we cap the max_microversion, e.g.:<br class="gmail_msg">
<br class="gmail_msg">
<a href="https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339</a><br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
<br class="gmail_msg">
Thanks,<br class="gmail_msg">
<br class="gmail_msg">
Matt<br class="gmail_msg">
<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br class="gmail_msg"></div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br class="gmail_msg"></div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..0 <a href="https://review.openstack.org/#/c/444727/1" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/444727/1</a> </div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..1 <a href="https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest" class="gmail_msg" target="_blank">https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest</a> </div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..2 <a href="https://review.openstack.org/#/c/444711/" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/444711/</a> </div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">..3 <a href="https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests" class="gmail_msg" target="_blank">https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests</a> </div><br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Thanks</div><div class="gmail_default gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">gmann</div><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div></div></div>
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div></div>