<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><span style="font-family:arial,sans-serif;color:rgb(34,34,34)">On Mon, Mar 13, 2017 at 7:37 PM, Andrea Frittoli </span><span dir="ltr" style="font-family:arial,sans-serif;color:rgb(34,34,34)"><<a href="mailto:andrea.frittoli@gmail.com" target="_blank">andrea.frittoli@gmail.com</a>></span><span style="font-family:arial,sans-serif;color:rgb(34,34,34)"> wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg">On Sat, Mar 11, 2017 at 10:45 PM Matt Riedemann <<a href="mailto:mriedemos@gmail.com" class="gmail-m_-755548765522883431gmail_msg" target="_blank">mriedemos@gmail.com</a>> wrote:<br class="gmail-m_-755548765522883431gmail_msg"></div><blockquote class="gmail_quote gmail-m_-755548765522883431gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 3/10/2017 3:02 PM, Andrea Frittoli wrote:<br class="gmail-m_-755548765522883431gmail_msg">
><br class="gmail-m_-755548765522883431gmail_msg">
> We had a couple of sessions related to this topic at the PTG [0][1].<br class="gmail-m_-755548765522883431gmail_msg">
><br class="gmail-m_-755548765522883431gmail_msg">
> We agreed that we want to still maintain integration tests only in<br class="gmail-m_-755548765522883431gmail_msg">
> Tempest, which means that API micro versions that have no integration<br class="gmail-m_-755548765522883431gmail_msg">
> impact can be tested via functional tests.<br class="gmail-m_-755548765522883431gmail_msg">
<br class="gmail-m_-755548765522883431gmail_msg">
To be clear, "integration" here means tests that span operations across<br class="gmail-m_-755548765522883431gmail_msg">
multiple services, correct? Like a compute test that first creates a<br class="gmail-m_-755548765522883431gmail_msg">
port in the networking service and a volume in the block storage service<br class="gmail-m_-755548765522883431gmail_msg">
and then uses those to create a server and maybe take a snapshot of it<br class="gmail-m_-755548765522883431gmail_msg">
which is then verified was uploaded to the image service.<br class="gmail-m_-755548765522883431gmail_msg"></blockquote></div></div><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><div class="gmail-m_-755548765522883431gmail_msg">Yes, indeed.</div></div></div><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><div class="gmail-m_-755548765522883431gmail_msg"> <br class="gmail-m_-755548765522883431gmail_msg"></div><blockquote class="gmail_quote gmail-m_-755548765522883431gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="gmail-m_-755548765522883431gmail_msg">
The non-integration things are self-contained in a single service, like<br class="gmail-m_-755548765522883431gmail_msg">
if all you need to do is create an aggregate, show it's details and<br class="gmail-m_-755548765522883431gmail_msg">
validate the response, at a particular microversion, we can just do that<br class="gmail-m_-755548765522883431gmail_msg">
in nova functional tests, and it's not necessary in Tempest.<br class="gmail-m_-755548765522883431gmail_msg">
<br class="gmail-m_-755548765522883431gmail_msg">
It might be worth having a definition of this policy in the Tempest docs<br class="gmail-m_-755548765522883431gmail_msg">
so when people ask this question again you can just point at the docs.<br class="gmail-m_-755548765522883431gmail_msg"></blockquote><div class="gmail-m_-755548765522883431gmail_msg"><br class="gmail-m_-755548765522883431gmail_msg"></div></div></div><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><div class="gmail-m_-755548765522883431gmail_msg">I agree, we need to go a better job at documenting this. </div><div class="gmail-m_-755548765522883431gmail_msg">I just want to avoid having a black and white rule that would not work.</div><div class="gmail-m_-755548765522883431gmail_msg"><br></div><div class="gmail-m_-755548765522883431gmail_msg">To me tests that should be in Tempest are tests that make sense to run</div><div class="gmail-m_-755548765522883431gmail_msg">against any change in any repo, and the reasons for that can be many:</div><div class="gmail-m_-755548765522883431gmail_msg">- the test involves multiple services (more that "it uses a token though")</div><div class="gmail-m_-755548765522883431gmail_msg">- the test covers a feature which is key for interoperability</div><div class="gmail-m_-755548765522883431gmail_msg">- the test must be reliable and not take too long</div><div class="gmail-m_-755548765522883431gmail_msg"><br></div></div></div></div></blockquote><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">​+1. I added those in <a href="https://review.openstack.org/#/c/444727/">https://review.openstack.org/#/c/444727/</a>​</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><div class="gmail-m_-755548765522883431gmail_msg"></div><div class="gmail-m_-755548765522883431gmail_msg">Based on this I don't it's possible to completely avoid the discussion about</div><div class="gmail-m_-755548765522883431gmail_msg">which test should in Tempest and which not, it's something to be considered</div><div class="gmail-m_-755548765522883431gmail_msg">on a test by test basis, based on a set of guidelines.</div><div class="gmail-m_-755548765522883431gmail_msg"> <br></div><div class="gmail-m_-755548765522883431gmail_msg">We have an initial definition of scope in [0], but it's probably worth to elaborate</div><div class="gmail-m_-755548765522883431gmail_msg">more on it. I'll put up a patch so that the discussion on it can continue in</div><div class="gmail-m_-755548765522883431gmail_msg">gerrit.</div><div class="gmail-m_-755548765522883431gmail_msg"><br></div><div class="gmail-m_-755548765522883431gmail_msg">[0] <a href="https://docs.openstack.org/developer/tempest/test-removal.html#tempest-scope" target="_blank">https://docs.openstack.<wbr>org/developer/tempest/test-<wbr>removal.html#tempest-scope</a></div><div class="gmail-m_-755548765522883431gmail_msg"><br></div></div></div><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><blockquote class="gmail_quote gmail-m_-755548765522883431gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="gmail-m_-755548765522883431gmail_msg">
><br class="gmail-m_-755548765522883431gmail_msg">
> In terms of which versions we test in the gate, for nova we always run<br class="gmail-m_-755548765522883431gmail_msg">
> with min_microversion = None and max_microversion = latest, which means<br class="gmail-m_-755548765522883431gmail_msg">
> that all tests will be executed.<br class="gmail-m_-755548765522883431gmail_msg">
> Since micro versions are incremental, and each micro version usually<br class="gmail-m_-755548765522883431gmail_msg">
> involves no or one test on Tempest side, I think it will be a while<br class="gmail-m_-755548765522883431gmail_msg">
> before this becomes an issue for the common gate.<br class="gmail-m_-755548765522883431gmail_msg">
<br class="gmail-m_-755548765522883431gmail_msg">
We test max_microversion=latest only on master. On the devstack stable<br class="gmail-m_-755548765522883431gmail_msg">
branch we cap the max_microversion, e.g.:<br class="gmail-m_-755548765522883431gmail_msg"></blockquote></div></div><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><div class="gmail-m_-755548765522883431gmail_msg">Thanks, good point.</div></div></div><div dir="ltr" class="gmail-m_-755548765522883431gmail_msg"><div class="gmail_quote gmail-m_-755548765522883431gmail_msg"><div class="gmail-m_-755548765522883431gmail_msg"> </div><blockquote class="gmail_quote gmail-m_-755548765522883431gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="gmail-m_-755548765522883431gmail_msg">
<a href="https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339" rel="noreferrer" class="gmail-m_-755548765522883431gmail_msg" target="_blank">https://github.com/openstack-<wbr>dev/devstack/blob/stable/<wbr>newton/lib/tempest#L339</a><span class="gmail-HOEnZb"><font color="#888888"><br class="gmail-m_-755548765522883431gmail_msg">
<br class="gmail-m_-755548765522883431gmail_msg">
--<br class="gmail-m_-755548765522883431gmail_msg">
<br class="gmail-m_-755548765522883431gmail_msg">
Thanks,<br class="gmail-m_-755548765522883431gmail_msg">
<br class="gmail-m_-755548765522883431gmail_msg">
Matt<br class="gmail-m_-755548765522883431gmail_msg">
<br class="gmail-m_-755548765522883431gmail_msg">
______________________________<wbr>______________________________<wbr>______________<br class="gmail-m_-755548765522883431gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail-m_-755548765522883431gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail-m_-755548765522883431gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="gmail-m_-755548765522883431gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail-m_-755548765522883431gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="gmail-m_-755548765522883431gmail_msg">
</font></span></blockquote></div></div></div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div></div>