<div dir="ltr">I would say smoke tests, and at least the minimum scenario tests.<div>Smoke tests takes 14 minutes (113 tests) to run, and I can check how long it takes the minimum scenario tests later. So it won't take a long time running.</div><div><br></div><div>Kind regards,</div><div>Arx Cruz</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 6, 2017 at 2:44 PM, Andrea Frittoli <span dir="ltr"><<a href="mailto:andrea.frittoli@gmail.com" target="_blank">andrea.frittoli@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I don't really have much context in what the decision is going to be based on here,<div>so I'll just add some random comments here and there.<br><br><div class="gmail_quote"><span class=""><div dir="ltr">On Thu, Apr 6, 2017 at 12:48 PM Arx Cruz <<a href="mailto:arxcruz@redhat.com" target="_blank">arxcruz@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-2195184585465792063gmail_msg">Having tempest running will allow these jobs to appear in openstack-health system as well.</div></blockquote><div><br></div></span><div>I agree that's a plus. It's also rather easy to produce subunit from whatever you</div><div>are using to run tests, and that's all you need in fact to get data into open stack-health</div><div>without touching the existing infrastructure. So in case you decide not to use Tempest,</div><div>open stack-health can still be on the list.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra m_-2195184585465792063gmail_msg"><br class="m_-2195184585465792063gmail_msg"><div class="gmail_quote m_-2195184585465792063gmail_msg">On Thu, Apr 6, 2017 at 1:29 PM, Justin Kilpatrick <span dir="ltr" class="m_-2195184585465792063gmail_msg"><<a href="mailto:jkilpatr@redhat.com" class="m_-2195184585465792063gmail_msg" target="_blank">jkilpatr@redhat.com</a>></span> wrote:<br class="m_-2195184585465792063gmail_msg"><blockquote class="gmail_quote m_-2195184585465792063gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Maybe I'm getting a little off topic with this question, but why was<br class="m_-2195184585465792063gmail_msg">
Tempest removed last time?<br class="m_-2195184585465792063gmail_msg">
<br class="m_-2195184585465792063gmail_msg">
I'm not well versed in the history of this discussion, but from what I<br class="m_-2195184585465792063gmail_msg">
understand Tempest in the gate has<br class="m_-2195184585465792063gmail_msg">
been an off and on again thing for a while but I've never heard the<br class="m_-2195184585465792063gmail_msg">
story of why it got removed.<br class="m_-2195184585465792063gmail_msg">
<div class="m_-2195184585465792063m_-8302464129252996066HOEnZb m_-2195184585465792063gmail_msg"><div class="m_-2195184585465792063m_-8302464129252996066h5 m_-2195184585465792063gmail_msg"><br class="m_-2195184585465792063gmail_msg">
On Thu, Apr 6, 2017 at 7:00 AM, Chris Dent <<a href="mailto:cdent%2Bos@anticdent.org" class="m_-2195184585465792063gmail_msg" target="_blank">cdent+os@anticdent.org</a>> wrote:<br class="m_-2195184585465792063gmail_msg">
> On Thu, 6 Apr 2017, Sagi Shnaidman wrote:<br class="m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg">
>> It may be useful to run a "limited edition" of API tests that maximize<br class="m_-2195184585465792063gmail_msg">
>> coverage and don't duplicate, for example just to check service working<br class="m_-2195184585465792063gmail_msg">
>> basically, without covering all its functionality. It will take very<br class="m_-2195184585465792063gmail_msg">
>> little<br class="m_-2195184585465792063gmail_msg">
>> time (i.e. 5 tests for each service) and will give a general picture of<br class="m_-2195184585465792063gmail_msg">
>> deployment success. It will cover fields that are not covered by pingtest<br class="m_-2195184585465792063gmail_msg">
>> as well. </div></div></blockquote></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra m_-2195184585465792063gmail_msg"><div class="gmail_quote m_-2195184585465792063gmail_msg"><blockquote class="gmail_quote m_-2195184585465792063gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-2195184585465792063m_-8302464129252996066HOEnZb m_-2195184585465792063gmail_msg"><div class="m_-2195184585465792063m_-8302464129252996066h5 m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg"></div></div></blockquote></div></div></blockquote><div><br></div></span><div><div>We have a smoke attribute here an there, but it's not well curated at all, so you're</div><div>probably better off maintaining your own list.</div><div>Since presumably you're more interested in verifying that a deployed cloud is</div><div>functional - as opposed to verify specific APIs are working properly - you may want</div><div>to look at scenario tests, where with a couple of test you can cover already a lot of</div><div>basic stuff, e.g. if you can boot a server from a volume with an image from glance, </div><div>and ssh into it, you have proven a lot already about the general health of your cloud.</div></div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra m_-2195184585465792063gmail_msg"><div class="gmail_quote m_-2195184585465792063gmail_msg"><blockquote class="gmail_quote m_-2195184585465792063gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-2195184585465792063m_-8302464129252996066HOEnZb m_-2195184585465792063gmail_msg"><div class="m_-2195184585465792063m_-8302464129252996066h5 m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg">
> It's sound like using some parts of tempest is perhaps the desired<br class="m_-2195184585465792063gmail_msg">
> thing here but in case a "limited edition" test against the APIs to<br class="m_-2195184585465792063gmail_msg">
> do what amounts to a smoke test is desired, it might be worthwhile<br class="m_-2195184585465792063gmail_msg">
> to investigate using gabbi[1] and its command line gabbi-run[2] tool for<br class="m_-2195184585465792063gmail_msg">
> some fairly simple and readable tests that can describe a sequence<br class="m_-2195184585465792063gmail_msg">
> of API interactions. There are lots of tools that can do the same<br class="m_-2195184585465792063gmail_msg">
> thing, so gabbi may not be the right choice but it's there as an<br class="m_-2195184585465792063gmail_msg">
> option.<br class="m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg">
> The telemetry group had (an may still have) some integration tests<br class="m_-2195184585465792063gmail_msg">
> that use gabbi files to integrate ceilometer, heat (starting some<br class="m_-2195184585465792063gmail_msg">
> vms), aodh and gnocchi and confirm that the expected flow happened.<br class="m_-2195184585465792063gmail_msg">
> Since the earlier raw scripts I think there's been some integration<br class="m_-2195184585465792063gmail_msg">
> with tempest, but gabbi files are still used[3].<br class="m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg">
> If this might be useful and I can help out, please ask.<br class="m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg">
> [1] <a href="http://gabbi.readthedocs.io/" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">http://gabbi.readthedocs.io/</a><br class="m_-2195184585465792063gmail_msg">
> [2] <a href="http://gabbi.readthedocs.io/en/latest/runner.html" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">http://gabbi.readthedocs.io/<wbr>en/latest/runner.html</a><br class="m_-2195184585465792063gmail_msg">
> [3]<br class="m_-2195184585465792063gmail_msg">
> <a href="https://github.com/openstack/ceilometer/tree/master/ceilometer/tests/integration" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">https://github.com/openstack/<wbr>ceilometer/tree/master/<wbr>ceilometer/tests/integration</a><br class="m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg">
> --<br class="m_-2195184585465792063gmail_msg">
> Chris Dent ¯\_(ツ)_/¯ <a href="https://anticdent.org/" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">https://anticdent.org/</a><br class="m_-2195184585465792063gmail_msg">
> freenode: cdent tw: @anticdent<br class="m_-2195184585465792063gmail_msg">
</div></div><div class="m_-2195184585465792063m_-8302464129252996066HOEnZb m_-2195184585465792063gmail_msg"><div class="m_-2195184585465792063m_-8302464129252996066h5 m_-2195184585465792063gmail_msg">> ______________________________<wbr>______________________________<wbr>______________<br class="m_-2195184585465792063gmail_msg">
> OpenStack Development Mailing List (not for usage questions)<br class="m_-2195184585465792063gmail_msg">
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="m_-2195184585465792063gmail_msg">
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="m_-2195184585465792063gmail_msg">
><br class="m_-2195184585465792063gmail_msg">
<br class="m_-2195184585465792063gmail_msg">
______________________________<wbr>______________________________<wbr>______________<br class="m_-2195184585465792063gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="m_-2195184585465792063gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="m_-2195184585465792063gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="m_-2195184585465792063gmail_msg">
</div></div></blockquote></div><br class="m_-2195184585465792063gmail_msg"></div>
______________________________<wbr>______________________________<wbr>______________<br class="m_-2195184585465792063gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="m_-2195184585465792063gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="m_-2195184585465792063gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_-2195184585465792063gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="m_-2195184585465792063gmail_msg">
</blockquote></div></div></div></div></div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>