<div dir="ltr">Hello everyone,<div><br></div><div>Yes, I am working on this now. We have some success already, but there is a lot of work to do. Of course, some things don't work ideally. For example, in [2] from the previous letter we have not 24 skipped tests, actually much more. So we have a bug somewhere :)</div><div><br></div><div>Regards,</div><div>Yaroslav Lobankov. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <span dir="ltr"><<a href="mailto:akurilin@mirantis.com" target="_blank">akurilin@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi!<br></div>Boris P. and I tried to push a spec[1] for automation tempest config generator, but we did not succeed to merge it. Imo, qa-team doesn't want to have such tool:(<span class=""><br><br>>However, there is a big concern:<br>>If the script contain a bug and creates the configuration which makes<br>>most tests skipped, we cannot do enough tests on the gate.<br>>Tempest contains 1432 tests and difficult to detect which tests are<br>>skipped as unexpected.<br><br></span>Yaroslav Lobankov is working on improvement for tempest config generator in Rally. Last time when we launch full tempest run[2], we got 1154 success tests and only 24 skipped. Also, there is a patch, which adds x-fail mechanism(it based on subunit-filter): you can transmit a file with test names + reasons and rally will modify results.<br><br>[1] - <a href="https://review.openstack.org/#/c/94473/" target="_blank">https://review.openstack.org/#/c/94473/</a><br><br>[2] - <a href="http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz" target="_blank">http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz</a><br></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <span dir="ltr"><<a href="mailto:ken1ohmichi@gmail.com" target="_blank">ken1ohmichi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Daniel,<br>
<br>
Thanks for pointing this up.<br>
<span><br>
2015-11-25 1:40 GMT+09:00 Daniel Mellado <<a href="mailto:daniel.mellado.es@ieee.org" target="_blank">daniel.mellado.es@ieee.org</a>>:<br>
> Hi All,<br>
><br>
> As you might already know, within Red Hat's tempest fork, we do have one<br>
> tempest configuration script which was built in the past by David Kranz [1]<br>
> and that's been actively used in our CI system. Regarding this topic, I'm<br>
> aware that quite some effort has been done in the past [2] and I would like<br>
> to complete the implementation of this blueprint/spec.<br>
><br>
> My plan would be to have this script under the /tempest/cmd or<br>
> /tempest/tools folder from tempest so it can be used to configure not the<br>
> tempest gate but any cloud we'd like to run tempest against.<br>
><br>
> Adding the configuration script was discussed briefly at the Mitaka summit<br>
> in the QA Priorities meting [3]. I propose we use the existing etherpad to<br>
> continue the discussion around and tracking of implementing "tempest<br>
> config-create" using the downstream config script as a starting point. [4]<br>
><br>
> If you have any questions, comments or opinion, please let me know.<br>
<br>
</span>This topic have happened several times, and I also felt this kind of<br>
tool was very useful for Tempest users, because Tempest contains 296<br>
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to<br>
set the configuration up.<br>
However, there is a big concern:<br>
If the script contain a bug and creates the configuration which makes<br>
most tests skipped, we cannot do enough tests on the gate.<br>
Tempest contains 1432 tests and difficult to detect which tests are<br>
skipped as unexpected.<br>
Actually we faced unexpected skipped tests on the gate before due to<br>
some bug, then the problem has been fixed.<br>
But I can imagine this kind of problem happens after implementing this<br>
kind of script.<br>
<br>
So now I am feeling Tempest users need to know what cloud they want to<br>
test with Tempest, and need to know what tests run with Tempest.<br>
Testers need to know what test target/items they are testing basically.<br>
<br>
Thanks<br>
Ken Ohmichi<br>
<br>
---<br>
<span><br>
> ---<br>
> [1]<br>
> <a href="https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py" rel="noreferrer" target="_blank">https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py</a><br>
> [2] <a href="https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator" rel="noreferrer" target="_blank">https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator</a><br>
> [3] <a href="https://etherpad.openstack.org/p/mitaka-qa-priorities" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/mitaka-qa-priorities</a><br>
> [4] <a href="https://etherpad.openstack.org/p/tempest-cli-improvements" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/tempest-cli-improvements</a><br>
><br>
> <a href="https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst" rel="noreferrer" target="_blank">https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst</a><br>
><br>
</span>> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr">Best regards,<br>Andrey Kurilin.<br></div></div>
</font></span></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>