<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still good -we just won't use it for the gates- <br>
Instead it'd be used as a wrapper for a user who would be interested
on trying it against a real/reals clouds.<br>
<br>
Ken, do you really think a tempest user should know all tempest
options? As you pointed out there are quite a few of them and even
if they should at least know their environment, this script would
set a minimum acceptable default. Do you think PTL and Pre-PTL
concerns that we spoke of would still apply to that scenario?<br>
<br>
Andrey, Yaroslav. Would you like to revisit the blueprint to adapt
it to tempest-cli improvements? What do you think about this,
Masayuki?<br>
<br>
Thanks for all your feedback! ;)<br>
<br>
El 27/11/15 a las 00:15, Andrey Kurilin escribió:<br>
<blockquote
cite="mid:CAEVmkawtkK6CzM8U6poCDD6b8JVZ=p9Ai_HzzcKXjNue=mCtuw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Sorry for wrong numbers. The bug-fix for issue with
counters is merged.<br>
Correct numbers(latest result from rally's gate[1]):<br>
- total number of executed tests: 1689<br>
- success: 1155<br>
- skipped: 534 (neutron,heat,sahara,ceilometer are disabled.
[2] should enable them)<br>
</div>
- failed: 0<br>
<div><br>
[1] - <a moz-do-not-send="true"
href="http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz">http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz</a><br>
[2] - <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/250540/">https://review.openstack.org/#/c/250540/</a><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Nov 26, 2015 at 3:23 PM,
Yaroslav Lobankov <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ylobankov@mirantis.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:ylobankov@mirantis.com">ylobankov@mirantis.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello everyone,
<div><br>
</div>
<div>Yes, I am working on this now. We have some success
already, but there is a lot of work to do. Of course,
some things don't work ideally. For example, in [2] from
the previous letter we have not 24 skipped tests,
actually much more. So we have a bug somewhere :)</div>
<div><br>
</div>
<div>Regards,</div>
<div>Yaroslav Lobankov. </div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Nov 26, 2015 at 3:59
PM, Andrey Kurilin <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:akurilin@mirantis.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:akurilin@mirantis.com">akurilin@mirantis.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Hi!<br>
</div>
Boris P. and I tried to push a spec[1] for
automation tempest config generator, but we did
not succeed to merge it. Imo, qa-team doesn't
want to have such tool:(<span><br>
<br>
>However, there is a big concern:<br>
>If the script contain a bug and creates
the configuration which makes<br>
>most tests skipped, we cannot do enough
tests on the gate.<br>
>Tempest contains 1432 tests and difficult
to detect which tests are<br>
>skipped as unexpected.<br>
<br>
</span>Yaroslav Lobankov is working on
improvement for tempest config generator in
Rally. Last time when we launch full tempest
run[2], we got 1154 success tests and only 24
skipped. Also, there is a patch, which adds
x-fail mechanism(it based on subunit-filter):
you can transmit a file with test names +
reasons and rally will modify results.<br>
<br>
[1] - <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/94473/"
target="_blank">https://review.openstack.org/#/c/94473/</a><br>
<br>
[2] - <a moz-do-not-send="true"
href="http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz"
target="_blank">http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz</a><br>
</div>
<div class="gmail_extra">
<div>
<div><br>
<div class="gmail_quote">On Thu, Nov 26,
2015 at 1:52 PM, Ken'ichi Ohmichi <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:ken1ohmichi@gmail.com"
target="_blank">ken1ohmichi@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">Hi Daniel,<br>
<br>
Thanks for pointing this up.<br>
<span><br>
2015-11-25 1:40 GMT+09:00 Daniel
Mellado <<a moz-do-not-send="true"
href="mailto:daniel.mellado.es@ieee.org" target="_blank">daniel.mellado.es@ieee.org</a>>:<br>
> Hi All,<br>
><br>
> As you might already know, within
Red Hat's tempest fork, we do have one<br>
> tempest configuration script
which was built in the past by David
Kranz [1]<br>
> and that's been actively used in
our CI system. Regarding this topic,
I'm<br>
> aware that quite some effort has
been done in the past [2] and I would
like<br>
> to complete the implementation of
this blueprint/spec.<br>
><br>
> My plan would be to have this
script under the /tempest/cmd or<br>
> /tempest/tools folder from
tempest so it can be used to configure
not the<br>
> tempest gate but any cloud we'd
like to run tempest against.<br>
><br>
> Adding the configuration script
was discussed briefly at the Mitaka
summit<br>
> in the QA Priorities meting [3].
I propose we use the existing etherpad
to<br>
> continue the discussion around
and tracking of implementing "tempest<br>
> config-create" using the
downstream config script as a starting
point. [4]<br>
><br>
> If you have any questions,
comments or opinion, please let me
know.<br>
<br>
</span>This topic have happened several
times, and I also felt this kind of<br>
tool was very useful for Tempest users,
because Tempest contains 296<br>
options($ grep cfg * -R | grep Opt | wc
-l) now and it is difficult to<br>
set the configuration up.<br>
However, there is a big concern:<br>
If the script contain a bug and creates
the configuration which makes<br>
most tests skipped, we cannot do enough
tests on the gate.<br>
Tempest contains 1432 tests and
difficult to detect which tests are<br>
skipped as unexpected.<br>
Actually we faced unexpected skipped
tests on the gate before due to<br>
some bug, then the problem has been
fixed.<br>
But I can imagine this kind of problem
happens after implementing this<br>
kind of script.<br>
<br>
So now I am feeling Tempest users need
to know what cloud they want to<br>
test with Tempest, and need to know what
tests run with Tempest.<br>
Testers need to know what test
target/items they are testing basically.<br>
<br>
Thanks<br>
Ken Ohmichi<br>
<br>
---<br>
<span><br>
> ---<br>
> [1]<br>
> <a moz-do-not-send="true"
href="https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py"
rel="noreferrer" target="_blank">https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py</a><br>
> [2] <a moz-do-not-send="true"
href="https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator"
rel="noreferrer" target="_blank">https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator</a><br>
> [3] <a moz-do-not-send="true"
href="https://etherpad.openstack.org/p/mitaka-qa-priorities"
rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/mitaka-qa-priorities</a><br>
> [4] <a moz-do-not-send="true"
href="https://etherpad.openstack.org/p/tempest-cli-improvements"
rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/tempest-cli-improvements</a><br>
><br>
> <a moz-do-not-send="true"
href="https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst"
rel="noreferrer" target="_blank">https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst</a><br>
><br>
</span>>
__________________________________________________________________________<br>
> OpenStack Development Mailing List
(not for usage questions)<br>
> Unsubscribe: <a
moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a></a><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not
for usage questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
</div>
</div>
<span><font color="#888888">-- <br>
<div>
<div dir="ltr">Best regards,<br>
Andrey Kurilin.<br>
</div>
</div>
</font></span></div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">
<div dir="ltr">Best regards,<br>
Andrey Kurilin.<br>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>