Open Stack

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
have such tool:(

>However, there is a big concern:
>If the script contain a bug and creates the configuration which makes
>most tests skipped, we cannot do enough tests on the gate.
>Tempest contains 1432 tests and difficult to detect which tests are
>skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config generator in
Rally. Last time when we launch full tempest run[2], we got 1154 success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -
http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
wrote:

> Hi Daniel,
>
> Thanks for pointing this up.
>
> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org>:
> > Hi All,
> >
> > As you might already know, within Red Hat's tempest fork, we do have one
> > tempest configuration script which was built in the past by David Kranz
> [1]
> > and that's been actively used in our CI system. Regarding this topic, I'm
> > aware that quite some effort has been done in the past [2] and I would
> like
> > to complete the implementation of this blueprint/spec.
> >
> > My plan would be to have this script under the /tempest/cmd or
> > /tempest/tools folder from tempest so it can be used to configure not the
> > tempest gate but any cloud we'd like to run tempest against.
> >
> > Adding the configuration script was discussed briefly at the Mitaka
> summit
> > in the QA Priorities meting [3]. I propose we use the existing etherpad
> to
> > continue the discussion around and tracking of implementing "tempest
> > config-create" using the downstream config script as a starting point.
> [4]
> >
> > If you have any questions, comments or opinion, please let me know.
>
> This topic have happened several times, and I also felt this kind of
> tool was very useful for Tempest users, because Tempest contains 296
> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
> set the configuration up.
> However, there is a big concern:
> If the script contain a bug and creates the configuration which makes
> most tests skipped, we cannot do enough tests on the gate.
> Tempest contains 1432 tests and difficult to detect which tests are
> skipped as unexpected.
> Actually we faced unexpected skipped tests on the gate before due to
> some bug, then the problem has been fixed.
> But I can imagine this kind of problem happens after implementing this
> kind of script.
>
> So now I am feeling Tempest users need to know what cloud they want to
> test with Tempest, and need to know what tests run with Tempest.
> Testers need to know what test target/items they are testing basically.
>
> Thanks
> Ken Ohmichi
>
> ---
>
> > ---
> > [1]
> >
> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
> > [2]
> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
> >
> >
> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Best regards,
Andrey Kurilin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151126/ba4f6c1b/attachment.html>