[openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Yaroslav Lobankov ylobankov at mirantis.com
Thu Nov 26 13:23:58 UTC 2015


Hello everyone,

Yes, I am working on this now. We have some success already, but there is a
lot of work to do. Of course, some things don't work ideally. For example,
in [2] from the previous letter we have not 24 skipped tests, actually much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <akurilin at mirantis.com>
wrote:

> Hi!
> Boris P. and I tried to push a spec[1] for automation tempest config
> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
> have such tool:(
>
> >However, there is a big concern:
> >If the script contain a bug and creates the configuration which makes
> >most tests skipped, we cannot do enough tests on the gate.
> >Tempest contains 1432 tests and difficult to detect which tests are
> >skipped as unexpected.
>
> Yaroslav Lobankov is working on improvement for tempest config generator
> in Rally. Last time when we launch full tempest run[2], we got 1154 success
> tests and only 24 skipped. Also, there is a patch, which adds x-fail
> mechanism(it based on subunit-filter): you can transmit a file with test
> names + reasons and rally will modify results.
>
> [1] - https://review.openstack.org/#/c/94473/
>
> [2] -
> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>
> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
> wrote:
>
>> Hi Daniel,
>>
>> Thanks for pointing this up.
>>
>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org>:
>> > Hi All,
>> >
>> > As you might already know, within Red Hat's tempest fork, we do have one
>> > tempest configuration script which was built in the past by David Kranz
>> [1]
>> > and that's been actively used in our CI system. Regarding this topic,
>> I'm
>> > aware that quite some effort has been done in the past [2] and I would
>> like
>> > to complete the implementation of this blueprint/spec.
>> >
>> > My plan would be to have this script under the /tempest/cmd or
>> > /tempest/tools folder from tempest so it can be used to configure not
>> the
>> > tempest gate but any cloud we'd like to run tempest against.
>> >
>> > Adding the configuration script was discussed briefly at the Mitaka
>> summit
>> > in the QA Priorities meting [3]. I propose we use the existing etherpad
>> to
>> > continue the discussion around and tracking of implementing "tempest
>> > config-create" using the downstream config script as a starting point.
>> [4]
>> >
>> > If you have any questions, comments or opinion, please let me know.
>>
>> This topic have happened several times, and I also felt this kind of
>> tool was very useful for Tempest users, because Tempest contains 296
>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
>> set the configuration up.
>> However, there is a big concern:
>> If the script contain a bug and creates the configuration which makes
>> most tests skipped, we cannot do enough tests on the gate.
>> Tempest contains 1432 tests and difficult to detect which tests are
>> skipped as unexpected.
>> Actually we faced unexpected skipped tests on the gate before due to
>> some bug, then the problem has been fixed.
>> But I can imagine this kind of problem happens after implementing this
>> kind of script.
>>
>> So now I am feeling Tempest users need to know what cloud they want to
>> test with Tempest, and need to know what tests run with Tempest.
>> Testers need to know what test target/items they are testing basically.
>>
>> Thanks
>> Ken Ohmichi
>>
>> ---
>>
>> > ---
>> > [1]
>> >
>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>> > [2]
>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>> >
>> >
>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Best regards,
> Andrey Kurilin.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151126/4efbd4f7/attachment.html>


More information about the OpenStack-dev mailing list