[openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Ken'ichi Ohmichi ken1ohmichi at gmail.com
Fri Nov 27 07:05:06 UTC 2015


2015-11-27 15:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org>:
> I still do think that even if there are some issues addressed to the
> feature, such as skipping tests in the gate, the feature itself it's still
> good -we just won't use it for the gates-
> Instead it'd be used as a wrapper for a user who would be interested on
> trying it against a real/reals clouds.
>
> Ken, do you really think a tempest user should know all tempest options?
> As you pointed out there are quite a few of them and even if they should at
> least know their environment, this script would set a minimum acceptable
> default. Do you think PTL and Pre-PTL concerns that we spoke of would still
> apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi

---

> Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
> tempest-cli improvements? What do you think about this, Masayuki?
>
> Thanks for all your feedback! ;)
>
> El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>
> Sorry for wrong numbers. The bug-fix for issue with counters is merged.
> Correct numbers(latest result from rally's gate[1]):
>  - total number of executed tests: 1689
>  - success: 1155
>  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
> enable them)
>  - failed: 0
>
> [1] -
> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
> [2] - https://review.openstack.org/#/c/250540/
>
> On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <ylobankov at mirantis.com>
> wrote:
>>
>> Hello everyone,
>>
>> Yes, I am working on this now. We have some success already, but there is
>> a lot of work to do. Of course, some things don't work ideally. For example,
>> in [2] from the previous letter we have not 24 skipped tests, actually much
>> more. So we have a bug somewhere :)
>>
>> Regards,
>> Yaroslav Lobankov.
>>
>> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <akurilin at mirantis.com>
>> wrote:
>>>
>>> Hi!
>>> Boris P. and I tried to push a spec[1] for automation tempest config
>>> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
>>> have such tool:(
>>>
>>> >However, there is a big concern:
>>> >If the script contain a bug and creates the configuration which makes
>>> >most tests skipped, we cannot do enough tests on the gate.
>>> >Tempest contains 1432 tests and difficult to detect which tests are
>>> >skipped as unexpected.
>>>
>>> Yaroslav Lobankov is working on improvement for tempest config generator
>>> in Rally. Last time when we launch full tempest run[2], we got 1154 success
>>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>>> mechanism(it based on subunit-filter): you can transmit a file with test
>>> names + reasons and rally will modify results.
>>>
>>> [1] - https://review.openstack.org/#/c/94473/
>>>
>>> [2] -
>>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>>>
>>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
>>> wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> Thanks for pointing this up.
>>>>
>>>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org>:
>>>> > Hi All,
>>>> >
>>>> > As you might already know, within Red Hat's tempest fork, we do have
>>>> > one
>>>> > tempest configuration script which was built in the past by David
>>>> > Kranz [1]
>>>> > and that's been actively used in our CI system. Regarding this topic,
>>>> > I'm
>>>> > aware that quite some effort has been done in the past [2] and I would
>>>> > like
>>>> > to complete the implementation of this blueprint/spec.
>>>> >
>>>> > My plan would be to have this script under the /tempest/cmd or
>>>> > /tempest/tools folder from tempest so it can be used to configure not
>>>> > the
>>>> > tempest gate but any cloud we'd like to run tempest against.
>>>> >
>>>> > Adding the configuration script was discussed briefly at the Mitaka
>>>> > summit
>>>> > in the QA Priorities meting [3]. I propose we use the existing
>>>> > etherpad to
>>>> > continue the discussion around and tracking of implementing "tempest
>>>> > config-create" using the downstream config script as a starting point.
>>>> > [4]
>>>> >
>>>> > If you have any questions, comments or opinion, please let me know.
>>>>
>>>> This topic have happened several times, and I also felt this kind of
>>>> tool was very useful for Tempest users, because Tempest contains 296
>>>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
>>>> set the configuration up.
>>>> However, there is a big concern:
>>>> If the script contain a bug and creates the configuration which makes
>>>> most tests skipped, we cannot do enough tests on the gate.
>>>> Tempest contains 1432 tests and difficult to detect which tests are
>>>> skipped as unexpected.
>>>> Actually we faced unexpected skipped tests on the gate before due to
>>>> some bug, then the problem has been fixed.
>>>> But I can imagine this kind of problem happens after implementing this
>>>> kind of script.
>>>>
>>>> So now I am feeling Tempest users need to know what cloud they want to
>>>> test with Tempest, and need to know what tests run with Tempest.
>>>> Testers need to know what test target/items they are testing basically.
>>>>
>>>> Thanks
>>>> Ken Ohmichi
>>>>
>>>> ---
>>>>
>>>> > ---
>>>> > [1]
>>>> >
>>>> > https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>>>> > [2]
>>>> > https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>>>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>>>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>>>> >
>>>> >
>>>> > https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>>>> >
>>>> >
>>>> > __________________________________________________________________________
>>>> > OpenStack Development Mailing List (not for usage questions)
>>>> > Unsubscribe:
>>>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey Kurilin.
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Best regards,
> Andrey Kurilin.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



More information about the OpenStack-dev mailing list