Open Stack

Dell - Internal Use - Confidential
 I don’t really understand the friction, it’s incredibly useful, especially for new users.

Use it or don’t use it it’s up to you,  nowhere is it implied that the user is forced to configure tempest with the tool.

Without a tool like this it is complex for new users to get up and going with tempest.  It is also more difficult for other parties to integrate with Tempest, Rally had to write their own configuration tooling for instance.

Thanks,
dp



From: Andrea Frittoli [mailto:andrea.frittoli at gmail.com]
Sent: Monday, November 30, 2015 8:34 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Looking at the tool, it seems to me that is servers a combination of functions:
- provision test resources
- support for distribution specific / cloud specific overrides to default
- support for configuration override via CLI
- discovery of configuration

Test resource provisioning is something that I agree is useful to have.
We plan in Mitaka to separate out of tempest.conf the configuration of test resources, and have a CLI tool to provision them [0]. We could re-use code from this tool to achieve that.

Support for distribution specific / cloud specific overrides is also something that is useful. In clouds where I control the deployment process I inject extra configs in tempest.conf based on the deployment options. In clouds where I don't, I maintain a partial tempest.conf with the list of options which I expects I have to modify to match the target cloud.

That's pretty easy to achieve though - simply append the extra configs to the bottom of tempest.conf and it's done. Duplicate configuration options are not an issue, the last one wins. Still we could support specifying a number of configuration files to the non-yet implemented "tempest run" command.

Support for configuration override via CLI is something that I can see it can be useful during development or troubleshooting, we could support that as options of the non-yet implemented "tempest run" command.

The last point is discovery - I believe we should use that only as we use it today in the gate - i.e. fail fast if the generated configuration does not match what can be discovered from the cloud.

So I would not get the script as is into tempest, but I think many of the functions implemented by it can fit into tempest - and some are already there.

andrea

[0] https://review.openstack.org/#/c/173334/

On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa <masayuki.igawa at gmail.com<mailto:masayuki.igawa at gmail.com>> wrote:
Hi,

I agree with Ken'ichi's opinion, basically. Tempest users should know "what do we test for?" and we shouldn't discover values that we test for automatically.
If users seems that "My current cloud is good. This is what I expect.", discovering function could work. But I suppose many of users would use tempest-config-generator for a new their cloud. So I feel the tool users could be misunderstanding easily.

But I also think that tempest users don't need to know all of the configurations.
So, how about something like introducing "a configuration wizard" for tempest configuration?
This is a just idea, though..


Anyway, if you have the motivation to introduce tempest-config, how about writing a spec for the feature for a concrete discussion?
(I think we should have agreements about the target issues, users, solutions, etc.)

Best Regards,
-- Masayuki Igawa

2015年11月29日(日) 22:07 Yair Fried <yfried at redhat.com<mailto:yfried at redhat.com>>:
Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a proper conf file is generated. It doesn't have to run the tests, just compare the output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is within tempest. That said, if the Tempest community doesn't want this tool, we'll have to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <jordan.pittier at scality.com<mailto:jordan.pittier at scality.com>> wrote:
Hi,
I think this script is valuable to some users: Rally and Red Hat expressed their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of bugs, Tempest could be misconfigured. So, we could have this tool inside the Tempest repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t use this config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com<mailto:ken1ohmichi at gmail.com>> wrote:
2015-11-27 15:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org<mailto:daniel.mellado.es at ieee.org>>:
> I still do think that even if there are some issues addressed to the
> feature, such as skipping tests in the gate, the feature itself it's still
> good -we just won't use it for the gates-
> Instead it'd be used as a wrapper for a user who would be interested on
> trying it against a real/reals clouds.
>
> Ken, do you really think a tempest user should know all tempest options?
> As you pointed out there are quite a few of them and even if they should at
> least know their environment, this script would set a minimum acceptable
> default. Do you think PTL and Pre-PTL concerns that we spoke of would still
> apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi

---

> Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
> tempest-cli improvements? What do you think about this, Masayuki?
>
> Thanks for all your feedback! ;)
>
> El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>
> Sorry for wrong numbers. The bug-fix for issue with counters is merged.
> Correct numbers(latest result from rally's gate[1]):
>  - total number of executed tests: 1689
>  - success: 1155
>  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
> enable them)
>  - failed: 0
>
> [1] -
> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
> [2] - https://review.openstack.org/#/c/250540/
>
> On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <ylobankov at mirantis.com<mailto:ylobankov at mirantis.com>>
> wrote:
>>
>> Hello everyone,
>>
>> Yes, I am working on this now. We have some success already, but there is
>> a lot of work to do. Of course, some things don't work ideally. For example,
>> in [2] from the previous letter we have not 24 skipped tests, actually much
>> more. So we have a bug somewhere :)
>>
>> Regards,
>> Yaroslav Lobankov.
>>
>> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <akurilin at mirantis.com<mailto:akurilin at mirantis.com>>
>> wrote:
>>>
>>> Hi!
>>> Boris P. and I tried to push a spec[1] for automation tempest config
>>> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
>>> have such tool:(
>>>
>>> >However, there is a big concern:
>>> >If the script contain a bug and creates the configuration which makes
>>> >most tests skipped, we cannot do enough tests on the gate.
>>> >Tempest contains 1432 tests and difficult to detect which tests are
>>> >skipped as unexpected.
>>>
>>> Yaroslav Lobankov is working on improvement for tempest config generator
>>> in Rally. Last time when we launch full tempest run[2], we got 1154 success
>>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>>> mechanism(it based on subunit-filter): you can transmit a file with test
>>> names + reasons and rally will modify results.
>>>
>>> [1] - https://review.openstack.org/#/c/94473/
>>>
>>> [2] -
>>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>>>
>>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com<mailto:ken1ohmichi at gmail.com>>
>>> wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> Thanks for pointing this up.
>>>>
>>>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org<mailto:daniel.mellado.es at ieee.org>>:
>>>> > Hi All,
>>>> >
>>>> > As you might already know, within Red Hat's tempest fork, we do have
>>>> > one
>>>> > tempest configuration script which was built in the past by David
>>>> > Kranz [1]
>>>> > and that's been actively used in our CI system. Regarding this topic,
>>>> > I'm
>>>> > aware that quite some effort has been done in the past [2] and I would
>>>> > like
>>>> > to complete the implementation of this blueprint/spec.
>>>> >
>>>> > My plan would be to have this script under the /tempest/cmd or
>>>> > /tempest/tools folder from tempest so it can be used to configure not
>>>> > the
>>>> > tempest gate but any cloud we'd like to run tempest against.
>>>> >
>>>> > Adding the configuration script was discussed briefly at the Mitaka
>>>> > summit
>>>> > in the QA Priorities meting [3]. I propose we use the existing
>>>> > etherpad to
>>>> > continue the discussion around and tracking of implementing "tempest
>>>> > config-create" using the downstream config script as a starting point.
>>>> > [4]
>>>> >
>>>> > If you have any questions, comments or opinion, please let me know.
>>>>
>>>> This topic have happened several times, and I also felt this kind of
>>>> tool was very useful for Tempest users, because Tempest contains 296
>>>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
>>>> set the configuration up.
>>>> However, there is a big concern:
>>>> If the script contain a bug and creates the configuration which makes
>>>> most tests skipped, we cannot do enough tests on the gate.
>>>> Tempest contains 1432 tests and difficult to detect which tests are
>>>> skipped as unexpected.
>>>> Actually we faced unexpected skipped tests on the gate before due to
>>>> some bug, then the problem has been fixed.
>>>> But I can imagine this kind of problem happens after implementing this
>>>> kind of script.
>>>>
>>>> So now I am feeling Tempest users need to know what cloud they want to
>>>> test with Tempest, and need to know what tests run with Tempest.
>>>> Testers need to know what test target/items they are testing basically.
>>>>
>>>> Thanks
>>>> Ken Ohmichi
>>>>
>>>> ---
>>>>
>>>> > ---
>>>> > [1]
>>>> >
>>>> > https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>>>> > [2]
>>>> > https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>>>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>>>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>>>> >
>>>> >
>>>> > https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>>>> >
>>>> >
>>>> > __________________________________________________________________________
>>>> > OpenStack Development Mailing List (not for usage questions)
>>>> > Unsubscribe:
>>>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey Kurilin.
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Best regards,
> Andrey Kurilin.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
-- Masayuki Igawa
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151130/0802dac2/attachment.html>