Open Stack

Looking at the tool, it seems to me that is servers a combination of
functions:
- provision test resources
- support for distribution specific / cloud specific overrides to default
- support for configuration override via CLI
- discovery of configuration

Test resource provisioning is something that I agree is useful to have.
We plan in Mitaka to separate out of tempest.conf the configuration of test
resources, and have a CLI tool to provision them [0]. We could re-use code
from this tool to achieve that.

Support for distribution specific / cloud specific overrides is also
something that is useful. In clouds where I control the deployment process
I inject extra configs in tempest.conf based on the deployment options. In
clouds where I don't, I maintain a partial tempest.conf with the list of
options which I expects I have to modify to match the target cloud.

That's pretty easy to achieve though - simply append the extra configs to
the bottom of tempest.conf and it's done. Duplicate configuration options
are not an issue, the last one wins. Still we could support specifying a
number of configuration files to the non-yet implemented "tempest run"
command.

Support for configuration override via CLI is something that I can see it
can be useful during development or troubleshooting, we could support that
as options of the non-yet implemented "tempest run" command.

The last point is discovery - I believe we should use that only as we use
it today in the gate - i.e. fail fast if the generated configuration does
not match what can be discovered from the cloud.

So I would not get the script as is into tempest, but I think many of the
functions implemented by it can fit into tempest - and some are already
there.

andrea

[0] https://review.openstack.org/#/c/173334/

On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa <masayuki.igawa at gmail.com>
wrote:

> Hi,
>
> I agree with Ken'ichi's opinion, basically. Tempest users should know
> "what do we test for?" and we shouldn't discover values that we test for
> automatically.
> If users seems that "My current cloud is good. This is what I expect.",
> discovering function could work. But I suppose many of users would use
> tempest-config-generator for a new their cloud. So I feel the tool users
> could be misunderstanding easily.
>
> But I also think that tempest users don't need to know all of the
> configurations.
> So, how about something like introducing "a configuration wizard" for
> tempest configuration?
> This is a just idea, though..
>
>
> Anyway, if you have the motivation to introduce tempest-config, how about
> writing a spec for the feature for a concrete discussion?
> (I think we should have agreements about the target issues, users,
> solutions, etc.)
>
> Best Regards,
> -- Masayuki Igawa
>
>
> 2015年11月29日(日) 22:07 Yair Fried <yfried at redhat.com>:
>
>> Hi,
>> I agree with Jordan.
>> We don't have to use the tool as part of the gate. It's target audience
>> is people and not CI systems. More specifically - new users.
>> However, we could add a gate (or a few) for the tool that makes sure a
>> proper conf file is generated. It doesn't have to run the tests, just
>> compare the output of the script to the conf generated by devstack.
>>
>> Re Rally - I believe the best place for tempest configuration script is
>> within tempest. That said, if the Tempest community doesn't want this tool,
>> we'll have to settle for the Rally solution.
>>
>> Regards
>> Yair
>>
>> On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <
>> jordan.pittier at scality.com> wrote:
>>
>>> Hi,
>>> I think this script is valuable to some users: Rally and Red Hat
>>> expressed their needs, they seem clear.
>>>
>>> This tool is far from bullet proof and if used blindly or in case of
>>> bugs, Tempest could be misconfigured. So, we could have this tool inside
>>> the Tempest repository (in the tools/) but not use it at all for the Gate.
>>>
>>> I am not sure I fully understand the resistance for this, if we don"t
>>> use this config generator for the gate, what's the risk ?
>>>
>>> Jordan
>>>
>>> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com
>>> > wrote:
>>>
>>>> 2015-11-27 15:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org>:
>>>> > I still do think that even if there are some issues addressed to the
>>>> > feature, such as skipping tests in the gate, the feature itself it's
>>>> still
>>>> > good -we just won't use it for the gates-
>>>> > Instead it'd be used as a wrapper for a user who would be interested
>>>> on
>>>> > trying it against a real/reals clouds.
>>>> >
>>>> > Ken, do you really think a tempest user should know all tempest
>>>> options?
>>>> > As you pointed out there are quite a few of them and even if they
>>>> should at
>>>> > least know their environment, this script would set a minimum
>>>> acceptable
>>>> > default. Do you think PTL and Pre-PTL concerns that we spoke of would
>>>> still
>>>> > apply to that scenario?
>>>>
>>>> If Tempest users run part of tests of Tempest, they need to know the
>>>> options which are used with these tests only.
>>>> For example, current Tempest contains ironic API tests and the
>>>> corresponding options.
>>>> If users don't want to run these tests because the cloud don't support
>>>> ironic API, they don't need to know/setup these options.
>>>> I feel users need to know necessary options which are used on tests
>>>> they want, because they need to investigate the reason if facing a
>>>> problem during Tempest tests.
>>>>
>>>> Now Tempest options contain their default values, but you need a
>>>> script for changing them from the default.
>>>> Don't these default values work for your cloud at all?
>>>> If so, these values should be changed to better.
>>>>
>>>> Thanks
>>>> Ken Ohmichi
>>>>
>>>> ---
>>>>
>>>> > Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it
>>>> to
>>>> > tempest-cli improvements? What do you think about this, Masayuki?
>>>> >
>>>> > Thanks for all your feedback! ;)
>>>> >
>>>> > El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>>>> >
>>>> > Sorry for wrong numbers. The bug-fix for issue with counters is
>>>> merged.
>>>> > Correct numbers(latest result from rally's gate[1]):
>>>> >  - total number of executed tests: 1689
>>>> >  - success: 1155
>>>> >  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2]
>>>> should
>>>> > enable them)
>>>> >  - failed: 0
>>>> >
>>>> > [1] -
>>>> >
>>>> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
>>>> > [2] - https://review.openstack.org/#/c/250540/
>>>> >
>>>> > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
>>>> ylobankov at mirantis.com>
>>>> > wrote:
>>>> >>
>>>> >> Hello everyone,
>>>> >>
>>>> >> Yes, I am working on this now. We have some success already, but
>>>> there is
>>>> >> a lot of work to do. Of course, some things don't work ideally. For
>>>> example,
>>>> >> in [2] from the previous letter we have not 24 skipped tests,
>>>> actually much
>>>> >> more. So we have a bug somewhere :)
>>>> >>
>>>> >> Regards,
>>>> >> Yaroslav Lobankov.
>>>> >>
>>>> >> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <
>>>> akurilin at mirantis.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Hi!
>>>> >>> Boris P. and I tried to push a spec[1] for automation tempest config
>>>> >>> generator, but we did not succeed to merge it. Imo, qa-team doesn't
>>>> want to
>>>> >>> have such tool:(
>>>> >>>
>>>> >>> >However, there is a big concern:
>>>> >>> >If the script contain a bug and creates the configuration which
>>>> makes
>>>> >>> >most tests skipped, we cannot do enough tests on the gate.
>>>> >>> >Tempest contains 1432 tests and difficult to detect which tests are
>>>> >>> >skipped as unexpected.
>>>> >>>
>>>> >>> Yaroslav Lobankov is working on improvement for tempest config
>>>> generator
>>>> >>> in Rally. Last time when we launch full tempest run[2], we got 1154
>>>> success
>>>> >>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>>>> >>> mechanism(it based on subunit-filter): you can transmit a file with
>>>> test
>>>> >>> names + reasons and rally will modify results.
>>>> >>>
>>>> >>> [1] - https://review.openstack.org/#/c/94473/
>>>> >>>
>>>> >>> [2] -
>>>> >>>
>>>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>>>> >>>
>>>> >>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <
>>>> ken1ohmichi at gmail.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> Hi Daniel,
>>>> >>>>
>>>> >>>> Thanks for pointing this up.
>>>> >>>>
>>>> >>>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <
>>>> daniel.mellado.es at ieee.org>:
>>>> >>>> > Hi All,
>>>> >>>> >
>>>> >>>> > As you might already know, within Red Hat's tempest fork, we do
>>>> have
>>>> >>>> > one
>>>> >>>> > tempest configuration script which was built in the past by David
>>>> >>>> > Kranz [1]
>>>> >>>> > and that's been actively used in our CI system. Regarding this
>>>> topic,
>>>> >>>> > I'm
>>>> >>>> > aware that quite some effort has been done in the past [2] and I
>>>> would
>>>> >>>> > like
>>>> >>>> > to complete the implementation of this blueprint/spec.
>>>> >>>> >
>>>> >>>> > My plan would be to have this script under the /tempest/cmd or
>>>> >>>> > /tempest/tools folder from tempest so it can be used to
>>>> configure not
>>>> >>>> > the
>>>> >>>> > tempest gate but any cloud we'd like to run tempest against.
>>>> >>>> >
>>>> >>>> > Adding the configuration script was discussed briefly at the
>>>> Mitaka
>>>> >>>> > summit
>>>> >>>> > in the QA Priorities meting [3]. I propose we use the existing
>>>> >>>> > etherpad to
>>>> >>>> > continue the discussion around and tracking of implementing
>>>> "tempest
>>>> >>>> > config-create" using the downstream config script as a starting
>>>> point.
>>>> >>>> > [4]
>>>> >>>> >
>>>> >>>> > If you have any questions, comments or opinion, please let me
>>>> know.
>>>> >>>>
>>>> >>>> This topic have happened several times, and I also felt this kind
>>>> of
>>>> >>>> tool was very useful for Tempest users, because Tempest contains
>>>> 296
>>>> >>>> options($ grep cfg * -R | grep Opt | wc -l) now and it is
>>>> difficult to
>>>> >>>> set the configuration up.
>>>> >>>> However, there is a big concern:
>>>> >>>> If the script contain a bug and creates the configuration which
>>>> makes
>>>> >>>> most tests skipped, we cannot do enough tests on the gate.
>>>> >>>> Tempest contains 1432 tests and difficult to detect which tests are
>>>> >>>> skipped as unexpected.
>>>> >>>> Actually we faced unexpected skipped tests on the gate before due
>>>> to
>>>> >>>> some bug, then the problem has been fixed.
>>>> >>>> But I can imagine this kind of problem happens after implementing
>>>> this
>>>> >>>> kind of script.
>>>> >>>>
>>>> >>>> So now I am feeling Tempest users need to know what cloud they
>>>> want to
>>>> >>>> test with Tempest, and need to know what tests run with Tempest.
>>>> >>>> Testers need to know what test target/items they are testing
>>>> basically.
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Ken Ohmichi
>>>> >>>>
>>>> >>>> ---
>>>> >>>>
>>>> >>>> > ---
>>>> >>>> > [1]
>>>> >>>> >
>>>> >>>> >
>>>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>>>> >>>> > [2]
>>>> >>>> >
>>>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>>>> >>>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>>>> >>>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> __________________________________________________________________________
>>>> >>>> > OpenStack Development Mailing List (not for usage questions)
>>>> >>>> > Unsubscribe:
>>>> >>>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>>> >
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>>> >
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> __________________________________________________________________________
>>>> >>>> OpenStack Development Mailing List (not for usage questions)
>>>> >>>> Unsubscribe:
>>>> >>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Best regards,
>>>> >>> Andrey Kurilin.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> __________________________________________________________________________
>>>> >>> OpenStack Development Mailing List (not for usage questions)
>>>> >>> Unsubscribe:
>>>> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> __________________________________________________________________________
>>>> >> OpenStack Development Mailing List (not for usage questions)
>>>> >> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Best regards,
>>>> > Andrey Kurilin.
>>>> >
>>>> >
>>>> >
>>>> __________________________________________________________________________
>>>> > OpenStack Development Mailing List (not for usage questions)
>>>> > Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >
>>>> >
>>>> >
>>>> >
>>>> __________________________________________________________________________
>>>> > OpenStack Development Mailing List (not for usage questions)
>>>> > Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> --
> -- Masayuki Igawa
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151130/5b2d5a59/attachment.html>