Open Stack

Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is
people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a
proper conf file is generated. It doesn't have to run the tests, just
compare the output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is
within tempest. That said, if the Tempest community doesn't want this tool,
we'll have to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <jordan.pittier at scality.com
> wrote:

> Hi,
> I think this script is valuable to some users: Rally and Red Hat expressed
> their needs, they seem clear.
>
> This tool is far from bullet proof and if used blindly or in case of bugs,
> Tempest could be misconfigured. So, we could have this tool inside the
> Tempest repository (in the tools/) but not use it at all for the Gate.
>
> I am not sure I fully understand the resistance for this, if we don"t use
> this config generator for the gate, what's the risk ?
>
> Jordan
>
> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
> wrote:
>
>> 2015-11-27 15:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org>:
>> > I still do think that even if there are some issues addressed to the
>> > feature, such as skipping tests in the gate, the feature itself it's
>> still
>> > good -we just won't use it for the gates-
>> > Instead it'd be used as a wrapper for a user who would be interested on
>> > trying it against a real/reals clouds.
>> >
>> > Ken, do you really think a tempest user should know all tempest options?
>> > As you pointed out there are quite a few of them and even if they
>> should at
>> > least know their environment, this script would set a minimum acceptable
>> > default. Do you think PTL and Pre-PTL concerns that we spoke of would
>> still
>> > apply to that scenario?
>>
>> If Tempest users run part of tests of Tempest, they need to know the
>> options which are used with these tests only.
>> For example, current Tempest contains ironic API tests and the
>> corresponding options.
>> If users don't want to run these tests because the cloud don't support
>> ironic API, they don't need to know/setup these options.
>> I feel users need to know necessary options which are used on tests
>> they want, because they need to investigate the reason if facing a
>> problem during Tempest tests.
>>
>> Now Tempest options contain their default values, but you need a
>> script for changing them from the default.
>> Don't these default values work for your cloud at all?
>> If so, these values should be changed to better.
>>
>> Thanks
>> Ken Ohmichi
>>
>> ---
>>
>> > Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
>> > tempest-cli improvements? What do you think about this, Masayuki?
>> >
>> > Thanks for all your feedback! ;)
>> >
>> > El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>> >
>> > Sorry for wrong numbers. The bug-fix for issue with counters is merged.
>> > Correct numbers(latest result from rally's gate[1]):
>> >  - total number of executed tests: 1689
>> >  - success: 1155
>> >  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
>> > enable them)
>> >  - failed: 0
>> >
>> > [1] -
>> >
>> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
>> > [2] - https://review.openstack.org/#/c/250540/
>> >
>> > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
>> ylobankov at mirantis.com>
>> > wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> Yes, I am working on this now. We have some success already, but there
>> is
>> >> a lot of work to do. Of course, some things don't work ideally. For
>> example,
>> >> in [2] from the previous letter we have not 24 skipped tests, actually
>> much
>> >> more. So we have a bug somewhere :)
>> >>
>> >> Regards,
>> >> Yaroslav Lobankov.
>> >>
>> >> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <akurilin at mirantis.com
>> >
>> >> wrote:
>> >>>
>> >>> Hi!
>> >>> Boris P. and I tried to push a spec[1] for automation tempest config
>> >>> generator, but we did not succeed to merge it. Imo, qa-team doesn't
>> want to
>> >>> have such tool:(
>> >>>
>> >>> >However, there is a big concern:
>> >>> >If the script contain a bug and creates the configuration which makes
>> >>> >most tests skipped, we cannot do enough tests on the gate.
>> >>> >Tempest contains 1432 tests and difficult to detect which tests are
>> >>> >skipped as unexpected.
>> >>>
>> >>> Yaroslav Lobankov is working on improvement for tempest config
>> generator
>> >>> in Rally. Last time when we launch full tempest run[2], we got 1154
>> success
>> >>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>> >>> mechanism(it based on subunit-filter): you can transmit a file with
>> test
>> >>> names + reasons and rally will modify results.
>> >>>
>> >>> [1] - https://review.openstack.org/#/c/94473/
>> >>>
>> >>> [2] -
>> >>>
>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>> >>>
>> >>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <
>> ken1ohmichi at gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Daniel,
>> >>>>
>> >>>> Thanks for pointing this up.
>> >>>>
>> >>>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org
>> >:
>> >>>> > Hi All,
>> >>>> >
>> >>>> > As you might already know, within Red Hat's tempest fork, we do
>> have
>> >>>> > one
>> >>>> > tempest configuration script which was built in the past by David
>> >>>> > Kranz [1]
>> >>>> > and that's been actively used in our CI system. Regarding this
>> topic,
>> >>>> > I'm
>> >>>> > aware that quite some effort has been done in the past [2] and I
>> would
>> >>>> > like
>> >>>> > to complete the implementation of this blueprint/spec.
>> >>>> >
>> >>>> > My plan would be to have this script under the /tempest/cmd or
>> >>>> > /tempest/tools folder from tempest so it can be used to configure
>> not
>> >>>> > the
>> >>>> > tempest gate but any cloud we'd like to run tempest against.
>> >>>> >
>> >>>> > Adding the configuration script was discussed briefly at the Mitaka
>> >>>> > summit
>> >>>> > in the QA Priorities meting [3]. I propose we use the existing
>> >>>> > etherpad to
>> >>>> > continue the discussion around and tracking of implementing
>> "tempest
>> >>>> > config-create" using the downstream config script as a starting
>> point.
>> >>>> > [4]
>> >>>> >
>> >>>> > If you have any questions, comments or opinion, please let me know.
>> >>>>
>> >>>> This topic have happened several times, and I also felt this kind of
>> >>>> tool was very useful for Tempest users, because Tempest contains 296
>> >>>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult
>> to
>> >>>> set the configuration up.
>> >>>> However, there is a big concern:
>> >>>> If the script contain a bug and creates the configuration which makes
>> >>>> most tests skipped, we cannot do enough tests on the gate.
>> >>>> Tempest contains 1432 tests and difficult to detect which tests are
>> >>>> skipped as unexpected.
>> >>>> Actually we faced unexpected skipped tests on the gate before due to
>> >>>> some bug, then the problem has been fixed.
>> >>>> But I can imagine this kind of problem happens after implementing
>> this
>> >>>> kind of script.
>> >>>>
>> >>>> So now I am feeling Tempest users need to know what cloud they want
>> to
>> >>>> test with Tempest, and need to know what tests run with Tempest.
>> >>>> Testers need to know what test target/items they are testing
>> basically.
>> >>>>
>> >>>> Thanks
>> >>>> Ken Ohmichi
>> >>>>
>> >>>> ---
>> >>>>
>> >>>> > ---
>> >>>> > [1]
>> >>>> >
>> >>>> >
>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>> >>>> > [2]
>> >>>> >
>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>> >>>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>> >>>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>> >>>> >
>> >>>> >
>> >>>> >
>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>> >>>> >
>> >>>> >
>> >>>> >
>> __________________________________________________________________________
>> >>>> > OpenStack Development Mailing List (not for usage questions)
>> >>>> > Unsubscribe:
>> >>>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>> >
>> >>>>
>> >>>>
>> >>>>
>> __________________________________________________________________________
>> >>>> OpenStack Development Mailing List (not for usage questions)
>> >>>> Unsubscribe:
>> >>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best regards,
>> >>> Andrey Kurilin.
>> >>>
>> >>>
>> >>>
>> __________________________________________________________________________
>> >>> OpenStack Development Mailing List (not for usage questions)
>> >>> Unsubscribe:
>> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>
>> >>
>> >>
>> __________________________________________________________________________
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Andrey Kurilin.
>> >
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151129/904b6053/attachment.html>