Open Stack

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest options?
As you pointed out there are quite a few of them and even if they should
at least know their environment, this script would set a minimum
acceptable default. Do you think PTL and Pre-PTL concerns that we spoke
of would still apply to that scenario?

Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:
> Sorry for wrong numbers. The bug-fix for issue with counters is merged.
> Correct numbers(latest result from rally's gate[1]):
>  - total number of executed tests: 1689
>  - success: 1155
>  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2]
> should enable them)
>  - failed: 0
>
> [1] -
> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
> [2] - https://review.openstack.org/#/c/250540/
>
> On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov
> <ylobankov at mirantis.com <mailto:ylobankov at mirantis.com>> wrote:
>
>     Hello everyone,
>
>     Yes, I am working on this now. We have some success already, but
>     there is a lot of work to do. Of course, some things don't work
>     ideally. For example, in [2] from the previous letter we have not
>     24 skipped tests, actually much more. So we have a bug somewhere :)
>
>     Regards,
>     Yaroslav Lobankov.  
>
>     On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin
>     <akurilin at mirantis.com <mailto:akurilin at mirantis.com>> wrote:
>
>         Hi!
>         Boris P. and I tried to push a spec[1] for automation tempest
>         config generator, but we did not succeed to merge it. Imo,
>         qa-team doesn't want to have such tool:(
>
>         >However, there is a big concern:
>         >If the script contain a bug and creates the configuration
>         which makes
>         >most tests skipped, we cannot do enough tests on the gate.
>         >Tempest contains 1432 tests and difficult to detect which
>         tests are
>         >skipped as unexpected.
>
>         Yaroslav Lobankov is working on improvement for tempest config
>         generator in Rally. Last time when we launch full tempest
>         run[2], we got 1154 success tests and only 24 skipped. Also,
>         there is a patch, which adds x-fail mechanism(it based on
>         subunit-filter): you can transmit a file with test names +
>         reasons and rally will modify results.
>
>         [1] - https://review.openstack.org/#/c/94473/
>
>         [2] -
>         http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>
>         On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi
>         <ken1ohmichi at gmail.com <mailto:ken1ohmichi at gmail.com>> wrote:
>
>             Hi Daniel,
>
>             Thanks for pointing this up.
>
>             2015-11-25 1:40 GMT+09:00 Daniel Mellado
>             <daniel.mellado.es at ieee.org
>             <mailto:daniel.mellado.es at ieee.org>>:
>             > Hi All,
>             >
>             > As you might already know, within Red Hat's tempest
>             fork, we do have one
>             > tempest configuration script which was built in the past
>             by David Kranz [1]
>             > and that's been actively used in our CI system.
>             Regarding this topic, I'm
>             > aware that quite some effort has been done in the past
>             [2] and I would like
>             > to complete the implementation of this blueprint/spec.
>             >
>             > My plan would be to have this script under the
>             /tempest/cmd or
>             > /tempest/tools folder from tempest so it can be used to
>             configure not the
>             > tempest gate but any cloud we'd like to run tempest against.
>             >
>             > Adding the configuration script was discussed briefly at
>             the Mitaka summit
>             > in the QA Priorities meting [3]. I propose we use the
>             existing etherpad to
>             > continue the discussion around and tracking of
>             implementing "tempest
>             > config-create" using the downstream config script as a
>             starting point. [4]
>             >
>             > If you have any questions, comments or opinion, please
>             let me know.
>
>             This topic have happened several times, and I also felt
>             this kind of
>             tool was very useful for Tempest users, because Tempest
>             contains 296
>             options($ grep cfg * -R | grep Opt | wc -l) now and it is
>             difficult to
>             set the configuration up.
>             However, there is a big concern:
>             If the script contain a bug and creates the configuration
>             which makes
>             most tests skipped, we cannot do enough tests on the gate.
>             Tempest contains 1432 tests and difficult to detect which
>             tests are
>             skipped as unexpected.
>             Actually we faced unexpected skipped tests on the gate
>             before due to
>             some bug, then the problem has been fixed.
>             But I can imagine this kind of problem happens after
>             implementing this
>             kind of script.
>
>             So now I am feeling Tempest users need to know what cloud
>             they want to
>             test with Tempest, and need to know what tests run with
>             Tempest.
>             Testers need to know what test target/items they are
>             testing basically.
>
>             Thanks
>             Ken Ohmichi
>
>             ---
>
>             > ---
>             > [1]
>             >
>             https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>             > [2]
>             https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>             > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>             > [4]
>             https://etherpad.openstack.org/p/tempest-cli-improvements
>             >
>             >
>             https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>             >
>             >
>             __________________________________________________________________________
>             > OpenStack Development Mailing List (not for usage questions)
>             > Unsubscribe:
>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>             <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>             >
>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>             >
>
>             __________________________________________________________________________
>             OpenStack Development Mailing List (not for usage questions)
>             Unsubscribe:
>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>             <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>         -- 
>         Best regards,
>         Andrey Kurilin.
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> -- 
> Best regards,
> Andrey Kurilin.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151127/b48c6e67/attachment.html>