Open Stack

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
 - total number of executed tests: 1689
 - success: 1155
 - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
enable them)
 - failed: 0

[1] -
http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <ylobankov at mirantis.com>
wrote:

> Hello everyone,
>
> Yes, I am working on this now. We have some success already, but there is
> a lot of work to do. Of course, some things don't work ideally. For
> example, in [2] from the previous letter we have not 24 skipped tests,
> actually much more. So we have a bug somewhere :)
>
> Regards,
> Yaroslav Lobankov.
>
> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <akurilin at mirantis.com>
> wrote:
>
>> Hi!
>> Boris P. and I tried to push a spec[1] for automation tempest config
>> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
>> have such tool:(
>>
>> >However, there is a big concern:
>> >If the script contain a bug and creates the configuration which makes
>> >most tests skipped, we cannot do enough tests on the gate.
>> >Tempest contains 1432 tests and difficult to detect which tests are
>> >skipped as unexpected.
>>
>> Yaroslav Lobankov is working on improvement for tempest config generator
>> in Rally. Last time when we launch full tempest run[2], we got 1154 success
>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>> mechanism(it based on subunit-filter): you can transmit a file with test
>> names + reasons and rally will modify results.
>>
>> [1] - https://review.openstack.org/#/c/94473/
>>
>> [2] -
>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>>
>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <ken1ohmichi at gmail.com>
>> wrote:
>>
>>> Hi Daniel,
>>>
>>> Thanks for pointing this up.
>>>
>>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado.es at ieee.org>:
>>> > Hi All,
>>> >
>>> > As you might already know, within Red Hat's tempest fork, we do have
>>> one
>>> > tempest configuration script which was built in the past by David
>>> Kranz [1]
>>> > and that's been actively used in our CI system. Regarding this topic,
>>> I'm
>>> > aware that quite some effort has been done in the past [2] and I would
>>> like
>>> > to complete the implementation of this blueprint/spec.
>>> >
>>> > My plan would be to have this script under the /tempest/cmd or
>>> > /tempest/tools folder from tempest so it can be used to configure not
>>> the
>>> > tempest gate but any cloud we'd like to run tempest against.
>>> >
>>> > Adding the configuration script was discussed briefly at the Mitaka
>>> summit
>>> > in the QA Priorities meting [3]. I propose we use the existing
>>> etherpad to
>>> > continue the discussion around and tracking of implementing "tempest
>>> > config-create" using the downstream config script as a starting point.
>>> [4]
>>> >
>>> > If you have any questions, comments or opinion, please let me know.
>>>
>>> This topic have happened several times, and I also felt this kind of
>>> tool was very useful for Tempest users, because Tempest contains 296
>>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
>>> set the configuration up.
>>> However, there is a big concern:
>>> If the script contain a bug and creates the configuration which makes
>>> most tests skipped, we cannot do enough tests on the gate.
>>> Tempest contains 1432 tests and difficult to detect which tests are
>>> skipped as unexpected.
>>> Actually we faced unexpected skipped tests on the gate before due to
>>> some bug, then the problem has been fixed.
>>> But I can imagine this kind of problem happens after implementing this
>>> kind of script.
>>>
>>> So now I am feeling Tempest users need to know what cloud they want to
>>> test with Tempest, and need to know what tests run with Tempest.
>>> Testers need to know what test target/items they are testing basically.
>>>
>>> Thanks
>>> Ken Ohmichi
>>>
>>> ---
>>>
>>> > ---
>>> > [1]
>>> >
>>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>>> > [2]
>>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>>> >
>>> >
>>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>>> >
>>> >
>>> __________________________________________________________________________
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> >
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey Kurilin.
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Best regards,
Andrey Kurilin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151127/b803391a/attachment.html>