Open Stack

On Jun 12, 2014 9:03 AM, "Matt Riedemann" <mriedem at linux.vnet.ibm.com>
wrote:
>
>
>
> On 6/12/2014 10:51 AM, Matthew Treinish wrote:
>>
>> On Fri, Jun 13, 2014 at 12:41:19AM +0930, Christopher Yeoh wrote:
>>>
>>> On Fri, Jun 13, 2014 at 12:25 AM, Dan Smith <dms at danplanet.com> wrote:
>>>
>>>> I think it'd be OK to move them to the experimental queue and a
periodic
>>>>>
>>>>> nightly job until the v2.1 stuff shakes out.  The v3 API is marked
>>>>> experimental right now so it seems fitting that it'd be running tests
in
>>>>> the experimental queue until at least the spec is approved and
>>>>> microversioning starts happening in the code base.
>>>>>
>>>>
>>>> I think this is reasonable. Continuing to run the full set of tests on
>>>> every patch for something we never expect to see the light of day (in
its
>>>> current form) seems wasteful to me. Plus, we're going to (presumably)
be
>>>> ramping up tests on v2.1, which means to me that we'll need to clear
out
>>>> some capacity to make room for that.
>>>>
>>>>
>>> Thats true, though I was suggesting as v2.1microversions rolls out  we
drop
>>> the test out of v3 and move it to v2.1microversions testing, so there's
no
>>> change in capacity required.
>>
>>
>> That's why I wasn't proposing that we rip the tests out of the tree. I'm
just
>> trying to weigh the benefit of leaving them enabled on every run against
>> the increased load they cause in an arguably overworked gate.
>>
>>>
>>> Matt - how much of the time overhead is scenario tests? That's something
>>> that would have a lot less impact if moved to and experimental queue.
>>> Although the v3 api as a whole won't be officially exposed, the api
tests
>>> test specific features fairly indepdently which are slated for
>>> v2.1microversions on a case by case basis and I don't want to see those
>>> regress. I guess my concern is how often the experimental queue results
get
>>> really looked at and how hard/quick it is to revert when lots of stuff
>>> merges in a short period of time)
>>
>>
>> The scenario tests tend to be the slower tests in tempest. I have to
disagree
>> that removing them would have lower impact. The scenario tests provide
the best
>> functional verification, which is part of the reason we always have
failures in
>> the gate on them. While it would make the gate faster the decrease in
what were
>> testing isn't worth it. Also, for reference I pulled the test run times
that
>> were greater than 10sec out of a recent gate run:
>> http://paste.openstack.org/show/83827/
>>
>> The experimental jobs aren't automatically run, they have to be manually
>> triggered by leaving a 'check experimental' comment. So for changes that
we want
>> to test the v3 api on a comment would have to left. To prevent
regression is why
>> we'd also have the nightly job, which I think is a better compromise for
the v3
>> tests while we wait to migrate them to the v2.1 microversion tests.
>>
>> Another, option is that we make the v3 job run only on the check queue
and not
>> on the gate. But the benefits of that are slightly more limited, because
we'd
>> still be holding up the check queue.
>>
>> -Matt Treinish
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> Yeah the scenario tests need to stay, that's how we've exposed the two
big ssh bugs in the last couple of weeks which are obvious issues at scale.
>
> I still think experimental/periodic is the way to go, not a hybrid of
check-on/gate-off.  If we want to explicitly test v3 API changes we can do
that with 'recheck experimental'.  Granted someone has to remember to run
those, much like checking/rechecking 3rd party CI results.

++

>
> One issue I've had with the nightly periodic job is finding out where the
results are in an easy to consume format.  Is there something out there for
that?  I'm thinking specifically of things we've turned off in the gate
before like multi-backend volume tests and allow_tenant_isolation=False.
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140612/8a23d143/attachment.html>