[openstack-dev] [Nova][QA] Disabling the v3 API tests in the gate

Sean Dague sean at dague.net
Thu Jun 12 16:10:23 UTC 2014

On 06/12/2014 12:02 PM, Matt Riedemann wrote:
> On 6/12/2014 10:51 AM, Matthew Treinish wrote:
>> On Fri, Jun 13, 2014 at 12:41:19AM +0930, Christopher Yeoh wrote:
>>> On Fri, Jun 13, 2014 at 12:25 AM, Dan Smith <dms at danplanet.com> wrote:
>>>> I think it'd be OK to move them to the experimental queue and a
>>>> periodic
>>>>> nightly job until the v2.1 stuff shakes out.  The v3 API is marked
>>>>> experimental right now so it seems fitting that it'd be running
>>>>> tests in
>>>>> the experimental queue until at least the spec is approved and
>>>>> microversioning starts happening in the code base.
>>>> I think this is reasonable. Continuing to run the full set of tests on
>>>> every patch for something we never expect to see the light of day
>>>> (in its
>>>> current form) seems wasteful to me. Plus, we're going to
>>>> (presumably) be
>>>> ramping up tests on v2.1, which means to me that we'll need to clear
>>>> out
>>>> some capacity to make room for that.
>>> Thats true, though I was suggesting as v2.1microversions rolls out 
>>> we drop
>>> the test out of v3 and move it to v2.1microversions testing, so
>>> there's no
>>> change in capacity required.
>> That's why I wasn't proposing that we rip the tests out of the tree.
>> I'm just
>> trying to weigh the benefit of leaving them enabled on every run against
>> the increased load they cause in an arguably overworked gate.
>>> Matt - how much of the time overhead is scenario tests? That's something
>>> that would have a lot less impact if moved to and experimental queue.
>>> Although the v3 api as a whole won't be officially exposed, the api
>>> tests
>>> test specific features fairly indepdently which are slated for
>>> v2.1microversions on a case by case basis and I don't want to see those
>>> regress. I guess my concern is how often the experimental queue
>>> results get
>>> really looked at and how hard/quick it is to revert when lots of stuff
>>> merges in a short period of time)
>> The scenario tests tend to be the slower tests in tempest. I have to
>> disagree
>> that removing them would have lower impact. The scenario tests provide
>> the best
>> functional verification, which is part of the reason we always have
>> failures in
>> the gate on them. While it would make the gate faster the decrease in
>> what were
>> testing isn't worth it. Also, for reference I pulled the test run
>> times that
>> were greater than 10sec out of a recent gate run:
>> http://paste.openstack.org/show/83827/
>> The experimental jobs aren't automatically run, they have to be manually
>> triggered by leaving a 'check experimental' comment. So for changes
>> that we want
>> to test the v3 api on a comment would have to left. To prevent
>> regression is why
>> we'd also have the nightly job, which I think is a better compromise
>> for the v3
>> tests while we wait to migrate them to the v2.1 microversion tests.
>> Another, option is that we make the v3 job run only on the check queue
>> and not
>> on the gate. But the benefits of that are slightly more limited,
>> because we'd
>> still be holding up the check queue.
>> -Matt Treinish
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Yeah the scenario tests need to stay, that's how we've exposed the two
> big ssh bugs in the last couple of weeks which are obvious issues at scale.
> I still think experimental/periodic is the way to go, not a hybrid of
> check-on/gate-off.  If we want to explicitly test v3 API changes we can
> do that with 'recheck experimental'.  Granted someone has to remember to
> run those, much like checking/rechecking 3rd party CI results.
> One issue I've had with the nightly periodic job is finding out where
> the results are in an easy to consume format.  Is there something out
> there for that?  I'm thinking specifically of things we've turned off in
> the gate before like multi-backend volume tests and
> allow_tenant_isolation=False.

It's getting emailed to the otherwise defunct openstack-qa list.
Subscribe there for nightlies.

Also agreed, the scenario tests find and prevent *tons* of real issues.
Those have to stay. There is a reason we use them in the smoke runs for
grenade, they are a very solid sniff test of real working.

I also think by policy we should probably pull v3 out of the main job,
as it's not a stable API. We've had issues in Tempest with people
landing tests, then trying to go and change the API. The biggest issue
in taking branchless tempest back to stable/havana was Nova v3 API, as
it's actually quite different in havana than icehouse.

We have a chicken / egg challenge in testing experimental APIs which
will need to get resolved, but for now I think turning off v3 is the
right approach.


Sean Dague

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140612/3f847cc4/attachment.pgp>

More information about the OpenStack-dev mailing list