Open Stack

On 12 June 2014 17:10, Sean Dague <sean at dague.net> wrote:
> On 06/12/2014 12:02 PM, Matt Riedemann wrote:
>>
>>
>> On 6/12/2014 10:51 AM, Matthew Treinish wrote:
>>> On Fri, Jun 13, 2014 at 12:41:19AM +0930, Christopher Yeoh wrote:
>>>> On Fri, Jun 13, 2014 at 12:25 AM, Dan Smith <dms at danplanet.com> wrote:
>>>>
>>>>> I think it'd be OK to move them to the experimental queue and a
>>>>> periodic
>>>>>> nightly job until the v2.1 stuff shakes out.  The v3 API is marked
>>>>>> experimental right now so it seems fitting that it'd be running
>>>>>> tests in
>>>>>> the experimental queue until at least the spec is approved and
>>>>>> microversioning starts happening in the code base.
>>>>>>
>>>>>
>>>>> I think this is reasonable. Continuing to run the full set of tests on
>>>>> every patch for something we never expect to see the light of day
>>>>> (in its
>>>>> current form) seems wasteful to me. Plus, we're going to
>>>>> (presumably) be
>>>>> ramping up tests on v2.1, which means to me that we'll need to clear
>>>>> out
>>>>> some capacity to make room for that.
>>>>>
>>>>>
>>>> Thats true, though I was suggesting as v2.1microversions rolls out
>>>> we drop
>>>> the test out of v3 and move it to v2.1microversions testing, so
>>>> there's no
>>>> change in capacity required.
>>>
>>> That's why I wasn't proposing that we rip the tests out of the tree.
>>> I'm just
>>> trying to weigh the benefit of leaving them enabled on every run against
>>> the increased load they cause in an arguably overworked gate.
>>>
>>>>
>>>> Matt - how much of the time overhead is scenario tests? That's something
>>>> that would have a lot less impact if moved to and experimental queue.
>>>> Although the v3 api as a whole won't be officially exposed, the api
>>>> tests
>>>> test specific features fairly indepdently which are slated for
>>>> v2.1microversions on a case by case basis and I don't want to see those
>>>> regress. I guess my concern is how often the experimental queue
>>>> results get
>>>> really looked at and how hard/quick it is to revert when lots of stuff
>>>> merges in a short period of time)
>>>
>>> The scenario tests tend to be the slower tests in tempest. I have to
>>> disagree
>>> that removing them would have lower impact. The scenario tests provide
>>> the best
>>> functional verification, which is part of the reason we always have
>>> failures in
>>> the gate on them. While it would make the gate faster the decrease in
>>> what were
>>> testing isn't worth it. Also, for reference I pulled the test run
>>> times that
>>> were greater than 10sec out of a recent gate run:
>>> http://paste.openstack.org/show/83827/
>>>
>>> The experimental jobs aren't automatically run, they have to be manually
>>> triggered by leaving a 'check experimental' comment. So for changes
>>> that we want
>>> to test the v3 api on a comment would have to left. To prevent
>>> regression is why
>>> we'd also have the nightly job, which I think is a better compromise
>>> for the v3
>>> tests while we wait to migrate them to the v2.1 microversion tests.
>>>
>>> Another, option is that we make the v3 job run only on the check queue
>>> and not
>>> on the gate. But the benefits of that are slightly more limited,
>>> because we'd
>>> still be holding up the check queue.
>>>
>>> -Matt Treinish
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> Yeah the scenario tests need to stay, that's how we've exposed the two
>> big ssh bugs in the last couple of weeks which are obvious issues at scale.
>>
>> I still think experimental/periodic is the way to go, not a hybrid of
>> check-on/gate-off.  If we want to explicitly test v3 API changes we can
>> do that with 'recheck experimental'.  Granted someone has to remember to
>> run those, much like checking/rechecking 3rd party CI results.
>>
>> One issue I've had with the nightly periodic job is finding out where
>> the results are in an easy to consume format.  Is there something out
>> there for that?  I'm thinking specifically of things we've turned off in
>> the gate before like multi-backend volume tests and
>> allow_tenant_isolation=False.
>
> It's getting emailed to the otherwise defunct openstack-qa list.
> Subscribe there for nightlies.
>
> Also agreed, the scenario tests find and prevent *tons* of real issues.
> Those have to stay. There is a reason we use them in the smoke runs for
> grenade, they are a very solid sniff test of real working.
>
> I also think by policy we should probably pull v3 out of the main job,
> as it's not a stable API. We've had issues in Tempest with people
> landing tests, then trying to go and change the API. The biggest issue
> in taking branchless tempest back to stable/havana was Nova v3 API, as
> it's actually quite different in havana than icehouse.
>
> We have a chicken / egg challenge in testing experimental APIs which
> will need to get resolved, but for now I think turning off v3 is the
> right approach.

+1

Seems like we should concentrate on v2 tests for now.

To stop v3 code from regressing, we should be merging and testing
v2.1, ASAP, using those v2 tests.

For the future micro versions, we still have those v3 test to be inspired by.

Chris, does that seem workable?

John