Open Stack

On Fri, Jun 13, 2014 at 12:41:19AM +0930, Christopher Yeoh wrote:
> On Fri, Jun 13, 2014 at 12:25 AM, Dan Smith <dms at danplanet.com> wrote:
> 
> > I think it'd be OK to move them to the experimental queue and a periodic
> >> nightly job until the v2.1 stuff shakes out.  The v3 API is marked
> >> experimental right now so it seems fitting that it'd be running tests in
> >> the experimental queue until at least the spec is approved and
> >> microversioning starts happening in the code base.
> >>
> >
> > I think this is reasonable. Continuing to run the full set of tests on
> > every patch for something we never expect to see the light of day (in its
> > current form) seems wasteful to me. Plus, we're going to (presumably) be
> > ramping up tests on v2.1, which means to me that we'll need to clear out
> > some capacity to make room for that.
> >
> >
> Thats true, though I was suggesting as v2.1microversions rolls out  we drop
> the test out of v3 and move it to v2.1microversions testing, so there's no
> change in capacity required.

That's why I wasn't proposing that we rip the tests out of the tree. I'm just
trying to weigh the benefit of leaving them enabled on every run against
the increased load they cause in an arguably overworked gate.

> 
> Matt - how much of the time overhead is scenario tests? That's something
> that would have a lot less impact if moved to and experimental queue.
> Although the v3 api as a whole won't be officially exposed, the api tests
> test specific features fairly indepdently which are slated for
> v2.1microversions on a case by case basis and I don't want to see those
> regress. I guess my concern is how often the experimental queue results get
> really looked at and how hard/quick it is to revert when lots of stuff
> merges in a short period of time)

The scenario tests tend to be the slower tests in tempest. I have to disagree
that removing them would have lower impact. The scenario tests provide the best
functional verification, which is part of the reason we always have failures in
the gate on them. While it would make the gate faster the decrease in what were
testing isn't worth it. Also, for reference I pulled the test run times that
were greater than 10sec out of a recent gate run:
http://paste.openstack.org/show/83827/

The experimental jobs aren't automatically run, they have to be manually
triggered by leaving a 'check experimental' comment. So for changes that we want
to test the v3 api on a comment would have to left. To prevent regression is why
we'd also have the nightly job, which I think is a better compromise for the v3
tests while we wait to migrate them to the v2.1 microversion tests.

Another, option is that we make the v3 job run only on the check queue and not
on the gate. But the benefits of that are slightly more limited, because we'd
still be holding up the check queue.

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140612/1e7d0683/attachment.pgp>