Open Stack

Excerpts from Sean Dague's message of 2016-02-10 04:33:44 -0800:
> The largeops tests at this point are mostly finding out that some of our
> new cloud providers are slow - http://tinyurl.com/j5u4nf5
> 
> This is fundamentally a performance test, with timings having been tuned
> to pass 98% of the time on two clouds that were very predictable in
> performance. We're now running on 4 clouds, and the variance between
> them all, and between every run on each can be as much as a factor of 2.
> 
> We could just bump all the timeouts again, but that's basically the same
> thing as dropping them.
> 
> These tests are not instrumented in a way that any real solution can be
> addressed in most cases. Tests without a path forward, that are failing
> good patches a lot, are very much the kind of thing we should remove
> from the system.
> 

I think we need to replace this with something that measures work
counters, and not clock time. As you say, some of the other test suites
out there already pick up a lot of this slack too. Also, I'm working at
this as well with the counter-inspection spec, so hopefully dropping
this now won't leave too much of a gap in coverage while we ramp up
counter-inspection.

+1 to getting rid of it now, as instability in the test suites, which
slows down development velocity, is worse than missing a few performance
regressions in the corners.