Open Stack

On Tue, Mar 6, 2018, at 1:28 PM, Lance Bragstad wrote:
> Hey all,
> 
> Last week during the PTG the keystone team sat down with a few of the
> infra folks to discuss performance testing. The major hurdle here has
> always been having dedicated hosts to use for performance testing,
> regardless of that being rally, tempest, or a home-grown script.
> Otherwise results vary wildly from run to run in the gate due to
> differences from providers or noisy neighbor problems.
> 
> Opening up the discussion here because it sounded like some providers
> (mnaser, mtreinish) had some thoughts on how we can reserve specific
> hardware for these cases.
> 
> Thoughts?

Currently the Infra team has access to a variety of clouds, but due to how scheduling works we can't rule out noisy neighbors (or even being our own noisy neighbor). mtreinish also has data showing that runtimes are too noisy to do statistical analysis on, even within a single cloud region. So this is indeed an issue in the current setup.

One approach that has been talked about in the past is to measure performance impacting operations using metrics other than execution time. For example number of sql queries or rabbit requests. I think this would also be valuable but won't give you proper performance measurements.

That brought us back to the idea of possibly working with some cloud providers like mnaser and/or mtreinish to have a small number of dedicated instances to run performance tests on. We could then avoid the noisy neighbor problem as well.

For the infra team we would likely need to have at least two providers providing these resources so that we could handle the loss of one without backing up job queues. I don't think the hardware needs to have an other special properties as we don't care about performance on specific hardware as much as comparing performance of the project over time on known hardware.

Curious to hear what others may have to say.

Thanks,
Clark