Open Stack

Is it easy ? No... it is hard, whether in an integrated test suite or on its own
Can it be solved ?  Yes, we have done incredible things with the current QA infrastructure
Should it be split off from other testing ? No, I want EVERY commit to have this check. Performance through benchmarking at scale is fundamental.

Integration with the current process mean all code has to pass the bar... a new project makes it optional and therefore makes the users do the debugging... just the kind of thing that drives those users away...

Tim

> -----Original Message-----
> From: Robert Collins [mailto:robertc at robertcollins.net]
> Sent: 20 October 2013 21:03
> To: OpenStack Development Mailing List
> Subject: Re: [openstack-dev] Announce of Rally - benchmarking system for OpenStack
> 
> On 21 October 2013 07:36, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> > There's several issues involved in doing automated regression checking
> > for
> > benchmarks:
> >
> > - You need a platform which is stable. Right now all our CI runs on
> > virtualized instances, and I don't think there's any particular
> > guarantee it'll be the same underlying hardware, further virtualized
> > systems tend to be very noisy and not give you the stability you need.
> > - You need your benchmarks to be very high precision, if you really
> > want to rule out regressions of more than N% without a lot of false positives.
> > - You need more than just checks on individual builds, you need long
> > term trend checking - 100 1% regressions are worse than a single 50% regression.
> 
> Let me offer a couple more key things:
>  - you need a platform that is representative of your deployments:
> 1000 physical hypervisors have rather different checkin patterns than
> 1 qemu hypervisor.
>  - you need a workload that is representative of your deployments:
> 10000 VM's spread over 500 physical hypervisors routing traffic through one neutron software switch will have rather different load
> characteristics than 5 qemu vm's in a kvm vm hosted all in one configuration.
> 
> neither the platform - # of components, their configuration, etc, nor the workload in devstack-gate are representative of production
> deployments of any except the most modest clouds. Thats fine - devstack-gate to date has been about base functionality, not digging down
> into race conditions.
> 
> I think having a dedicated tool aimed at:
>  - setting up *many different* production-like environments and running
>  - many production-like workloads and
>  - reporting back which ones work and which ones don't
> 
> makes a huge amount of sense.
> 
> from the reports from that tool we can craft targeted unit test or isolated functional tests to capture the problem and prevent it
> worsening or regressing (once fixed). See for instance Joe Gordons'
> fake hypervisor which is great for targeted testing.
> 
> That said, I also agree with the sentiment expressed that the workload-driving portion of Rally doesn't seem different enough to Tempest
> to warrant being separate; it seems to me that Rally could be built like this:
> 
> - a thing that does deployments spread out over a phase space of configurations
> - instrumentation for deployments that permit the data visibility needed to analyse problems
> - tests for tempest that stress a deployment
> 
> So the single-button-push Rally would:
>  - take a set of hardware
>  - in a loop
>  - deploy a configuration, run Tempest, report data
> 
> That would reuse Tempest and still be a single button push data gathering thing, and if Tempest isn't capable of generating enough
> concurrency/load [for a single test - ignore parallel execution of different tests] then that seems like something we should fix in Tempest,
> because concurrency/race conditions are things we need tests for in devstack-gate.
> 
> -Rob
> 
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev