Open Stack

Hi Ray!

We've been thinking about similar things here at Y!.

So far we've made good progress just running loops that, for example, boot VMs in batches of 20.  It has turned up performance problems and concurrency bugs.

In terms of in-VM testing, a coworker built a little thing does a benchmark at boot time, (you pass the command to install and start in your user data).  It posts the benchmark results back to a redis database so you can collect them easily.  It just uses simple benchmarks:

* dns resolver latency with dig
* disk bandwidth with dd
* disk latency with ioping
* download speed using wget from a CDN
* cpu speed using bc calculating 2^2^20

There are a lot of things that could be added, of course... the other thing we were thinking of doing is giving it a schedule so it runs regularly instead of just once at bootup.  Not sure how useful that will be.  At least a bit to run in a loop could be handy to test scheduler changes and interactions between multiple VMs...

Time to boot is done simply by posting the timestamp of the beginning of the benchmark run back to redis.

We want to move towards a test driver that has the following properties:

* load generation is driven by a config file, so you can have multiple different tests 
* once the test is complete, logs, tracing data, and config is all gathered and saved
* a reporting tool that can show the results of one test run, or compare the results of at least two

Cheers,
Tim

On Apr 23, 2013, at 3:37 PM, Ray Pekowski wrote:

> I would like to start a discussion around performance workloads for OpenStack.  I plan to develop benchmarks to simulate such workloads and it would be a good idea to list and prioritize some top number of them.  Maybe this is too broad of a question, but it would be good to have that feedback too.  Here is an an initial stab at it, listed in priority order (highest first):
> 	• VM boot with varying
> 		• numbers of VMs
> 		• compute nodes
> 		• amount of compute workload per boot
> 		• sizes (flavors)
> 		• VMs per tenant (project)
> 		• migrations
> 	• Block devices with varying
> 		• numbers
> 		• sizes
> 		• attachments to VMs
> 		• snapshoting
> 	• Network creations with varying
> 		• numbers of subnets
> 		• floating IP assignments
> 		• networks per tenant (project)
> 	• Images with varying
> 		• number of images
> 		• sizes
> 		• formats
> 		• time to boot to log in and to finish booting
> Ray
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev