<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Ray,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Thanks for initiating this thread.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">We are using Jmeter to solve some of these perf load generation issues:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * load generation is driven by a config file, so you can have multiple<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> different tests<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * once the test is complete, logs, tracing data, and config is all<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> gathered and saved<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * a reporting tool that can show the results of one test run, or compare<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> the results of at least two<o:p></o:p></i></span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Courier New"">Out performance testing is aimed towards scalability, workload and stress testing and we use Jmeter with some 3<sup>rd</sup> party-plugins as our main framework.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Courier New"">I will share the set-up and framework and we can discuss it further.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Courier New"">Thanks,<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Courier New"">Bageshree<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></i></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">On Wed, Apr 24, 2013 at 11:15 AM, Timothy Daly <<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style="color:blue">timjr at yahoo-inc.com</span></a>>
wrote:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> So far we've made good progress just running loops that, for example, boot<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> VMs in batches of 20. It has turned up performance problems and<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> concurrency bugs.<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">I attended the OpenStack Summit session titled "Scaling the Boot Barrier:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Identifying & Eliminating Contention in Openstack" given by Peter Feiner.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">It was very good and mentioned several VM boot bottlenecks that he<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">identified and fixed and one unfixed major bottleneck in libvirt. Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mentioned he would either drive or make that fix. If you have any input in<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">that area it might be good to contact Peter.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> In terms of in-VM testing, a coworker built a little thing does a<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> benchmark at boot time, (you pass the command to install and start in your<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> user data). It posts the benchmark results back to a redis database so you<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> can collect them easily. It just uses simple benchmarks:<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * dns resolver latency with dig<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * disk bandwidth with dd<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * disk latency with ioping<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * download speed using wget from a CDN<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * cpu speed using bc calculating 2^2^20<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> There are a lot of things that could be added, of course... the other<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> thing we were thinking of doing is giving it a schedule so it runs<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> regularly instead of just once at bootup. Not sure how useful that will<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> be. At least a bit to run in a loop could be handy to test scheduler<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> changes and interactions between multiple VMs...<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Yes, there is a lot of opportunity for variation in this area. I have a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">LAMP stack application (a variation of DVD Store used in VMMark) and load<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">generator that drives CPU, disk I/O and network resources. I think having<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">VMs run for a configurable amount of time to run such a workload would be<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">interesting. You could create a scenario that had an average number of VMs<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">by adjusting the run time, where run time = Average number of VMs divided<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">by VM boot rate. For example if you want 10 VMs on average divide that by<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">a VM boot rate of 10 per minute, the run time per VM should be 1 minute.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Run that for example for at least an hour. You could replace that with a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">VM workload that sat idle for 1 minute and you would get an idea of how VM<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">boots impact VM workload and resource usage and how VM workload impacts VM<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">boots. You would want the load generator to be able to drive an "offered<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">load".<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> Time to boot is done simply by posting the timestamp of the beginning of<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> the benchmark run back to redis.<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Nice, I was thinking along those lines.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> We want to move towards a test driver that has the following properties:<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * load generation is driven by a config file, so you can have multiple<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> different tests<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * once the test is complete, logs, tracing data, and config is all<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> gathered and saved<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> * a reporting tool that can show the results of one test run, or compare<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i> the results of at least two<o:p></o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">><i><o:p> </o:p></i></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">+1 to that<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>