<div dir="ltr">Hi Sean, <div><br></div><div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">>> Honestly, there has been so much work done in Tempest with the stress tests and the scenario tests in this release, and a large growing community around those, that it doesn't make any sense to me that you put item #3 as a non Tempest thing.</span><br style="font-family:arial,sans-serif;font-size:13px">
</div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><font face="arial, sans-serif">I really don't like to copy paste functionality that is already implemented, but I have some other ideas about benchmark engine, I would like to implement them and try as soon as possible, to determine "what we actually need". It is much simpler to make such experiments in Rally, because there is a small community around it, and I don't care at this moment about backward comparability. </font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">When I determine all benchmark engine parameters (what I need). I will make one more deep investigation, to see how complex will be to implement the same in tempest. If it will be possible and not extra complex we will start implementing required functionality in tempest. And when tempest will be ready we will switch to it.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"><br></font></div>
<div>>> <span style="font-family:arial,sans-serif;font-size:13px">Are you guys doing a summit session somewhere on this.</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div>
<font face="arial, sans-serif">There is a slot, but it is not approved <a href="http://summit.openstack.org/cfp/details/158">http://summit.openstack.org/cfp/details/158</a> yet. </font></div><div>Also there is a talk: <a href="http://openstacksummitnovember2013.sched.org/event/661ddc95f6b06ed3a634f12de09afa1d#.UmQITpTk9Z9">http://openstacksummitnovember2013.sched.org/event/661ddc95f6b06ed3a634f12de09afa1d#.UmQITpTk9Z9</a></div>
<div><br></div><div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">>> It also feels like the efforts around #4 would be much better served in the OpenStack community if they were integrated around testr and subunit so they could be reused in many contexts.</span><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I already tried to use pytest as a base and that was the biggest mistake..</font></div><div><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"><br></font></div><div><span style="font-family:arial,sans-serif;font-size:13px">>> I also think 1.b.3 is probably better done in the way the coverage extension was done for nova, something which is baked in and can be administratively turned on, not something which requires a hot patch to the system.</span><br style="font-family:arial,sans-serif;font-size:13px">
</div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><br></div><div>I hope these patches will be merged. But you see there is a "community deadlock" here. You are not able to merge such patches in upstream until you make two things:</div>
<div>1) Get a real examples of usage (so e.g. with hot patching in Rally)</div><div>2) Approve that such changes don't impact on performance. (Rally)</div><div><br></div><div>So we will prepare all patches, show live demonstration and approve that they don't impact on performance (especially when they are turned off) ;)</div>
<div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">>> It's cool to have performance analysis tooling, but if it arrives in a way that doesn't integrate well with the rest of OpenStack, the impact is going to be far less than it could be. I'd like us to get the full bang for our buck out of efforts like this, especially if there is hope for it to graduate from stackforge into one of our standard toolkits.</span><br>
</div><div><div class="" style="font-family:arial,sans-serif;font-size:13px"></div></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><font face="arial, sans-serif">I don't understand the phrase "</font><span style="font-family:arial,sans-serif;font-size:13px">doesn't </span><span style="font-size:13px;font-family:arial,sans-serif">integrate well with the rest of OpenStack</span><span style="font-family:arial,sans-serif">"</span></div>
<div><span style="font-family:arial,sans-serif"><br></span></div><div><span style="font-family:arial,sans-serif">1. Project has classical OpenStack structure </span><br></div><div><span style="font-family:arial,sans-serif">2. We use all common code from oslo (db, localizations, oslo.config in future we are planing to use oslo.messaging)</span></div>
<div><font face="arial, sans-serif">3. As a base and first engine we decide to use DevStack</font></div><div><font face="arial, sans-serif">4. To verify deployments (we will use tempest asap)</font></div><div><font face="arial, sans-serif">5. In benchmark engine we are using only native OS python clients to make requests to OpenStack API</font></div>
<div>6. OpenStack development workflow:</div><div>6.a) project is on stackforge and we use "stock jenkins"</div><div>6.b) launchpad with BPs, Bugs and Questions</div><div>6.c) wiki on <a href="http://wiki.openstack.org/wiki/Rally">http://wiki.openstack.org/wiki/Rally</a>, </div>
<div><font face="arial, sans-serif"><br></font></div><div>What "the rest of OpenStack" are you speaking?</div><div><font face="arial, sans-serif"><br></font></div><div><br></div><div><font face="arial, sans-serif">Best regards,</font></div>
<div><font face="arial, sans-serif">Boris Pavlovic</font></div><div><font face="arial, sans-serif">---</font></div><div><font face="arial, sans-serif">Mirantis Inc.</font></div><div><br></div><div><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Sun, Oct 20, 2013 at 2:38 AM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 10/18/2013 12:17 PM, Boris Pavlovic wrote:<br>
</div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
John,<br>
<br>
Actually seems like a pretty good suggestion IMO, at least something<br>
worth some investigation and consideration before quickly discounting<br>
it. Rather than "that's not what tempest is", maybe it's something<br>
tempest "could do". Don't know, not saying one way or the other, just<br>
wondering if it's worth some investigation or thought.<br>
<br>
<br>
These investigations I made before start working around Rally. It was<br>
about 3 months ago.<br>
It is not "quickly discounting" I didn't have yesterday time to make<br>
long response, so I will write it today:<br>
<br>
I really don't like to make a copy of another projects, so I tried to<br>
reuse all projects & libs that we already have.<br>
<br>
To explain why we shouldn't merge Rally & Tempest in one project (and<br>
should keep both) we should analyze their use cases.<br>
<br>
<br>
1. DevStack - one "click" and get your OpenStack cloud from sources<br>
<br>
2. Tempest - one "click" and get your OpenStack Cloud verified<br>
<br>
Both of these projects are great, because they are very useful and solve<br>
complicated tasks without "pain" for end user. (and I like them)<br>
<br>
3. Rally is also one "click" system that solve OpenStack benchmarking.<br>
<br>
To clarify situation. We should analyze what I mean by one "click"<br>
benchmarking and what are the use cases.<br>
<br>
Use Cases:<br>
1. Investigate how deployments influence on OS performance (find the set<br>
of good OpenStack deployment architectures)<br>
2. Automatically get numbers & profiling info about how your changes<br>
influence on OS performance<br>
3. Using Rally profiler detect scale & performance issues.<br>
Like here when we are trying to delete 3 VMs by one request they are<br>
deleted one by one because of DB lock on quotas table<br>
<a href="http://37.58.79.43:8080/traces/0011f252c9d98e31" target="_blank">http://37.58.79.43:8080/<u></u>traces/0011f252c9d98e31</a><br>
4. Determine maximal load that could handle production cloud<br>
<br>
To cover these cases we should actually test OpenStack deployments<br>
making simultaneously OpenStack API calls.<br>
<br>
So to get results we have to:<br>
1. Deploy OpenStack cloud somewhere. (Or get existing cloud)<br>
2. Verify It<br>
3. Run Benchmarks<br>
4. Collect all results & present it in human readable form.<br>
<br>
<br>
The goal of Rally was designed to automate these steps:<br>
1.a Use existing cloud.<br>
1.b.1 Automatically get (virtual) Servers from (soft layer, Amazon,<br>
RackSpace or you private public cloud, or OpenStack cloud)<br>
1.b.2 DeployOpenStack on these servers from source (using Devstack,<br>
Anvli, Fuel or TrippleO...).<br>
1.b.3 Patch this OpenStack with tomograph to get profiling information<br>
(I hope we will merge these patches into upstream).<br>
2. Using tempest verify this cloud (we are going to switch from<br>
fuel-ostf-tests)<br>
3. Run specified parametrized (to be able to make different load)<br>
benchmark scenarios<br>
4. Collect all information about execution & present it in human<br>
readable form. (Tomograph, Zipking, matplotlib...)<br>
</blockquote>
<br></div></div>
Honestly, there has been so much work done in Tempest with the stress tests and the scenario tests in this release, and a large growing community around those, that it doesn't make any sense to me that you put item #3 as a non Tempest thing.<br>
<br>
It feels very "not invented here".<br>
<br>
Are you guys doing a summit session somewhere on this.<br>
<br>
It also feels like the efforts around #4 would be much better served in the OpenStack community if they were integrated around testr and subunit so they could be reused in many contexts.<br>
<br>
I also think 1.b.3 is probably better done in the way the coverage extension was done for nova, something which is baked in and can be administratively turned on, not something which requires a hot patch to the system.<br>
<br>
It's cool to have performance analysis tooling, but if it arrives in a way that doesn't integrate well with the rest of OpenStack, the impact is going to be far less than it could be. I'd like us to get the full bang for our buck out of efforts like this, especially if there is hope for it to graduate from stackforge into one of our standard toolkits.<span class="HOEnZb"><font color="#888888"><br>
<br>
-Sean<br>
<br>
-- <br>
Sean Dague<br>
<a href="http://dague.net" target="_blank">http://dague.net</a></font></span><div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div>