<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi All, <br>
<br>
I wanted to share the current status of this work. <br>
<br>
Thanks to Boris and the team for implementing keystone benchmark
with Rally. <br>
It has been merged with upstream Rally code base couple of days
back. <br>
<br>
I have tested above with devstack on Fedora 19 and Ubuntu 12.04. <br>
<br>
With Rally Deploy engine we can deploy OpenStack distribution like
DevStack<br>
before running the benchmark. <br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a
href="https://wiki.openstack.org/wiki/Rally/DeployEngines#DevstackEngine">https://wiki.openstack.org/wiki/Rally/DeployEngines#DevstackEngine</a><br>
<br>
I am planning to prepare the <i>localrc</i> file for <i>devstack</i>
by which we can configure<br>
keystone with different configuration like with PKI/UUID tokens,
with or without<br>
memcached etc. This would help us is automating the process. I
would be<br>
documenting everything and would share it with the community.<br>
<br>
Regards, <br>
Neependra<br>
<br>
<br>
On 12/20/2013 02:09 PM, Neependra Khare wrote:<br>
</div>
<blockquote cite="mid:52B40255.7030000@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Hi Joshua, <br>
<br>
On 12/20/2013 01:34 PM, Joshua Harlow wrote:<br>
</div>
<blockquote
cite="mid:CD558DF4-AF44-4085-A504-818EBEB74448@yahoo-inc.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<div>Awesome i will see if I can make it, maybe u guys can send
out a summary afterwards? Glad to see performance work (and
the associated tooling) get the attention it deserves!</div>
</blockquote>
We had a meeting yesterday. Etherpad has most of things we
discussed:-<br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a moz-do-not-send="true"
href="https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks">https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks</a><br>
<br>
We decided to have specific pattern in workload (like user name)
for keystone<br>
benchmarks as that would make clean-up easier.<br>
<br>
INIT methods like creating N users, would setup environment before
running the<br>
actual setup. Jay suggested set that may be we can use some form
of snapshot <br>
for images, which have per-populated dataset. So that, we do not
have to spend <br>
time in INIT methods every time we run the tests. <br>
<br>
Next steps<br>
- Boris would have the clean-up and basic INIT functionality
written down for Keystone<br>
, which would then help us to write our first Keystone benchmark.
<br>
- Meanwhile we'll update the benchmark we want to run with Rally
in <i>"</i><i><span class="" style="margin: 0px; padding: 1px
0px; cursor: auto; line-height: 1.5em;">List of benchmarks <br>
that we would like to have</span></i><i>"</i> section of the
etherpad. <br>
<br>
Boris, Jay, Hugh, Tristan and others,<br>
If I missed any important point then please feel free to add here.
<br>
<br>
Regards, <br>
Neependra<br>
<br>
<br>
<blockquote
cite="mid:CD558DF4-AF44-4085-A504-818EBEB74448@yahoo-inc.com"
type="cite">
<div><br>
Sent from my really tiny device...</div>
<div><br>
On Dec 18, 2013, at 5:32 AM, "Neependra Khare" <<a
moz-do-not-send="true" href="mailto:nkhare@redhat.com">nkhare@redhat.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div class="moz-cite-prefix">On 12/17/2013 05:19 PM,
Neependra Khare wrote:<br>
</div>
<blockquote cite="mid:52B03A3A.2090203@redhat.com"
type="cite">
<div class="moz-cite-prefix">On 12/17/2013 12:48 AM,
Joshua Harlow wrote:<br>
</div>
<blockquote
cite="mid:CED490CB.5071F%25harlowja@yahoo-inc.com"
type="cite">
<pre wrap="">Another thing that u might consider.
The rally project[1] has been using a tool called tomograph[2] and making
tomograph better and it has been collecting some similar use-cases and
test-cases around various openstack performance related work (also it has
been working on defining said methodology, how to setup for performance
tests, and more...).
Some examples of various keystone/nova/glance related calls (see the low
level trace information gathered there):
</pre>
</blockquote>
Thanks Josh. <br>
We have started a blueprint to have keystone related
benchmarks<br>
with Rally.<br>
<a moz-do-not-send="true"
href="https://blueprints.launchpad.net/rally/+spec/keystone-benchmark">https://blueprints.launchpad.net/rally/+spec/keystone-benchmark</a><br>
</blockquote>
We are planning to have g+ hangout tomorrow (Dec 19, 2013)
to discuss<br>
Keystone related benchmarks with Rally at 11:00 EST<br>
<a moz-do-not-send="true"
href="https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks">https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks</a><br>
<br>
Feel free to join it. Just update your name on above
mentioned etherpad <br>
under <span style="color: rgb(0, 0, 0); font-family: Arial,
sans-serif; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: 16px; orphans: auto; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(217,
193, 121); display: inline !important; float: none;">
Interested participants </span>section. <br>
<br>
Regards,<br>
Neependra<br>
<br>
<br>
<blockquote cite="mid:52B03A3A.2090203@redhat.com"
type="cite"><br>
Regards,<br>
Neependra<br>
<blockquote
cite="mid:CED490CB.5071F%25harlowja@yahoo-inc.com"
type="cite">
<pre wrap="">-
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g">https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g</a>
lance-image-list.png
- <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24">http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24</a>
It would seem like some of our scripts/test-cases would actually fit quite
nicely into rally instead of being one-offs.
I know that boris-42 would appreciate a single solution instead of
one-offs. It seems like rally is becoming that.
[1] <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://wiki.openstack.org/wiki/Rally">https://wiki.openstack.org/wiki/Rally</a>
[2] <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://github.com/stackforge/tomograph">https://github.com/stackforge/tomograph</a>
Jump in IRC and the rally team I'm sure would love to chat
(#openstack-rally, freenode).
-Josh
On 12/16/13, 10:00 AM, "Jay Pipes" <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:jaypipes@gmail.com"><jaypipes@gmail.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 12/16/2013 02:25 AM, Neependra Khare wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Jay,
Thanks for your comments. Please find my reply in-line.
On 12/14/2013 12:58 AM, Jay Pipes wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I have listed down the methodology I'll be following for this test:-
</pre>
<blockquote type="cite">
<pre wrap=""><a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis">https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis</a>
k.2C_Memory.2C_Database_bottlenecks
</pre>
</blockquote>
<pre wrap="">My first suggestion would be to rework the performance benchmarking
work items to have clearer indications regarding *what are the metrics
being tested* in each work item.
</pre>
</blockquote>
<pre wrap="">Performance characterization is an iterative process. I am open to
rework on the work-items as we
go along.
</pre>
</blockquote>
<pre wrap="">Right, but the smaller the work item, the easier the iterations are :)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">For example, the first work item is "Identify CPU, Disk, Memory, and
Database Bottlenecks".
The first test case listed is:
"Test #1, Create users in parallel and look for CPU, disk or memory
bottleneck."
I think that is a bit too big of an initial bite ;)
Instead, it may be more effective to instead break down the
performance analysis based on the metrics you wish to test and the
relative conclusions you wish your work to generate.
For example, consider this possible work item:
"Determine the maximum number of token authentication calls that can
be performed"
</pre>
</blockquote>
<pre wrap="">Tests like these would be very subjective to the hardware and software
resources we have like
no. of CPUs, Memcahced etc. Its is very important to see if we can find
any obvious bottlenecks.
</pre>
</blockquote>
<pre wrap="">No, that's not my point. When you have a specific metric like "number of
token authentication calls that can be performed in X minutes", you can
iterate based on singular changes -- not to the hardware, but to the
configuration of the software. If you are trying to solve the problem of
"where are my bottlenecks", without first identifying what metrics will
describe how a piece of software scales, then you are putting the cart
before the horse.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Within that work item, you can then further expand a testing matrix,
like so:
* Measure the total number of token authentication calls performed by
a single client against a single-process, Python-only Keystone server
* Measure the total number of token authentication calls performed by
a single client against a multi-process Keystone server running inside
an nginx or Apache container server -- with 2, 4, 8, 16, and 32
pre-forked processes
</pre>
</blockquote>
<pre wrap="">Any pointers on configuring multi-process Keystone would be helpful. I
see a method
mentioned in "Run N keystone Processes" section of following:-
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf">http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf</a>
ormance"
</pre>
</blockquote>
<pre wrap="">Absolutely. You can spawn Keystone server in multiple pre-forked Apache
processes by configuring Keystone in an Apache vhost. Some general docs:
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://docs.openstack.org/developer/keystone/apache-httpd.html">http://docs.openstack.org/developer/keystone/apache-httpd.html</a>
Take a look at provision.sh script in eNovance's keystone-wsgi-apache
repo:
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#">https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#</a>
L152
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">* Measure the above using increasing numbers of concurrent clients --
10, 50, 100, 500, 1000.
There's, of course, nothing wrong with measuring things like CPU, disk
and I/O performance during tests, however there should be a clear
metric that is being measured for each test.
</pre>
</blockquote>
<pre wrap="">Agreed. Let me start collecting results from the tests you suggested
above and I mentioned
on the wiki. Once we have those, we can rework on the work-items. Does
that sound OK ?
</pre>
</blockquote>
<pre wrap="">Sure, absolutely. I'm just big on first defining the metrics by which
scale can be described, and THEN describing the test variations and
iterations...
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">My second suggestion would be to drop the requirement of using RDO --
or any version of OpenStack for that matter.
</pre>
</blockquote>
<pre wrap="">My end goal would be to have scripts that one can run on any of the
OpenStack distribution.
RDO is mentioned here an example here.
</pre>
</blockquote>
<pre wrap="">Probably worth just removing the RDO reference entirely from the wiki,
since, as you agree below, benchmarking Keystone actually does not
require installing OpenStack as a whole at all...
Best,
-jay
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">In these kinds of tests, where you are not measuring the integrated
performance of multiple endpoints, but are instead measuring the
performance of a single endpoint (Keystone), there's no reason, IMHO,
to install all of OpenStack. Installing and serving the Keystone
server (and it's various drivers) is all that is needed. The fewer
"balls up in the air" during a benchmarking session, the fewer
side-effects are around to effect the outcome of the benchmark...
</pre>
</blockquote>
<pre wrap="">Agreed. As mentioned in following I suggested to install just Keystone
on the instances, where the tests would be performed :-
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u">https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u</a>
sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck.
Thanks,
Neependra
</pre>
<blockquote type="cite">
<pre wrap="">Best,
-jay
_______________________________________________
Mailing list:
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
Post to : <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>
Unsubscribe :
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">_______________________________________________
Mailing list:
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
Post to : <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>
Unsubscribe :
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
</pre>
</blockquote>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Mailing list: <a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
Post to : <a class="moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>
Unsubscribe : <a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
</pre>
</blockquote>
<br>
</body>
</html>