Open Stack

Another thing that u might consider.

The rally project[1] has been using a tool called tomograph[2] and making
tomograph better and it has been collecting some similar use-cases and
test-cases around various openstack performance related work (also it has
been working on defining said methodology, how to setup for performance
tests, and more...).

Some examples of various keystone/nova/glance related calls (see the low
level trace information gathered there):

- 
https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g
lance-image-list.png
- http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24

It would seem like some of our scripts/test-cases would actually fit quite
nicely into rally instead of being one-offs.

I know that boris-42 would appreciate a single solution instead of
one-offs. It seems like rally is becoming that.

[1] https://wiki.openstack.org/wiki/Rally
[2] https://github.com/stackforge/tomograph

Jump in IRC and the rally team I'm sure would love to chat
(#openstack-rally, freenode).

-Josh

On 12/16/13, 10:00 AM, "Jay Pipes" <jaypipes at gmail.com> wrote:

>On 12/16/2013 02:25 AM, Neependra Khare wrote:
>> Hi Jay,
>>
>> Thanks for your comments.  Please find my reply in-line.
>>
>> On 12/14/2013 12:58 AM, Jay Pipes wrote:
>>> I have listed down the methodology I'll be following for this test:-
>>>> 
>>>>https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis
>>>>k.2C_Memory.2C_Database_bottlenecks
>>>>
>>>
>>> My first suggestion would be to rework the performance benchmarking
>>> work items to have clearer indications regarding *what are the metrics
>>> being tested* in each work item.
>> Performance characterization is an iterative process. I am open to
>> rework on the work-items as we
>> go along.
>
>Right, but the smaller the work item, the easier the iterations are :)
>
>>> For example, the first work item is "Identify CPU, Disk, Memory, and
>>> Database Bottlenecks".
>>>
>>> The first test case listed is:
>>>
>>> "Test #1, Create users in parallel and look for CPU, disk or memory
>>> bottleneck."
>>>
>>> I think that is a bit too big of an initial bite ;)
>>>
>>> Instead, it may be more effective to instead break down the
>>> performance analysis based on the metrics you wish to test and the
>>> relative conclusions you wish your work to generate.
>>>
>>> For example, consider this possible work item:
>>>
>>> "Determine the maximum number of token authentication calls that can
>>> be performed"
>> Tests like these would be very subjective to the hardware and software
>> resources we have like
>> no. of CPUs, Memcahced etc.  Its is very important to see if we can find
>> any obvious bottlenecks.
>
>No, that's not my point. When you have a specific metric like "number of
>token authentication calls that can be performed in X minutes", you can
>iterate based on singular changes -- not to the hardware, but to the
>configuration of the software. If you are trying to solve the problem of
>"where are my bottlenecks", without first identifying what metrics will
>describe how a piece of software scales, then you are putting the cart
>before the horse.
>
>>> Within that work item, you can then further expand a testing matrix,
>>> like so:
>>>
>>> * Measure the total number of token authentication calls performed by
>>> a single client against a single-process, Python-only Keystone server
>>> * Measure the total number of token authentication calls performed by
>>> a single client against a multi-process Keystone server running inside
>>> an nginx or Apache container server -- with 2, 4, 8, 16, and 32
>>> pre-forked processes
>> Any pointers on configuring multi-process Keystone would be helpful. I
>> see a method
>> mentioned in "Run N keystone Processes" section of following:-
>> 
>>http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf
>>ormance"
>
>Absolutely. You can spawn Keystone server in multiple pre-forked Apache
>processes by configuring Keystone in an Apache vhost. Some general docs:
>
>http://docs.openstack.org/developer/keystone/apache-httpd.html
>
>Take a look at provision.sh script in eNovance's keystone-wsgi-apache
>repo:
>
>https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#
>L152
>
>>> * Measure the above using increasing numbers of concurrent clients --
>>> 10, 50, 100, 500, 1000.
>>>
>>> There's, of course, nothing wrong with measuring things like CPU, disk
>>> and I/O performance during tests, however there should be a clear
>>> metric that is being measured for each test.
>> Agreed. Let me start collecting results from the tests you suggested
>> above and I mentioned
>> on the wiki. Once we have those, we can rework on the work-items. Does
>> that sound OK ?
>
>Sure, absolutely. I'm just big on first defining the metrics by which
>scale can be described, and THEN describing the test variations and
>iterations...
>
>>> My second suggestion would be to drop the requirement of using RDO --
>>> or any version of OpenStack for that matter.
>> My end goal would be to have scripts that one can run on any of the
>> OpenStack distribution.
>> RDO is mentioned here an example here.
>
>Probably worth just removing the RDO reference entirely from the wiki,
>since, as you agree below, benchmarking Keystone actually does not
>require installing OpenStack as a whole at all...
>
>Best,
>-jay
>
>>> In these kinds of tests, where you are not measuring the integrated
>>> performance of multiple endpoints, but are instead measuring the
>>> performance of a single endpoint (Keystone), there's no reason, IMHO,
>>> to install all of OpenStack. Installing and serving the Keystone
>>> server (and it's various drivers) is all that is needed. The fewer
>>> "balls up in the air" during a benchmarking session, the fewer
>>> side-effects are around to effect the outcome of the benchmark...
>> Agreed. As mentioned in following I suggested to install just Keystone
>> on the instances, where the tests would be performed :-
>> 
>>https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u
>>sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck.
>>
>>
>> Thanks,
>> Neependra
>>
>>
>>>
>>> Best,
>>> -jay
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
>
>_______________________________________________
>Mailing list: 
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>Post to     : openstack at lists.openstack.org
>Unsubscribe : 
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack