Open Stack

On 12/17/2013 12:48 AM, Joshua Harlow wrote:
> Another thing that u might consider.
>
> The rally project[1] has been using a tool called tomograph[2] and making
> tomograph better and it has been collecting some similar use-cases and
> test-cases around various openstack performance related work (also it has
> been working on defining said methodology, how to setup for performance
> tests, and more...).
>
> Some examples of various keystone/nova/glance related calls (see the low
> level trace information gathered there):
Thanks Josh.
We have started a blueprint to have keystone related benchmarks
with Rally.
https://blueprints.launchpad.net/rally/+spec/keystone-benchmark

Regards,
Neependra
> -
> https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g
> lance-image-list.png
> - http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24
>
> It would seem like some of our scripts/test-cases would actually fit quite
> nicely into rally instead of being one-offs.
>
> I know that boris-42 would appreciate a single solution instead of
> one-offs. It seems like rally is becoming that.
>
> [1] https://wiki.openstack.org/wiki/Rally
> [2] https://github.com/stackforge/tomograph
>
> Jump in IRC and the rally team I'm sure would love to chat
> (#openstack-rally, freenode).
>
> -Josh
>
> On 12/16/13, 10:00 AM, "Jay Pipes" <jaypipes at gmail.com> wrote:
>
>> On 12/16/2013 02:25 AM, Neependra Khare wrote:
>>> Hi Jay,
>>>
>>> Thanks for your comments.  Please find my reply in-line.
>>>
>>> On 12/14/2013 12:58 AM, Jay Pipes wrote:
>>>> I have listed down the methodology I'll be following for this test:-
>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis
>>>>> k.2C_Memory.2C_Database_bottlenecks
>>>>>
>>>> My first suggestion would be to rework the performance benchmarking
>>>> work items to have clearer indications regarding *what are the metrics
>>>> being tested* in each work item.
>>> Performance characterization is an iterative process. I am open to
>>> rework on the work-items as we
>>> go along.
>> Right, but the smaller the work item, the easier the iterations are :)
>>
>>>> For example, the first work item is "Identify CPU, Disk, Memory, and
>>>> Database Bottlenecks".
>>>>
>>>> The first test case listed is:
>>>>
>>>> "Test #1, Create users in parallel and look for CPU, disk or memory
>>>> bottleneck."
>>>>
>>>> I think that is a bit too big of an initial bite ;)
>>>>
>>>> Instead, it may be more effective to instead break down the
>>>> performance analysis based on the metrics you wish to test and the
>>>> relative conclusions you wish your work to generate.
>>>>
>>>> For example, consider this possible work item:
>>>>
>>>> "Determine the maximum number of token authentication calls that can
>>>> be performed"
>>> Tests like these would be very subjective to the hardware and software
>>> resources we have like
>>> no. of CPUs, Memcahced etc.  Its is very important to see if we can find
>>> any obvious bottlenecks.
>> No, that's not my point. When you have a specific metric like "number of
>> token authentication calls that can be performed in X minutes", you can
>> iterate based on singular changes -- not to the hardware, but to the
>> configuration of the software. If you are trying to solve the problem of
>> "where are my bottlenecks", without first identifying what metrics will
>> describe how a piece of software scales, then you are putting the cart
>> before the horse.
>>
>>>> Within that work item, you can then further expand a testing matrix,
>>>> like so:
>>>>
>>>> * Measure the total number of token authentication calls performed by
>>>> a single client against a single-process, Python-only Keystone server
>>>> * Measure the total number of token authentication calls performed by
>>>> a single client against a multi-process Keystone server running inside
>>>> an nginx or Apache container server -- with 2, 4, 8, 16, and 32
>>>> pre-forked processes
>>> Any pointers on configuring multi-process Keystone would be helpful. I
>>> see a method
>>> mentioned in "Run N keystone Processes" section of following:-
>>>
>>> http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf
>>> ormance"
>> Absolutely. You can spawn Keystone server in multiple pre-forked Apache
>> processes by configuring Keystone in an Apache vhost. Some general docs:
>>
>> http://docs.openstack.org/developer/keystone/apache-httpd.html
>>
>> Take a look at provision.sh script in eNovance's keystone-wsgi-apache
>> repo:
>>
>> https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#
>> L152
>>
>>>> * Measure the above using increasing numbers of concurrent clients --
>>>> 10, 50, 100, 500, 1000.
>>>>
>>>> There's, of course, nothing wrong with measuring things like CPU, disk
>>>> and I/O performance during tests, however there should be a clear
>>>> metric that is being measured for each test.
>>> Agreed. Let me start collecting results from the tests you suggested
>>> above and I mentioned
>>> on the wiki. Once we have those, we can rework on the work-items. Does
>>> that sound OK ?
>> Sure, absolutely. I'm just big on first defining the metrics by which
>> scale can be described, and THEN describing the test variations and
>> iterations...
>>
>>>> My second suggestion would be to drop the requirement of using RDO --
>>>> or any version of OpenStack for that matter.
>>> My end goal would be to have scripts that one can run on any of the
>>> OpenStack distribution.
>>> RDO is mentioned here an example here.
>> Probably worth just removing the RDO reference entirely from the wiki,
>> since, as you agree below, benchmarking Keystone actually does not
>> require installing OpenStack as a whole at all...
>>
>> Best,
>> -jay
>>
>>>> In these kinds of tests, where you are not measuring the integrated
>>>> performance of multiple endpoints, but are instead measuring the
>>>> performance of a single endpoint (Keystone), there's no reason, IMHO,
>>>> to install all of OpenStack. Installing and serving the Keystone
>>>> server (and it's various drivers) is all that is needed. The fewer
>>>> "balls up in the air" during a benchmarking session, the fewer
>>>> side-effects are around to effect the outcome of the benchmark...
>>> Agreed. As mentioned in following I suggested to install just Keystone
>>> on the instances, where the tests would be performed :-
>>>
>>> https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u
>>> sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck.
>>>
>>>
>>> Thanks,
>>> Neependra
>>>
>>>
>>>> Best,
>>>> -jay
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list:
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> Post to     : openstack at lists.openstack.org
>>>> Unsubscribe :
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131217/3a443d0e/attachment.html>