Open Stack

On 12/17/2013 05:19 PM, Neependra Khare wrote:
> On 12/17/2013 12:48 AM, Joshua Harlow wrote:
>> Another thing that u might consider.
>>
>> The rally project[1] has been using a tool called tomograph[2] and making
>> tomograph better and it has been collecting some similar use-cases and
>> test-cases around various openstack performance related work (also it has
>> been working on defining said methodology, how to setup for performance
>> tests, and more...).
>>
>> Some examples of various keystone/nova/glance related calls (see the low
>> level trace information gathered there):
> Thanks Josh.
> We have started a blueprint to have keystone related benchmarks
> with Rally.
> https://blueprints.launchpad.net/rally/+spec/keystone-benchmark
We are planning to have g+ hangout tomorrow (Dec 19, 2013)  to discuss
Keystone related benchmarks with Rally at 11:00 EST
https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks

Feel free to join it. Just update your name on above mentioned etherpad
under Interested participants section.

Regards,
Neependra


>
> Regards,
> Neependra
>> -
>> https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g
>> lance-image-list.png
>> -http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24
>>
>> It would seem like some of our scripts/test-cases would actually fit quite
>> nicely into rally instead of being one-offs.
>>
>> I know that boris-42 would appreciate a single solution instead of
>> one-offs. It seems like rally is becoming that.
>>
>> [1]https://wiki.openstack.org/wiki/Rally
>> [2]https://github.com/stackforge/tomograph
>>
>> Jump in IRC and the rally team I'm sure would love to chat
>> (#openstack-rally, freenode).
>>
>> -Josh
>>
>> On 12/16/13, 10:00 AM, "Jay Pipes"<jaypipes at gmail.com>  wrote:
>>
>>> On 12/16/2013 02:25 AM, Neependra Khare wrote:
>>>> Hi Jay,
>>>>
>>>> Thanks for your comments.  Please find my reply in-line.
>>>>
>>>> On 12/14/2013 12:58 AM, Jay Pipes wrote:
>>>>> I have listed down the methodology I'll be following for this test:-
>>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis
>>>>>> k.2C_Memory.2C_Database_bottlenecks
>>>>>>
>>>>> My first suggestion would be to rework the performance benchmarking
>>>>> work items to have clearer indications regarding *what are the metrics
>>>>> being tested* in each work item.
>>>> Performance characterization is an iterative process. I am open to
>>>> rework on the work-items as we
>>>> go along.
>>> Right, but the smaller the work item, the easier the iterations are :)
>>>
>>>>> For example, the first work item is "Identify CPU, Disk, Memory, and
>>>>> Database Bottlenecks".
>>>>>
>>>>> The first test case listed is:
>>>>>
>>>>> "Test #1, Create users in parallel and look for CPU, disk or memory
>>>>> bottleneck."
>>>>>
>>>>> I think that is a bit too big of an initial bite ;)
>>>>>
>>>>> Instead, it may be more effective to instead break down the
>>>>> performance analysis based on the metrics you wish to test and the
>>>>> relative conclusions you wish your work to generate.
>>>>>
>>>>> For example, consider this possible work item:
>>>>>
>>>>> "Determine the maximum number of token authentication calls that can
>>>>> be performed"
>>>> Tests like these would be very subjective to the hardware and software
>>>> resources we have like
>>>> no. of CPUs, Memcahced etc.  Its is very important to see if we can find
>>>> any obvious bottlenecks.
>>> No, that's not my point. When you have a specific metric like "number of
>>> token authentication calls that can be performed in X minutes", you can
>>> iterate based on singular changes -- not to the hardware, but to the
>>> configuration of the software. If you are trying to solve the problem of
>>> "where are my bottlenecks", without first identifying what metrics will
>>> describe how a piece of software scales, then you are putting the cart
>>> before the horse.
>>>
>>>>> Within that work item, you can then further expand a testing matrix,
>>>>> like so:
>>>>>
>>>>> * Measure the total number of token authentication calls performed by
>>>>> a single client against a single-process, Python-only Keystone server
>>>>> * Measure the total number of token authentication calls performed by
>>>>> a single client against a multi-process Keystone server running inside
>>>>> an nginx or Apache container server -- with 2, 4, 8, 16, and 32
>>>>> pre-forked processes
>>>> Any pointers on configuring multi-process Keystone would be helpful. I
>>>> see a method
>>>> mentioned in "Run N keystone Processes" section of following:-
>>>>
>>>> http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf
>>>> ormance"
>>> Absolutely. You can spawn Keystone server in multiple pre-forked Apache
>>> processes by configuring Keystone in an Apache vhost. Some general docs:
>>>
>>> http://docs.openstack.org/developer/keystone/apache-httpd.html
>>>
>>> Take a look at provision.sh script in eNovance's keystone-wsgi-apache
>>> repo:
>>>
>>> https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#
>>> L152
>>>
>>>>> * Measure the above using increasing numbers of concurrent clients --
>>>>> 10, 50, 100, 500, 1000.
>>>>>
>>>>> There's, of course, nothing wrong with measuring things like CPU, disk
>>>>> and I/O performance during tests, however there should be a clear
>>>>> metric that is being measured for each test.
>>>> Agreed. Let me start collecting results from the tests you suggested
>>>> above and I mentioned
>>>> on the wiki. Once we have those, we can rework on the work-items. Does
>>>> that sound OK ?
>>> Sure, absolutely. I'm just big on first defining the metrics by which
>>> scale can be described, and THEN describing the test variations and
>>> iterations...
>>>
>>>>> My second suggestion would be to drop the requirement of using RDO --
>>>>> or any version of OpenStack for that matter.
>>>> My end goal would be to have scripts that one can run on any of the
>>>> OpenStack distribution.
>>>> RDO is mentioned here an example here.
>>> Probably worth just removing the RDO reference entirely from the wiki,
>>> since, as you agree below, benchmarking Keystone actually does not
>>> require installing OpenStack as a whole at all...
>>>
>>> Best,
>>> -jay
>>>
>>>>> In these kinds of tests, where you are not measuring the integrated
>>>>> performance of multiple endpoints, but are instead measuring the
>>>>> performance of a single endpoint (Keystone), there's no reason, IMHO,
>>>>> to install all of OpenStack. Installing and serving the Keystone
>>>>> server (and it's various drivers) is all that is needed. The fewer
>>>>> "balls up in the air" during a benchmarking session, the fewer
>>>>> side-effects are around to effect the outcome of the benchmark...
>>>> Agreed. As mentioned in following I suggested to install just Keystone
>>>> on the instances, where the tests would be performed :-
>>>>
>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u
>>>> sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck.
>>>>
>>>>
>>>> Thanks,
>>>> Neependra
>>>>
>>>>
>>>>> Best,
>>>>> -jay
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list:
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> Post to     :openstack at lists.openstack.org
>>>>> Unsubscribe :
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     :openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131218/ad68d6ca/attachment.html>