Open Stack

On 01/03/2014 02:08 PM, Joshua Harlow wrote:
> Awesome!!
>
> I am looking forward to any results u get. For said results when they 
> arrive make sure they are reproducible and accurate, and then people 
> can start solving any performance problems in a way that can be 
> compared to the initial baseline...
>
Thinking more on reproducible environment, I thought to experiment with 
/docker/
to configure Keystone and Rally. This would allow us to quickly compare 
different configuration
of Keystone, though I not sure about the overhead of /docker/. I have 
put down the details at:-
https://github.com/nkhare/keystone-performance

Any comments and suggestions are welcome.

Regards,
Neependra


> 2014 the year of rally :P
>
> Sent from my really tiny device...
>
> On Jan 2, 2014, at 11:32 PM, "Neependra Khare" <nkhare at redhat.com 
> <mailto:nkhare at redhat.com>> wrote:
>
>> Hi All,
>>
>> I wanted to share the current status of this work.
>>
>> Thanks to Boris and the team for implementing keystone benchmark with 
>> Rally.
>> It has been merged with upstream Rally code base couple of days back.
>>
>> I have tested above with devstack on Fedora 19 and Ubuntu 12.04.
>>
>> With Rally Deploy engine we can deploy OpenStack distribution like 
>> DevStack
>> before running the benchmark.
>> https://wiki.openstack.org/wiki/Rally/DeployEngines#DevstackEngine
>>
>> I am planning to prepare the /localrc/ file for /devstack/ by which 
>> we can configure
>> keystone with different configuration like with PKI/UUID tokens, with 
>> or without
>> memcached etc. This would help us is automating the process. I would be
>> documenting everything and would share it with the community.
>>
>> Regards,
>> Neependra
>>
>>
>> On 12/20/2013 02:09 PM, Neependra Khare wrote:
>>> Hi Joshua,
>>>
>>> On 12/20/2013 01:34 PM, Joshua Harlow wrote:
>>>> Awesome i will see if I can make it, maybe u guys can send out a 
>>>> summary afterwards? Glad to see performance work (and the 
>>>> associated tooling) get the attention it deserves!
>>> We had a meeting yesterday. Etherpad has most of things we discussed:-
>>> https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks
>>>
>>> We decided to have specific pattern in workload (like user name) for 
>>> keystone
>>> benchmarks as that would make clean-up easier.
>>>
>>> INIT methods like creating N users, would setup environment before 
>>> running the
>>> actual setup.  Jay suggested set that may be we can use some form of 
>>> snapshot
>>> for images, which have per-populated dataset. So that, we do not 
>>> have to spend
>>> time in INIT methods every time we run the tests.
>>>
>>> Next steps
>>> - Boris would have the clean-up and basic INIT functionality written 
>>> down for Keystone
>>> , which would then help us to write our first Keystone benchmark.
>>> - Meanwhile we'll update the benchmark we want to run with Rally in 
>>> /"//List of benchmarks
>>> that we would like to have//"/ section of the etherpad.
>>>
>>> Boris, Jay, Hugh, Tristan and others,
>>> If I missed any important point then please feel free to add here.
>>>
>>> Regards,
>>> Neependra
>>>
>>>
>>>>
>>>> Sent from my really tiny device...
>>>>
>>>> On Dec 18, 2013, at 5:32 AM, "Neependra Khare" <nkhare at redhat.com 
>>>> <mailto:nkhare at redhat.com>> wrote:
>>>>
>>>>> On 12/17/2013 05:19 PM, Neependra Khare wrote:
>>>>>> On 12/17/2013 12:48 AM, Joshua Harlow wrote:
>>>>>>> Another thing that u might consider.
>>>>>>>
>>>>>>> The rally project[1] has been using a tool called tomograph[2] and making
>>>>>>> tomograph better and it has been collecting some similar use-cases and
>>>>>>> test-cases around various openstack performance related work (also it has
>>>>>>> been working on defining said methodology, how to setup for performance
>>>>>>> tests, and more...).
>>>>>>>
>>>>>>> Some examples of various keystone/nova/glance related calls (see the low
>>>>>>> level trace information gathered there):
>>>>>> Thanks Josh.
>>>>>> We have started a blueprint to have keystone related benchmarks
>>>>>> with Rally.
>>>>>> https://blueprints.launchpad.net/rally/+spec/keystone-benchmark
>>>>> We are planning to have g+ hangout tomorrow (Dec 19, 2013)  to discuss
>>>>> Keystone related benchmarks with Rally at 11:00 EST
>>>>> https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks
>>>>>
>>>>> Feel free to join it. Just update your name on above mentioned 
>>>>> etherpad
>>>>> under Interested participants section.
>>>>>
>>>>> Regards,
>>>>> Neependra
>>>>>
>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Neependra
>>>>>>> -
>>>>>>> https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g
>>>>>>> lance-image-list.png
>>>>>>> -http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24
>>>>>>>
>>>>>>> It would seem like some of our scripts/test-cases would actually fit quite
>>>>>>> nicely into rally instead of being one-offs.
>>>>>>>
>>>>>>> I know that boris-42 would appreciate a single solution instead of
>>>>>>> one-offs. It seems like rally is becoming that.
>>>>>>>
>>>>>>> [1]https://wiki.openstack.org/wiki/Rally
>>>>>>> [2]https://github.com/stackforge/tomograph
>>>>>>>
>>>>>>> Jump in IRC and the rally team I'm sure would love to chat
>>>>>>> (#openstack-rally, freenode).
>>>>>>>
>>>>>>> -Josh
>>>>>>>
>>>>>>> On 12/16/13, 10:00 AM, "Jay Pipes"<jaypipes at gmail.com>  wrote:
>>>>>>>
>>>>>>>> On 12/16/2013 02:25 AM, Neependra Khare wrote:
>>>>>>>>> Hi Jay,
>>>>>>>>>
>>>>>>>>> Thanks for your comments.  Please find my reply in-line.
>>>>>>>>>
>>>>>>>>> On 12/14/2013 12:58 AM, Jay Pipes wrote:
>>>>>>>>>> I have listed down the methodology I'll be following for this test:-
>>>>>>>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis
>>>>>>>>>>> k.2C_Memory.2C_Database_bottlenecks
>>>>>>>>>>>
>>>>>>>>>> My first suggestion would be to rework the performance benchmarking
>>>>>>>>>> work items to have clearer indications regarding *what are the metrics
>>>>>>>>>> being tested* in each work item.
>>>>>>>>> Performance characterization is an iterative process. I am open to
>>>>>>>>> rework on the work-items as we
>>>>>>>>> go along.
>>>>>>>> Right, but the smaller the work item, the easier the iterations are :)
>>>>>>>>
>>>>>>>>>> For example, the first work item is "Identify CPU, Disk, Memory, and
>>>>>>>>>> Database Bottlenecks".
>>>>>>>>>>
>>>>>>>>>> The first test case listed is:
>>>>>>>>>>
>>>>>>>>>> "Test #1, Create users in parallel and look for CPU, disk or memory
>>>>>>>>>> bottleneck."
>>>>>>>>>>
>>>>>>>>>> I think that is a bit too big of an initial bite ;)
>>>>>>>>>>
>>>>>>>>>> Instead, it may be more effective to instead break down the
>>>>>>>>>> performance analysis based on the metrics you wish to test and the
>>>>>>>>>> relative conclusions you wish your work to generate.
>>>>>>>>>>
>>>>>>>>>> For example, consider this possible work item:
>>>>>>>>>>
>>>>>>>>>> "Determine the maximum number of token authentication calls that can
>>>>>>>>>> be performed"
>>>>>>>>> Tests like these would be very subjective to the hardware and software
>>>>>>>>> resources we have like
>>>>>>>>> no. of CPUs, Memcahced etc.  Its is very important to see if we can find
>>>>>>>>> any obvious bottlenecks.
>>>>>>>> No, that's not my point. When you have a specific metric like "number of
>>>>>>>> token authentication calls that can be performed in X minutes", you can
>>>>>>>> iterate based on singular changes -- not to the hardware, but to the
>>>>>>>> configuration of the software. If you are trying to solve the problem of
>>>>>>>> "where are my bottlenecks", without first identifying what metrics will
>>>>>>>> describe how a piece of software scales, then you are putting the cart
>>>>>>>> before the horse.
>>>>>>>>
>>>>>>>>>> Within that work item, you can then further expand a testing matrix,
>>>>>>>>>> like so:
>>>>>>>>>>
>>>>>>>>>> * Measure the total number of token authentication calls performed by
>>>>>>>>>> a single client against a single-process, Python-only Keystone server
>>>>>>>>>> * Measure the total number of token authentication calls performed by
>>>>>>>>>> a single client against a multi-process Keystone server running inside
>>>>>>>>>> an nginx or Apache container server -- with 2, 4, 8, 16, and 32
>>>>>>>>>> pre-forked processes
>>>>>>>>> Any pointers on configuring multi-process Keystone would be helpful. I
>>>>>>>>> see a method
>>>>>>>>> mentioned in "Run N keystone Processes" section of following:-
>>>>>>>>>
>>>>>>>>> http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf
>>>>>>>>> ormance"
>>>>>>>> Absolutely. You can spawn Keystone server in multiple pre-forked Apache
>>>>>>>> processes by configuring Keystone in an Apache vhost. Some general docs:
>>>>>>>>
>>>>>>>> http://docs.openstack.org/developer/keystone/apache-httpd.html
>>>>>>>>
>>>>>>>> Take a look at provision.sh script in eNovance's keystone-wsgi-apache
>>>>>>>> repo:
>>>>>>>>
>>>>>>>> https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#
>>>>>>>> L152
>>>>>>>>
>>>>>>>>>> * Measure the above using increasing numbers of concurrent clients --
>>>>>>>>>> 10, 50, 100, 500, 1000.
>>>>>>>>>>
>>>>>>>>>> There's, of course, nothing wrong with measuring things like CPU, disk
>>>>>>>>>> and I/O performance during tests, however there should be a clear
>>>>>>>>>> metric that is being measured for each test.
>>>>>>>>> Agreed. Let me start collecting results from the tests you suggested
>>>>>>>>> above and I mentioned
>>>>>>>>> on the wiki. Once we have those, we can rework on the work-items. Does
>>>>>>>>> that sound OK ?
>>>>>>>> Sure, absolutely. I'm just big on first defining the metrics by which
>>>>>>>> scale can be described, and THEN describing the test variations and
>>>>>>>> iterations...
>>>>>>>>
>>>>>>>>>> My second suggestion would be to drop the requirement of using RDO --
>>>>>>>>>> or any version of OpenStack for that matter.
>>>>>>>>> My end goal would be to have scripts that one can run on any of the
>>>>>>>>> OpenStack distribution.
>>>>>>>>> RDO is mentioned here an example here.
>>>>>>>> Probably worth just removing the RDO reference entirely from the wiki,
>>>>>>>> since, as you agree below, benchmarking Keystone actually does not
>>>>>>>> require installing OpenStack as a whole at all...
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> -jay
>>>>>>>>
>>>>>>>>>> In these kinds of tests, where you are not measuring the integrated
>>>>>>>>>> performance of multiple endpoints, but are instead measuring the
>>>>>>>>>> performance of a single endpoint (Keystone), there's no reason, IMHO,
>>>>>>>>>> to install all of OpenStack. Installing and serving the Keystone
>>>>>>>>>> server (and it's various drivers) is all that is needed. The fewer
>>>>>>>>>> "balls up in the air" during a benchmarking session, the fewer
>>>>>>>>>> side-effects are around to effect the outcome of the benchmark...
>>>>>>>>> Agreed. As mentioned in following I suggested to install just Keystone
>>>>>>>>> on the instances, where the tests would be performed :-
>>>>>>>>>
>>>>>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u
>>>>>>>>> sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Neependra
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> -jay
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Mailing list:
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>> Post to     :openstack at lists.openstack.org
>>>>>>>>>> Unsubscribe :
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list:
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>> Post to     :openstack at lists.openstack.org
>>>>>>>> Unsubscribe :
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>
>>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     :openstack at lists.openstack.org
>>> Unsubscribe :http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>> _______________________________________________
>> Mailing list: 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org 
>> <mailto:openstack at lists.openstack.org>
>> Unsubscribe : 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140108/d3ce9952/attachment.html>