Open Stack

Hi All,

Current status of this work.

As you may know with Rally we can deploy OpenStack using Devstack before 
running
the benchmark. I have been putting some effort to make this easier by 
providing a
custom /localrc /file which can be copied to the /Devstack /environment 
either from Rally
client or from a URL and to cleanup the devstack environment after the 
test is done.
https://review.openstack.org/#/c/66786/
https://review.openstack.org/#/c/66800/

I am expecting that these will get merged to Rally master branch pretty 
soon.

If anyone has /localrc /files for different /Keystone /configuration 
then feel free to
share them. We can use them as a reference for benchmark runs.

Thanks,
Neependra

On 01/03/2014 12:36 PM, Neependra Khare wrote:
> Hi All,
>
> I wanted to share the current status of this work.
>
> Thanks to Boris and the team for implementing keystone benchmark with 
> Rally.
> It has been merged with upstream Rally code base couple of days back.
>
> I have tested above with devstack on Fedora 19 and Ubuntu 12.04.
>
> With Rally Deploy engine we can deploy OpenStack distribution like 
> DevStack
> before running the benchmark.
> https://wiki.openstack.org/wiki/Rally/DeployEngines#DevstackEngine
>
> I am planning to prepare the /localrc/ file for /devstack/ by which we 
> can configure
> keystone with different configuration like with PKI/UUID tokens, with 
> or without
> memcached etc. This would help us is automating the process. I would be
> documenting everything and would share it with the community.
>
> Regards,
> Neependra
>
>
> On 12/20/2013 02:09 PM, Neependra Khare wrote:
>> Hi Joshua,
>>
>> On 12/20/2013 01:34 PM, Joshua Harlow wrote:
>>> Awesome i will see if I can make it, maybe u guys can send out a 
>>> summary afterwards? Glad to see performance work (and the associated 
>>> tooling) get the attention it deserves!
>> We had a meeting yesterday. Etherpad has most of things we discussed:-
>> https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks
>>
>> We decided to have specific pattern in workload (like user name) for 
>> keystone
>> benchmarks as that would make clean-up easier.
>>
>> INIT methods like creating N users, would setup environment before 
>> running the
>> actual setup.  Jay suggested set that may be we can use some form of 
>> snapshot
>> for images, which have per-populated dataset. So that, we do not have 
>> to spend
>> time in INIT methods every time we run the tests.
>>
>> Next steps
>> - Boris would have the clean-up and basic INIT functionality written 
>> down for Keystone
>> , which would then help us to write our first Keystone benchmark.
>> - Meanwhile we'll update the benchmark we want to run with Rally in 
>> /"//List of benchmarks
>> that we would like to have//"/ section of the etherpad.
>>
>> Boris, Jay, Hugh, Tristan and others,
>> If I missed any important point then please feel free to add here.
>>
>> Regards,
>> Neependra
>>
>>
>>>
>>> Sent from my really tiny device...
>>>
>>> On Dec 18, 2013, at 5:32 AM, "Neependra Khare" <nkhare at redhat.com 
>>> <mailto:nkhare at redhat.com>> wrote:
>>>
>>>> On 12/17/2013 05:19 PM, Neependra Khare wrote:
>>>>> On 12/17/2013 12:48 AM, Joshua Harlow wrote:
>>>>>> Another thing that u might consider.
>>>>>>
>>>>>> The rally project[1] has been using a tool called tomograph[2] and making
>>>>>> tomograph better and it has been collecting some similar use-cases and
>>>>>> test-cases around various openstack performance related work (also it has
>>>>>> been working on defining said methodology, how to setup for performance
>>>>>> tests, and more...).
>>>>>>
>>>>>> Some examples of various keystone/nova/glance related calls (see the low
>>>>>> level trace information gathered there):
>>>>> Thanks Josh.
>>>>> We have started a blueprint to have keystone related benchmarks
>>>>> with Rally.
>>>>> https://blueprints.launchpad.net/rally/+spec/keystone-benchmark
>>>> We are planning to have g+ hangout tomorrow (Dec 19, 2013)  to discuss
>>>> Keystone related benchmarks with Rally at 11:00 EST
>>>> https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks
>>>>
>>>> Feel free to join it. Just update your name on above mentioned 
>>>> etherpad
>>>> under Interested participants section.
>>>>
>>>> Regards,
>>>> Neependra
>>>>
>>>>
>>>>>
>>>>> Regards,
>>>>> Neependra
>>>>>> -
>>>>>> https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g
>>>>>> lance-image-list.png
>>>>>> -http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24
>>>>>>
>>>>>> It would seem like some of our scripts/test-cases would actually fit quite
>>>>>> nicely into rally instead of being one-offs.
>>>>>>
>>>>>> I know that boris-42 would appreciate a single solution instead of
>>>>>> one-offs. It seems like rally is becoming that.
>>>>>>
>>>>>> [1]https://wiki.openstack.org/wiki/Rally
>>>>>> [2]https://github.com/stackforge/tomograph
>>>>>>
>>>>>> Jump in IRC and the rally team I'm sure would love to chat
>>>>>> (#openstack-rally, freenode).
>>>>>>
>>>>>> -Josh
>>>>>>
>>>>>> On 12/16/13, 10:00 AM, "Jay Pipes"<jaypipes at gmail.com>  wrote:
>>>>>>
>>>>>>> On 12/16/2013 02:25 AM, Neependra Khare wrote:
>>>>>>>> Hi Jay,
>>>>>>>>
>>>>>>>> Thanks for your comments.  Please find my reply in-line.
>>>>>>>>
>>>>>>>> On 12/14/2013 12:58 AM, Jay Pipes wrote:
>>>>>>>>> I have listed down the methodology I'll be following for this test:-
>>>>>>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis
>>>>>>>>>> k.2C_Memory.2C_Database_bottlenecks
>>>>>>>>>>
>>>>>>>>> My first suggestion would be to rework the performance benchmarking
>>>>>>>>> work items to have clearer indications regarding *what are the metrics
>>>>>>>>> being tested* in each work item.
>>>>>>>> Performance characterization is an iterative process. I am open to
>>>>>>>> rework on the work-items as we
>>>>>>>> go along.
>>>>>>> Right, but the smaller the work item, the easier the iterations are :)
>>>>>>>
>>>>>>>>> For example, the first work item is "Identify CPU, Disk, Memory, and
>>>>>>>>> Database Bottlenecks".
>>>>>>>>>
>>>>>>>>> The first test case listed is:
>>>>>>>>>
>>>>>>>>> "Test #1, Create users in parallel and look for CPU, disk or memory
>>>>>>>>> bottleneck."
>>>>>>>>>
>>>>>>>>> I think that is a bit too big of an initial bite ;)
>>>>>>>>>
>>>>>>>>> Instead, it may be more effective to instead break down the
>>>>>>>>> performance analysis based on the metrics you wish to test and the
>>>>>>>>> relative conclusions you wish your work to generate.
>>>>>>>>>
>>>>>>>>> For example, consider this possible work item:
>>>>>>>>>
>>>>>>>>> "Determine the maximum number of token authentication calls that can
>>>>>>>>> be performed"
>>>>>>>> Tests like these would be very subjective to the hardware and software
>>>>>>>> resources we have like
>>>>>>>> no. of CPUs, Memcahced etc.  Its is very important to see if we can find
>>>>>>>> any obvious bottlenecks.
>>>>>>> No, that's not my point. When you have a specific metric like "number of
>>>>>>> token authentication calls that can be performed in X minutes", you can
>>>>>>> iterate based on singular changes -- not to the hardware, but to the
>>>>>>> configuration of the software. If you are trying to solve the problem of
>>>>>>> "where are my bottlenecks", without first identifying what metrics will
>>>>>>> describe how a piece of software scales, then you are putting the cart
>>>>>>> before the horse.
>>>>>>>
>>>>>>>>> Within that work item, you can then further expand a testing matrix,
>>>>>>>>> like so:
>>>>>>>>>
>>>>>>>>> * Measure the total number of token authentication calls performed by
>>>>>>>>> a single client against a single-process, Python-only Keystone server
>>>>>>>>> * Measure the total number of token authentication calls performed by
>>>>>>>>> a single client against a multi-process Keystone server running inside
>>>>>>>>> an nginx or Apache container server -- with 2, 4, 8, 16, and 32
>>>>>>>>> pre-forked processes
>>>>>>>> Any pointers on configuring multi-process Keystone would be helpful. I
>>>>>>>> see a method
>>>>>>>> mentioned in "Run N keystone Processes" section of following:-
>>>>>>>>
>>>>>>>> http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf
>>>>>>>> ormance"
>>>>>>> Absolutely. You can spawn Keystone server in multiple pre-forked Apache
>>>>>>> processes by configuring Keystone in an Apache vhost. Some general docs:
>>>>>>>
>>>>>>> http://docs.openstack.org/developer/keystone/apache-httpd.html
>>>>>>>
>>>>>>> Take a look at provision.sh script in eNovance's keystone-wsgi-apache
>>>>>>> repo:
>>>>>>>
>>>>>>> https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#
>>>>>>> L152
>>>>>>>
>>>>>>>>> * Measure the above using increasing numbers of concurrent clients --
>>>>>>>>> 10, 50, 100, 500, 1000.
>>>>>>>>>
>>>>>>>>> There's, of course, nothing wrong with measuring things like CPU, disk
>>>>>>>>> and I/O performance during tests, however there should be a clear
>>>>>>>>> metric that is being measured for each test.
>>>>>>>> Agreed. Let me start collecting results from the tests you suggested
>>>>>>>> above and I mentioned
>>>>>>>> on the wiki. Once we have those, we can rework on the work-items. Does
>>>>>>>> that sound OK ?
>>>>>>> Sure, absolutely. I'm just big on first defining the metrics by which
>>>>>>> scale can be described, and THEN describing the test variations and
>>>>>>> iterations...
>>>>>>>
>>>>>>>>> My second suggestion would be to drop the requirement of using RDO --
>>>>>>>>> or any version of OpenStack for that matter.
>>>>>>>> My end goal would be to have scripts that one can run on any of the
>>>>>>>> OpenStack distribution.
>>>>>>>> RDO is mentioned here an example here.
>>>>>>> Probably worth just removing the RDO reference entirely from the wiki,
>>>>>>> since, as you agree below, benchmarking Keystone actually does not
>>>>>>> require installing OpenStack as a whole at all...
>>>>>>>
>>>>>>> Best,
>>>>>>> -jay
>>>>>>>
>>>>>>>>> In these kinds of tests, where you are not measuring the integrated
>>>>>>>>> performance of multiple endpoints, but are instead measuring the
>>>>>>>>> performance of a single endpoint (Keystone), there's no reason, IMHO,
>>>>>>>>> to install all of OpenStack. Installing and serving the Keystone
>>>>>>>>> server (and it's various drivers) is all that is needed. The fewer
>>>>>>>>> "balls up in the air" during a benchmarking session, the fewer
>>>>>>>>> side-effects are around to effect the outcome of the benchmark...
>>>>>>>> Agreed. As mentioned in following I suggested to install just Keystone
>>>>>>>> on the instances, where the tests would be performed :-
>>>>>>>>
>>>>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u
>>>>>>>> sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Neependra
>>>>>>>>
>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> -jay
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list:
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>> Post to     :openstack at lists.openstack.org
>>>>>>>>> Unsubscribe :
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>> _______________________________________________
>>>>>>> Mailing list:
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>> Post to     :openstack at lists.openstack.org
>>>>>>> Unsubscribe :
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>
>>>>
>>
>>
>>
>> _______________________________________________
>> Mailing list:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     :openstack at lists.openstack.org
>> Unsubscribe :http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140117/3461ab2a/attachment.html>