Open Stack

Hi Joshua,

On 12/20/2013 01:34 PM, Joshua Harlow wrote:
> Awesome i will see if I can make it, maybe u guys can send out a 
> summary afterwards? Glad to see performance work (and the associated 
> tooling) get the attention it deserves!
We had a meeting yesterday. Etherpad has most of things we discussed:-
https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks

We decided to have specific pattern in workload (like user name) for 
keystone
benchmarks as that would make clean-up easier.

INIT methods like creating N users, would setup environment before 
running the
actual setup.  Jay suggested set that may be we can use some form of 
snapshot
for images, which have per-populated dataset. So that, we do not have to 
spend
time in INIT methods every time we run the tests.

Next steps
- Boris would have the clean-up and basic INIT functionality written 
down for Keystone
, which would then help us to write our first Keystone benchmark.
- Meanwhile we'll update the benchmark we want to run with Rally in 
/"//List of benchmarks
that we would like to have//"/ section of the etherpad.

Boris, Jay, Hugh, Tristan and others,
If I missed any important point then please feel free to add here.

Regards,
Neependra


>
> Sent from my really tiny device...
>
> On Dec 18, 2013, at 5:32 AM, "Neependra Khare" <nkhare at redhat.com 
> <mailto:nkhare at redhat.com>> wrote:
>
>> On 12/17/2013 05:19 PM, Neependra Khare wrote:
>>> On 12/17/2013 12:48 AM, Joshua Harlow wrote:
>>>> Another thing that u might consider.
>>>>
>>>> The rally project[1] has been using a tool called tomograph[2] and making
>>>> tomograph better and it has been collecting some similar use-cases and
>>>> test-cases around various openstack performance related work (also it has
>>>> been working on defining said methodology, how to setup for performance
>>>> tests, and more...).
>>>>
>>>> Some examples of various keystone/nova/glance related calls (see the low
>>>> level trace information gathered there):
>>> Thanks Josh.
>>> We have started a blueprint to have keystone related benchmarks
>>> with Rally.
>>> https://blueprints.launchpad.net/rally/+spec/keystone-benchmark
>> We are planning to have g+ hangout tomorrow (Dec 19, 2013)  to discuss
>> Keystone related benchmarks with Rally at 11:00 EST
>> https://etherpad.openstack.org/p/Rally_Keystone_Benchmarks
>>
>> Feel free to join it. Just update your name on above mentioned etherpad
>> under Interested participants section.
>>
>> Regards,
>> Neependra
>>
>>
>>>
>>> Regards,
>>> Neependra
>>>> -
>>>> https://raw.github.com/stackforge/tomograph/master/doc/screenshots/zipkin-g
>>>> lance-image-list.png
>>>> -http://www.slideshare.net/mirantis/rally-benchmarkingatscale/24
>>>>
>>>> It would seem like some of our scripts/test-cases would actually fit quite
>>>> nicely into rally instead of being one-offs.
>>>>
>>>> I know that boris-42 would appreciate a single solution instead of
>>>> one-offs. It seems like rally is becoming that.
>>>>
>>>> [1]https://wiki.openstack.org/wiki/Rally
>>>> [2]https://github.com/stackforge/tomograph
>>>>
>>>> Jump in IRC and the rally team I'm sure would love to chat
>>>> (#openstack-rally, freenode).
>>>>
>>>> -Josh
>>>>
>>>> On 12/16/13, 10:00 AM, "Jay Pipes"<jaypipes at gmail.com>  wrote:
>>>>
>>>>> On 12/16/2013 02:25 AM, Neependra Khare wrote:
>>>>>> Hi Jay,
>>>>>>
>>>>>> Thanks for your comments.  Please find my reply in-line.
>>>>>>
>>>>>> On 12/14/2013 12:58 AM, Jay Pipes wrote:
>>>>>>> I have listed down the methodology I'll be following for this test:-
>>>>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Identify_CPU.2C_Dis
>>>>>>>> k.2C_Memory.2C_Database_bottlenecks
>>>>>>>>
>>>>>>> My first suggestion would be to rework the performance benchmarking
>>>>>>> work items to have clearer indications regarding *what are the metrics
>>>>>>> being tested* in each work item.
>>>>>> Performance characterization is an iterative process. I am open to
>>>>>> rework on the work-items as we
>>>>>> go along.
>>>>> Right, but the smaller the work item, the easier the iterations are :)
>>>>>
>>>>>>> For example, the first work item is "Identify CPU, Disk, Memory, and
>>>>>>> Database Bottlenecks".
>>>>>>>
>>>>>>> The first test case listed is:
>>>>>>>
>>>>>>> "Test #1, Create users in parallel and look for CPU, disk or memory
>>>>>>> bottleneck."
>>>>>>>
>>>>>>> I think that is a bit too big of an initial bite ;)
>>>>>>>
>>>>>>> Instead, it may be more effective to instead break down the
>>>>>>> performance analysis based on the metrics you wish to test and the
>>>>>>> relative conclusions you wish your work to generate.
>>>>>>>
>>>>>>> For example, consider this possible work item:
>>>>>>>
>>>>>>> "Determine the maximum number of token authentication calls that can
>>>>>>> be performed"
>>>>>> Tests like these would be very subjective to the hardware and software
>>>>>> resources we have like
>>>>>> no. of CPUs, Memcahced etc.  Its is very important to see if we can find
>>>>>> any obvious bottlenecks.
>>>>> No, that's not my point. When you have a specific metric like "number of
>>>>> token authentication calls that can be performed in X minutes", you can
>>>>> iterate based on singular changes -- not to the hardware, but to the
>>>>> configuration of the software. If you are trying to solve the problem of
>>>>> "where are my bottlenecks", without first identifying what metrics will
>>>>> describe how a piece of software scales, then you are putting the cart
>>>>> before the horse.
>>>>>
>>>>>>> Within that work item, you can then further expand a testing matrix,
>>>>>>> like so:
>>>>>>>
>>>>>>> * Measure the total number of token authentication calls performed by
>>>>>>> a single client against a single-process, Python-only Keystone server
>>>>>>> * Measure the total number of token authentication calls performed by
>>>>>>> a single client against a multi-process Keystone server running inside
>>>>>>> an nginx or Apache container server -- with 2, 4, 8, 16, and 32
>>>>>>> pre-forked processes
>>>>>> Any pointers on configuring multi-process Keystone would be helpful. I
>>>>>> see a method
>>>>>> mentioned in "Run N keystone Processes" section of following:-
>>>>>>
>>>>>> http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Perf
>>>>>> ormance"
>>>>> Absolutely. You can spawn Keystone server in multiple pre-forked Apache
>>>>> processes by configuring Keystone in an Apache vhost. Some general docs:
>>>>>
>>>>> http://docs.openstack.org/developer/keystone/apache-httpd.html
>>>>>
>>>>> Take a look at provision.sh script in eNovance's keystone-wsgi-apache
>>>>> repo:
>>>>>
>>>>> https://github.com/enovance/keystone-wsgi-apache/blob/master/provision.sh#
>>>>> L152
>>>>>
>>>>>>> * Measure the above using increasing numbers of concurrent clients --
>>>>>>> 10, 50, 100, 500, 1000.
>>>>>>>
>>>>>>> There's, of course, nothing wrong with measuring things like CPU, disk
>>>>>>> and I/O performance during tests, however there should be a clear
>>>>>>> metric that is being measured for each test.
>>>>>> Agreed. Let me start collecting results from the tests you suggested
>>>>>> above and I mentioned
>>>>>> on the wiki. Once we have those, we can rework on the work-items. Does
>>>>>> that sound OK ?
>>>>> Sure, absolutely. I'm just big on first defining the metrics by which
>>>>> scale can be described, and THEN describing the test variations and
>>>>> iterations...
>>>>>
>>>>>>> My second suggestion would be to drop the requirement of using RDO --
>>>>>>> or any version of OpenStack for that matter.
>>>>>> My end goal would be to have scripts that one can run on any of the
>>>>>> OpenStack distribution.
>>>>>> RDO is mentioned here an example here.
>>>>> Probably worth just removing the RDO reference entirely from the wiki,
>>>>> since, as you agree below, benchmarking Keystone actually does not
>>>>> require installing OpenStack as a whole at all...
>>>>>
>>>>> Best,
>>>>> -jay
>>>>>
>>>>>>> In these kinds of tests, where you are not measuring the integrated
>>>>>>> performance of multiple endpoints, but are instead measuring the
>>>>>>> performance of a single endpoint (Keystone), there's no reason, IMHO,
>>>>>>> to install all of OpenStack. Installing and serving the Keystone
>>>>>>> server (and it's various drivers) is all that is needed. The fewer
>>>>>>> "balls up in the air" during a benchmarking session, the fewer
>>>>>>> side-effects are around to effect the outcome of the benchmark...
>>>>>> Agreed. As mentioned in following I suggested to install just Keystone
>>>>>> on the instances, where the tests would be performed :-
>>>>>>
>>>>>> https://wiki.openstack.org/wiki/KeystonePerformance#Test_.231.2C_Create_u
>>>>>> sers_in_parallel_and_look_for_CPU.2C_disk_or_memory_bottleneck.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Neependra
>>>>>>
>>>>>>
>>>>>>> Best,
>>>>>>> -jay
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list:
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>> Post to     :openstack at lists.openstack.org
>>>>>>> Unsubscribe :
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> _______________________________________________
>>>>> Mailing list:
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> Post to     :openstack at lists.openstack.org
>>>>> Unsubscribe :
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131220/a7e47067/attachment.html>