[openstack-dev] [keystone][all] Incorporating performance feedback into the review process
boris at pavlovic.me
Fri Jun 10 23:01:06 UTC 2016
I share just how it looked from my side.
I really support your idea (no matter what you pick to use your
tooling/rally/jmeter) it is very valuable, especially if it will become
This really should be done by someone.
On Fri, Jun 10, 2016 at 3:26 PM, Lance Bragstad <lbragstad at gmail.com> wrote:
> 1. I care about performance. I just believe that a big hurdle has been
> finding infrastructure that allows us to run performance tests in a
> consistent manner. Dedicated infrastructure plays a big role in this,
> which is hard (if not impossible) to obtain in the gate - making the gate
> a suboptimal place for performance testing. Consistency is also an issue
> because the gate is comprised of resources donated from several different
> providers. Matt lays this out pretty well in his reply above. This sounds
> like a TODO to hook rally into the keystone-performance/ansible pipeline,
> then we would have rally and keystone running on bare metal.
> 2. See response to #5.
> 3. What were the changes made to keystone that caused rally to fail?
> If you have some links I'd be curious to revisit them and improve them if I
> 4. Blocked because changes weren't reviewed? As far as I know
> OSProfiler is in keystone's default pipeline.
> 5. It doesn't look like there are any open patches for rally
> integration with keystone . The closed ones have either been
> merged  or abandon  because they are
> work-in-progress or unattended.
> I'm only looking for this bot to leave a comment. I don't intend on it
> being a voting job any time soon, it's just providing a datapoint for
> patches that we suspect to have an impact on performance. It's running on
> dedicated hardware, but only from a single service provider - so mileage
> may vary depending on where and how you run keystone. But, it does take us
> a step in the right direction. People don't have to use it if they don't
> want to.
> Thanks for the feedback!
>  https://review.openstack.org/#/c/240251/
>  https://review.openstack.org/#/c/188457/
>  https://review.openstack.org/#/c/188352/
>  https://review.openstack.org/#/c/90405/
>  https://review.openstack.org/#/c/301367/
>  https://review.openstack.org/#/c/188479/
>  https://review.openstack.org/#/c/98836/
>  https://review.openstack.org/#/c/91677/
> On Fri, Jun 10, 2016 at 4:26 PM, Boris Pavlovic <boris at pavlovic.me> wrote:
>> It is amazing effort, I am wishing you good luck with Keystone team,
>> however i faced some issues when I started similar effort
>> about 3 years ago with Rally. Here are some points, that are going to be
>> very useful for you:
>> 1. I think that Keystone team doesn't care about performance &
>> scalability at all
>> 2. Keystone team ignored/discard all help from Rally team to make
>> this effort successful
>> 3. When Rally job started failing, because of introduced performance
>> issues in Keystone, they decided to remove job
>> 4. They blocked almost forever work on OSProfiler so we are blind and
>> can't see where is the issue in code
>> 5. They didn't help to develop any Rally plugin or even review the
>> Rally test cases that we proposed to them
>> Best regards,
>> Boris Pavlovic
>> On Mon, Jun 6, 2016 at 10:45 AM, Clint Byrum <clint at fewbar.com> wrote:
>>> Excerpts from Brant Knudson's message of 2016-06-03 15:16:20 -0500:
>>> > On Fri, Jun 3, 2016 at 2:35 PM, Lance Bragstad <lbragstad at gmail.com>
>>> > > Hey all,
>>> > >
>>> > > I have been curious about impact of providing performance feedback
>>> as part
>>> > > of the review process. From what I understand, keystone used to have
>>> > > performance job that would run against proposed patches (I've only
>>> > > about it so someone else will have to keep me honest about its
>>> > > but it sounds like it wasn't valued.
>>> > >
>>> > >
>>> > We had a job running rally for a year (I think) that nobody ever
>>> looked at
>>> > so we decided it was a waste and stopped running it.
>>> > > I think revisiting this topic is valuable, but it raises a series of
>>> > > questions.
>>> > >
>>> > > Initially it probably only makes sense to test a reasonable set of
>>> > > defaults. What do we want these defaults to be? Should they be
>>> > > by DevStack, openstack-ansible, or something else?
>>> > >
>>> > >
>>> > A performance test is going to depend on the environment (the machines,
>>> > disks, network, etc), the existing data (tokens, revocations, users,
>>> > and the config (fernet, uuid, caching, etc.). If these aren't
>>> > between runs then the results are not going to be usable. (This is the
>>> > problem with running rally on infra hardware.) If the data isn't
>>> > (1000s of tokens, etc.) then the results are going to be at best not
>>> > or at worst misleading.
>>> That's why I started the counter-inspection spec:
>>> It just tries to count operations, and graph those. I've, unfortunately,
>>> been pulled off to other things of late, but I do intend to loop back
>>> and hit this hard over the next few months to try and get those graphs.
>>> What we'd get initially is just graphs of how many messages we push
>>> through RabbitMQ, and how many rows/queries/transactions we push through
>>> mysql. We may also want to add counters like how many API requests
>>> happened, and how many retries happen inside the code itself.
>>> There's a _TON_ we can do now to ensure that we know what the trends are
>>> when something gets "slow", so we can look for a gradual "death by 1000
>>> papercuts" trend or a hockey stick that can be tied to a particular
>>> > What does the performance test criteria look like and where does it
>>> > > Does it just consist of running tempest?
>>> > >
>>> > >
>>> > I don't think tempest is going to give us numbers that we're looking
>>> > for performance. I've seen a few scripts and have my own for testing
>>> > performance of token validation, token creation, user creation, etc.
>>> > I think will do the exact tests we want and we can get the results
>>> > formatted however we like.
>>> Agreed that tempest will only give a limited view. Ideally one would
>>> also test things like "after we've booted 1000 vms, do we end up reading
>>> 1000 more rows, or 1000 * 1000 more rows.
>>> > From a contributor and reviewer perspective, it would be nice to have
>>> > > ability to compare performance results across patch sets. I
>>> understand that
>>> > > keeping all performance results for every patch for an extended
>>> period of
>>> > > time is unrealistic. Maybe we take a daily performance snapshot
>>> > > master and use that to map performance patterns over time?
>>> > >
>>> > >
>>> > Where are you planning to store the results?
>>> Infra has a graphite/statsd cluster which is made for collecting metrics
>>> on tests. It might need to be expanded a bit, but it should be
>>> relatively cheap to do so given the benefit of having some of these
>>> OpenStack Development Mailing List (not for usage questions)
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> OpenStack Development Mailing List (not for usage questions)
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev