[Openstack] [Keystone] performance issues after havana upgrade

Jonathan Proulx jon at jonproulx.com
Sun Jan 12 17:31:31 UTC 2014


puzzling side effect?

I just made a small change to neutron.conf (adjusted a default quota)
and restarted neutron-server, now neutron (but not other services) is
spweing:

Invalid user token - rejecting request

(quite possibly only from dashboard requests CLI seems to work).  I've
tried restarting keystone (in both wsgi and eventlet modes),
restarting neutron-server w/ reverted config and flushing/restarting
memcached in various combinations.

I don't really see how restarting neutron-server could confuse token
validation...


On Sun, Jan 12, 2014 at 10:38 AM, Morgan Fainberg <morgan at metacloud.com> wrote:
> Thanks for confirming this!  It also validates my new logic going into
> icehouse (I might have had some ulterior motives here, or not so
> ulterior as the case may be).  I'll make sure we resolve the test
> issues (unrelated to the patch) and get it into the Havana tree so you
> don't need to maintain it outside of the releases.
>
> Cheers,
> Morgan
>
> Sent from my tablet-like-device
>
>> On Jan 11, 2014, at 11:01 PM, Jonathan Proulx <jon at jonproulx.com> wrote:
>>
>>> On Sat, Jan 11, 2014 at 10:57 PM, Morgan Fainberg <m at metacloud.com> wrote:
>>> Sounds good!  Just remember that prior to the fix I posted there, for each
>>> token in the user’s index, it incurred a round-trip to memcached to validate
>>> the token wasn’t expired.  This change makes it so that there are
>>> significantly less trips from keystone to memcached.
>>>
>>> If this doesn’t 100% solve the issue, we should start digging further into
>>> what is going on, but I am confident this will (at the very least) help a
>>> reasonable amount.
>>
>> You sir are a miracle worker, my hat is off!
>>
>> The responsiveness of everything is better than it's ever been, my
>> users will think this is the best feature the upgrade.
>>
>> For example earlier today I managed to launch 10 VMs in parallel,
>> eventually, I'd guess on the order of 5-10min.  One of my usual
>> acceptance tests is being able to launch 100 VMs in that time.  Just
>> now Iaunched 100 in <2min from request until they'd all been
>> provisioned and were booting.  Now there's too many moving pieces and
>> too few experimental samples to make any publishable claims, but your
>> patch is the only thing that changed.
>>
>> Thanks,
>> -Jon




More information about the Openstack mailing list