[openstack-dev] [keystone] token revocation woes
matt at mattfischer.com
Tue Jul 21 22:58:37 UTC 2015
Excuse the delayed reply, was waiting for a brilliant solution from
someone. Without one, personally I'd prefer the cronjob as it seems to be
the type of thing cron was designed for. That will be a painful change as
people now rely on this behavior so I don't know if its feasible. I will be
setting up monitoring for the revocation count and alerting me if it
crosses probably 500 or so. If the problem gets worse then I think a custom
no-op or sql driver is the next step.
On Wed, Jul 15, 2015 at 4:00 PM, Dolph Mathews <dolph.mathews at gmail.com>
> On Wed, Jul 15, 2015 at 4:51 PM, Matt Fischer <matt at mattfischer.com>
>> I'm having some issues with keystone revocation events. The bottom line
>> is that due to the way keystone handles the clean-up of these events,
>> having more than a few leads to:
>> - bad performance, up to 2x slower token validation with about 600
>> events based on my perf measurements.
>> - database deadlocks, which cause API calls to fail, more likely with
>> more events it seems
>> I am seeing this behavior in code from trunk on June 11 using Fernet
>> tokens, but the token backend does not seem to make a difference.
>> Here's what happens to the db in terms of deadlock:
>> 2015-07-15 21:25:41.082 31800 TRACE keystone.common.wsgi DBDeadlock:
>> (OperationalError) (1213, 'Deadlock found when trying to get lock; try
>> restarting transaction') 'DELETE FROM revocation_event WHERE
>> revocation_event.revoked_at < %s' (datetime.datetime(2015, 7, 15, 18, 55,
>> 41, 55186),)
>> When this starts happening, I just go truncate the table, but this is not
>> ideal. If  is really true then the design is not great, it sounds like
>> keystone is doing a revocation event clean-up on every token validation
>> call. Reading and deleting/locking from my db cluster is not something I
>> want to do on every validate call.
> Unfortunately, that's *exactly* what keystone is doing. Adam and I had a
> conversation about this problem in Vancouver which directly resulted in
> opening the bug referenced on the operator list:
> Neither of us remembered the actual implemented behavior, which is what
> you've run into and Deepti verified in the bug's comments.
>> So, can I turn of token revocation for now? I didn't see an obvious no-op
> Not sure how, other than writing your own no-op driver, or perhaps an
> extended driver that doesn't try to clean the table on every read?
>> And in the long-run can this be fixed? I'd rather do almost anything
>> else, including writing a cronjob than what happens now.
> If anyone has a better solution than the current one, that's also better
> than requiring a cron job on something like keystone-manage
> revocation_flush I'd love to hear it.
>>  -
>> OpenStack Development Mailing List (not for usage questions)
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev