<div dir="ltr"><div class="gmail_extra">Dolph,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Excuse the delayed reply, was waiting for a brilliant solution from someone. Without one, personally I'd prefer the cronjob as it seems to be the type of thing cron was designed for. That will be a painful change as people now rely on this behavior so I don't know if its feasible. I will be setting up monitoring for the revocation count and alerting me if it crosses probably 500 or so. If the problem gets worse then I think a custom no-op or sql driver is the next step.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 15, 2015 at 4:00 PM, Dolph Mathews <span dir="ltr"><<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Wed, Jul 15, 2015 at 4:51 PM, Matt Fischer <span dir="ltr"><<a href="mailto:matt@mattfischer.com" target="_blank">matt@mattfischer.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm having some issues with keystone revocation events. The bottom line is that due to the way keystone handles the clean-up of these events[1], having more than a few leads to:<div><br></div><div> - bad performance, up to 2x slower token validation with about 600 events based on my perf measurements.</div><div> - database deadlocks, which cause API calls to fail, more likely with more events it seems</div><div><br></div><div>I am seeing this behavior in code from trunk on June 11 using Fernet tokens, but the token backend does not seem to make a difference.</div><div><br></div><div>Here's what happens to the db in terms of deadlock:</div><div>2015-07-15 21:25:41.082 31800 TRACE keystone.common.wsgi DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'DELETE FROM revocation_event WHERE revocation_event.revoked_at < %s' (datetime.datetime(2015, 7, 15, 18, 55, 41, 55186),)<br></div><div><br></div><div>When this starts happening, I just go truncate the table, but this is not ideal. If [1] is really true then the design is not great, it sounds like keystone is doing a revocation event clean-up on every token validation call. Reading and deleting/locking from my db cluster is not something I want to do on every validate call.</div></div></blockquote><div><br></div></span><div>Unfortunately, that's *exactly* what keystone is doing. Adam and I had a conversation about this problem in Vancouver which directly resulted in opening the bug referenced on the operator list:<br><br> <a href="https://bugs.launchpad.net/keystone/+bug/1456797" target="_blank">https://bugs.launchpad.net/keystone/+bug/1456797</a><br><br></div><div>Neither of us remembered the actual implemented behavior, which is what you've run into and Deepti verified in the bug's comments.<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>So, can I turn of token revocation for now? I didn't see an obvious no-op driver.</div></div></blockquote><div><br></div></span><div>Not sure how, other than writing your own no-op driver, or perhaps an extended driver that doesn't try to clean the table on every read?<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>And in the long-run can this be fixed? I'd rather do almost anything else, including writing a cronjob than what happens now.</div></div></blockquote><div><br></div></span><div>If anyone has a better solution than the current one, that's also better than requiring a cron job on something like keystone-manage revocation_flush I'd love to hear it.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class=""><div dir="ltr"><div><br></div><div>[1] - <a href="http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html" target="_blank">http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html</a></div></div>
<br></span>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>