<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>This is an indicator that the bottleneck is not the db strictly speaking, but also related to the way we match. This means we need to spend some serious cycles on improving both the stored record(s) for revocation events and the matching algorithm. <br><br>Sent via mobile</div><div><br>On Jul 22, 2015, at 11:51, Matt Fischer <<a href="mailto:matt@mattfischer.com">matt@mattfischer.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Dolph,<div><br></div><div>Per our IRC discussion, I was unable to see any performance improvement here although not calling DELETE so often will reduce the number of deadlocks when we're under heavy load especially given the globally replicated DB we use.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 21, 2015 at 5:26 PM, Dolph Mathews <span dir="ltr"><<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Well, you might be in luck! Morgan Fainberg actually implemented an improvement that was apparently documented by Adam Young way back in March: <div><br> <a href="https://bugs.launchpad.net/keystone/+bug/1287757" target="_blank">https://bugs.launchpad.net/keystone/+bug/1287757</a><br></div><div><br></div><div>There's a link to the stable/kilo backport in comment #2 - I'd be eager to hear how it performs for you!</div><div><div class="h5"><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 21, 2015 at 5:58 PM, Matt Fischer <span dir="ltr"><<a href="mailto:matt@mattfischer.com" target="_blank">matt@mattfischer.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">Dolph,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Excuse the delayed reply, was waiting for a brilliant solution from someone. Without one, personally I'd prefer the cronjob as it seems to be the type of thing cron was designed for. That will be a painful change as people now rely on this behavior so I don't know if its feasible. I will be setting up monitoring for the revocation count and alerting me if it crosses probably 500 or so. If the problem gets worse then I think a custom no-op or sql driver is the next step.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks.</div><div><div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 15, 2015 at 4:00 PM, Dolph Mathews <span dir="ltr"><<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Jul 15, 2015 at 4:51 PM, Matt Fischer <span dir="ltr"><<a href="mailto:matt@mattfischer.com" target="_blank">matt@mattfischer.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm having some issues with keystone revocation events. The bottom line is that due to the way keystone handles the clean-up of these events[1], having more than a few leads to:<div><br></div><div> - bad performance, up to 2x slower token validation with about 600 events based on my perf measurements.</div><div> - database deadlocks, which cause API calls to fail, more likely with more events it seems</div><div><br></div><div>I am seeing this behavior in code from trunk on June 11 using Fernet tokens, but the token backend does not seem to make a difference.</div><div><br></div><div>Here's what happens to the db in terms of deadlock:</div><div>2015-07-15 21:25:41.082 31800 TRACE keystone.common.wsgi DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'DELETE FROM revocation_event WHERE revocation_event.revoked_at < %s' (datetime.datetime(2015, 7, 15, 18, 55, 41, 55186),)<br></div><div><br></div><div>When this starts happening, I just go truncate the table, but this is not ideal. If [1] is really true then the design is not great, it sounds like keystone is doing a revocation event clean-up on every token validation call. Reading and deleting/locking from my db cluster is not something I want to do on every validate call.</div></div></blockquote><div><br></div></span><div>Unfortunately, that's *exactly* what keystone is doing. Adam and I had a conversation about this problem in Vancouver which directly resulted in opening the bug referenced on the operator list:<br><br> <a href="https://bugs.launchpad.net/keystone/+bug/1456797" target="_blank">https://bugs.launchpad.net/keystone/+bug/1456797</a><br><br></div><div>Neither of us remembered the actual implemented behavior, which is what you've run into and Deepti verified in the bug's comments.<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>So, can I turn of token revocation for now? I didn't see an obvious no-op driver.</div></div></blockquote><div><br></div></span><div>Not sure how, other than writing your own no-op driver, or perhaps an extended driver that doesn't try to clean the table on every read?<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>And in the long-run can this be fixed? I'd rather do almost anything else, including writing a cronjob than what happens now.</div></div></blockquote><div><br></div></span><div>If anyone has a better solution than the current one, that's also better than requiring a cron job on something like keystone-manage revocation_flush I'd love to hear it.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><span><div dir="ltr"><div><br></div><div>[1] - <a href="http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html" target="_blank">http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html</a></div></div>
<br></span>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div></div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></blockquote><blockquote type="cite"><div><span>__________________________________________________________________________</span><br><span>OpenStack Development Mailing List (not for usage questions)</span><br><span>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe</span><br><span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><br></div></blockquote></body></html>