<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 07/23/2015 10:45 AM, Lance Bragstad
wrote:<br>
</div>
<blockquote
cite="mid:CAE6oFcFdPxdYUb05e4esw3jMeV7TrRUP2PtVk7kfBvrzGLxZxg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Jul 22, 2015 at 10:06 PM,
Adam Young <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ayoung@redhat.com" target="_blank">ayoung@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<div>On 07/22/2015 05:39 PM, Adam Young wrote:<br>
</div>
<blockquote type="cite">
<div>On 07/22/2015 03:41 PM, Morgan Fainberg wrote:<br>
</div>
<blockquote type="cite">
<div>This is an indicator that the bottleneck is
not the db strictly speaking, but also related
to the way we match. This means we need to spend
some serious cycles on improving both the stored
record(s) for revocation events and the matching
algorithm. <br>
</div>
</blockquote>
<br>
The simplest approach to revocation checking is to
do a linear search through the events. I think the
old version of the code that did that is in a code
review, and I will pull it out.<br>
<br>
If we remove the tree, then the matching will have
to run through each of the records and see if there
is a match; the test will be linear with the number
of records (slightly shorter if a token is actually
revoked).<br>
</blockquote>
<br>
</span> This was the origianal, linear search version of
the code.<br>
<br>
<a moz-do-not-send="true"
href="https://review.openstack.org/#/c/55908/50/keystone/contrib/revoke/model.py,cm"
target="_blank">https://review.openstack.org/#/c/55908/50/keystone/contrib/revoke/model.py,cm</a>
<div>
<div class="h5"><br>
<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>What initially landed for Revocation Events was the
tree-structure, right? We didn't land a linear approach
prior to that and then switch to the tree, did we? <br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Correct. The tree version made it into the review stack about
revision 60 or so.<br>
<br>
It might be useful to think of a hybrid approach. For example, it
makes no sense to do a linear search for revoke-by-userid, as that
will be the most common revocation, and also the one that will
benefit the most from a speed up. A binary search would probably
make more sense there. However, it does mean that the events would
have to be sorted, which might be more costly than the search
itself, depending on how often it is performed.<br>
<br>
The one way that the events should always be sorted is by the
"revoked before" time, and this value can be found via a binary
search.<br>
<br>
<br>
<br>
<br>
<br>
<blockquote
cite="mid:CAE6oFcFdPxdYUb05e4esw3jMeV7TrRUP2PtVk7kfBvrzGLxZxg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<div class="h5">
<blockquote type="cite"> <br>
<br>
<br>
<br>
<br>
<blockquote type="cite">
<div><br>
Sent via mobile</div>
<div><br>
On Jul 22, 2015, at 11:51, Matt Fischer <<a
moz-do-not-send="true"
href="mailto:matt@mattfischer.com"
target="_blank">matt@mattfischer.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Dolph,
<div><br>
</div>
<div>Per our IRC discussion, I was unable
to see any performance improvement here
although not calling DELETE so often
will reduce the number of deadlocks when
we're under heavy load especially given
the globally replicated DB we use.</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jul 21,
2015 at 5:26 PM, Dolph Mathews <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:dolph.mathews@gmail.com"
target="_blank">dolph.mathews@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Well, you might be in
luck! Morgan Fainberg actually
implemented an improvement that was
apparently documented by Adam Young
way back in March:
<div><br>
<a moz-do-not-send="true"
href="https://bugs.launchpad.net/keystone/+bug/1287757"
target="_blank">https://bugs.launchpad.net/keystone/+bug/1287757</a><br>
</div>
<div><br>
</div>
<div>There's a link to the
stable/kilo backport in comment #2
- I'd be eager to hear how it
performs for you!</div>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Jul 21, 2015 at 5:58
PM, Matt Fischer <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:matt@mattfischer.com" target="_blank">matt@mattfischer.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div
class="gmail_extra">Dolph,</div>
<div
class="gmail_extra"><br>
</div>
<div
class="gmail_extra">Excuse
the delayed reply,
was waiting for a
brilliant solution
from someone.
Without one,
personally I'd
prefer the cronjob
as it seems to be
the type of thing
cron was designed
for. That will be a
painful change as
people now rely on
this behavior so I
don't know if its
feasible. I will be
setting up
monitoring for the
revocation count and
alerting me if it
crosses probably 500
or so. If the
problem gets worse
then I think a
custom no-op or sql
driver is the next
step.</div>
<div
class="gmail_extra"><br>
</div>
<div
class="gmail_extra">Thanks.</div>
<div>
<div>
<div
class="gmail_extra"><br>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Jul 15,
2015 at 4:00
PM, Dolph
Mathews <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:dolph.mathews@gmail.com"
target="_blank">dolph.mathews@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><br>
<div
class="gmail_extra"><br>
<div
class="gmail_quote"><span>On
Wed, Jul 15,
2015 at 4:51
PM, Matt
Fischer <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:matt@mattfischer.com"
target="_blank">matt@mattfischer.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I'm
having some
issues with
keystone
revocation
events. The
bottom line is
that due to
the way
keystone
handles the
clean-up of
these
events[1],
having more
than a few
leads to:
<div><br>
</div>
<div> - bad
performance,
up to 2x
slower token
validation
with about 600
events based
on my perf
measurements.</div>
<div> -
database
deadlocks,
which cause
API calls to
fail, more
likely with
more events it
seems</div>
<div><br>
</div>
<div>I am
seeing this
behavior in
code from
trunk on June
11 using
Fernet tokens,
but the token
backend does
not seem to
make a
difference.</div>
<div><br>
</div>
<div>Here's
what happens
to the db in
terms of
deadlock:</div>
<div>2015-07-15
21:25:41.082
31800 TRACE
keystone.common.wsgi
DBDeadlock:
(OperationalError)
(1213,
'Deadlock
found when
trying to get
lock; try
restarting
transaction')
'DELETE FROM
revocation_event
WHERE
revocation_event.revoked_at
< %s'
(datetime.datetime(2015,
7, 15, 18, 55,
41, 55186),)<br>
</div>
<div><br>
</div>
<div>When this
starts
happening, I
just go
truncate the
table, but
this is not
ideal. If [1]
is really true
then the
design is not
great, it
sounds like
keystone is
doing a
revocation
event clean-up
on every token
validation
call. Reading
and
deleting/locking
from my db
cluster is not
something I
want to do on
every validate
call.</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>Unfortunately,
that's
*exactly* what
keystone is
doing. Adam
and I had a
conversation
about this
problem in
Vancouver
which directly
resulted in
opening the
bug referenced
on the
operator list:<br>
<br>
<a
moz-do-not-send="true"
href="https://bugs.launchpad.net/keystone/+bug/1456797" target="_blank">https://bugs.launchpad.net/keystone/+bug/1456797</a><br>
<br>
</div>
<div>Neither
of us
remembered the
actual
implemented
behavior,
which is what
you've run
into and
Deepti
verified in
the bug's
comments.<br>
</div>
<span>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>So, can I
turn of token
revocation for
now? I didn't
see an obvious
no-op driver.</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>Not sure
how, other
than writing
your own no-op
driver, or
perhaps an
extended
driver that
doesn't try to
clean the
table on every
read?<br>
</div>
<span>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>And in
the long-run
can this be
fixed? I'd
rather do
almost
anything else,
including
writing a
cronjob than
what happens
now.</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>If anyone
has a better
solution than
the current
one, that's
also better
than requiring
a cron job on
something like
keystone-manage
revocation_flush
I'd love to
hear it.<br>
</div>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><span>
<div dir="ltr">
<div><br>
</div>
<div>[1] - <a
moz-do-not-send="true"
href="http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html"
target="_blank">http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html</a></div>
</div>
<br>
</span>__________________________________________________________________________<br>
OpenStack
Development
Mailing List
(not for usage
questions)<br>
Unsubscribe: <a
moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a
moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
__________________________________________________________________________<br>
OpenStack
Development
Mailing List
(not for usage
questions)<br>
Unsubscribe: <a
moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a
moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
__________________________________________________________________________<br>
OpenStack Development
Mailing List (not for
usage questions)<br>
Unsubscribe: <a
moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a
moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List
(not for usage questions)<br>
Unsubscribe: <a
moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>__________________________________________________________________________</span><br>
<span>OpenStack Development Mailing List
(not for usage questions)</span><br>
<span>Unsubscribe: <a
moz-do-not-send="true"
href="mailto:OpenStack-dev-request@lists.openstack.org"
target="_blank">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe</span><br>
<span><a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><br>
</div>
</blockquote>
<br>
<fieldset></fieldset>
<br>
<pre>__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a moz-do-not-send="true" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a moz-do-not-send="true" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a moz-do-not-send="true" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a moz-do-not-send="true" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>