<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 16, 2017 at 5:20 PM, Dolph Mathews <span dir="ltr"><<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you for the data and your test scripts! As Lance and Stanek already alluded, Fernet performance is very sensitive to keystone's configuration. Can your share your keystone.conf as well?<div><br></div><div>I'll also be in Atlanta and would love to talk Fernet performance, even if we don't have a formal time slot on the schedule.</div></div></blockquote><div><br></div><div>++ Our schedule is coming together and this is what we have so far [0]. If there is an open time slot that works for your schedule, don't hesitate to let me know.</div><div><br></div><div><br></div><div>[0] <a href="https://etherpad.openstack.org/p/keystone-pike-ptg">https://etherpad.openstack.org/p/keystone-pike-ptg</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-HOEnZb"><div class="gmail-h5"><br><div class="gmail_quote"><div dir="ltr">On Wed, Feb 15, 2017 at 9:08 AM Lance Bragstad <<a href="mailto:lbragstad@gmail.com" target="_blank">lbragstad@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail-m_2903814908616500941gmail_msg">In addition to what David said, have you played around with caching in keystone [0]? After the initial implementation of fernet landed, we attempted to make it the default token provider. We ended up reverting the default back to uuid because we hit several issues. Around the Liberty and Mitaka timeframe, we reworked the caching implementation to fix those issues and improve overall performance of all token formats, especially fernet.<div class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg"></div><div class="gmail-m_2903814908616500941gmail_msg">We have a few different performance perspectives available, too. Some were run nearly 2 years ago [1] and some are run today [2]. Since the Newton release, we've made drastic improvements to the overall structure of the token provider [3] [4] [5]. At the very least, it should make understanding keystone's approach to tokens easier. Maintaining out-of-tree token providers should also be easier since we cleaned up a lot of the interfaces that affect developers maintaining their own providers.</div><div class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg"></div><div class="gmail-m_2903814908616500941gmail_msg">We can try and set something up at the PTG. We are getting pretty tight for time slots, but I'm sure we can find some time to work through the issues you're seeing (also, feel free to hop into #openstack-keystone on freenode if you want to visit prior to the PTG).<div class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg"></div><div class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg"></div><div class="gmail-m_2903814908616500941gmail_msg">[0] <a href="https://docs.openstack.org/developer/keystone/configuration.html#caching-layer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://docs.openstack.<wbr>org/developer/keystone/<wbr>configuration.html#caching-<wbr>layer</a></div></div><div class="gmail-m_2903814908616500941gmail_msg">[1] <a href="http://dolphm.com/benchmarking-openstack-keystone-token-formats/" class="gmail-m_2903814908616500941gmail_msg" target="_blank">http://dolphm.com/<wbr>benchmarking-openstack-<wbr>keystone-token-formats/</a></div><div class="gmail-m_2903814908616500941gmail_msg">[2] <a href="https://github.com/lbragstad/keystone-performance" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://github.com/<wbr>lbragstad/keystone-performance</a></div><div class="gmail-m_2903814908616500941gmail_msg">[3] <a href="https://review.openstack.org/#/q/status:merged+project:openstack/keystone+branch:master+topic:make-fernet-default" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://review.openstack.<wbr>org/#/q/status:merged+project:<wbr>openstack/keystone+branch:<wbr>master+topic:make-fernet-<wbr>default</a></div><div class="gmail-m_2903814908616500941gmail_msg">[4] <a href="https://review.openstack.org/#/q/status:merged+project:openstack/keystone+branch:master+topic:cleanup-token-provider" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://review.openstack.<wbr>org/#/q/status:merged+project:<wbr>openstack/keystone+branch:<wbr>master+topic:cleanup-token-<wbr>provider</a></div><div class="gmail-m_2903814908616500941gmail_msg">[5] <a href="http://specs.openstack.org/openstack/keystone-specs/specs/keystone/ocata/token-provider-cleanup.html" class="gmail-m_2903814908616500941gmail_msg" target="_blank">http://specs.openstack.<wbr>org/openstack/keystone-specs/<wbr>specs/keystone/ocata/token-<wbr>provider-cleanup.html</a></div></div><div class="gmail_extra gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg"><div class="gmail_quote gmail-m_2903814908616500941gmail_msg">On Wed, Feb 15, 2017 at 8:44 AM, David Stanek <span dir="ltr" class="gmail-m_2903814908616500941gmail_msg"><<a href="mailto:dstanek@dstanek.com" class="gmail-m_2903814908616500941gmail_msg" target="_blank">dstanek@dstanek.com</a>></span> wrote:<br class="gmail-m_2903814908616500941gmail_msg"><blockquote class="gmail_quote gmail-m_2903814908616500941gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_2903814908616500941gmail_msg">On 15-Feb 18:16, 王玺源 wrote:<br class="gmail-m_2903814908616500941gmail_msg">
> Hello everyone,<br class="gmail-m_2903814908616500941gmail_msg">
>   PKI/PKIZ token has been removed from keystone in Ocata. But recently our<br class="gmail-m_2903814908616500941gmail_msg">
> production team did some test about PKI and Fernet token (With Keystone<br class="gmail-m_2903814908616500941gmail_msg">
> Mitaka). They found that in large-scale production environment, Fernet<br class="gmail-m_2903814908616500941gmail_msg">
> token's performance is not as good as PKI. Here is the test data:<br class="gmail-m_2903814908616500941gmail_msg">
><br class="gmail-m_2903814908616500941gmail_msg">
> <a href="https://docs.google.com/document/d/12cL9bq9EARjZw9IS3YxVmYsGfdauM25NzZcdzPE0fvY/edit?usp=sharing" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://docs.google.com/<wbr>document/d/<wbr>12cL9bq9EARjZw9IS3YxVmYsGfdauM<wbr>25NzZcdzPE0fvY/edit?usp=<wbr>sharing</a><br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
</span>This is nice to see. Thanks.<br class="gmail-m_2903814908616500941gmail_msg">
<span class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
><br class="gmail-m_2903814908616500941gmail_msg">
> From the data, we can see that:<br class="gmail-m_2903814908616500941gmail_msg">
> 1. In large-scale concurrency test, PKI is much faster than Fernet.<br class="gmail-m_2903814908616500941gmail_msg">
> 2. PKI token revoke can't immediately make the token invalid. So it has the<br class="gmail-m_2903814908616500941gmail_msg">
> revoke issue.  <a href="https://wiki.openstack.org/wiki/OSSN/OSSN-0062" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://wiki.openstack.org/<wbr>wiki/OSSN/OSSN-0062</a><br class="gmail-m_2903814908616500941gmail_msg">
><br class="gmail-m_2903814908616500941gmail_msg">
> But in our production team's opinion, the revoke issue is a small problem,<br class="gmail-m_2903814908616500941gmail_msg">
> and can be avoided by some periphery ways. (More detail solution could be<br class="gmail-m_2903814908616500941gmail_msg">
> explained by them in the follow email).<br class="gmail-m_2903814908616500941gmail_msg">
> They think that the performance issue is the most important thing. Maybe<br class="gmail-m_2903814908616500941gmail_msg">
> you can see that in some production environment, performance is the first<br class="gmail-m_2903814908616500941gmail_msg">
> thing to be considered.<br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
</span>I'd like to hear solutions to this if you have already come up with<br class="gmail-m_2903814908616500941gmail_msg">
them. This issue, however, isn't the only one that led us to remove PKI<br class="gmail-m_2903814908616500941gmail_msg">
tokens.<br class="gmail-m_2903814908616500941gmail_msg">
<span class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg">
><br class="gmail-m_2903814908616500941gmail_msg">
> So here I'd like to ask you, especially the keystone experts:<br class="gmail-m_2903814908616500941gmail_msg">
> 1. Is there any chance to bring PKI/PKIZ back to Keystone?<br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
</span>I would guess that, at least in the immediate future, we would not want<br class="gmail-m_2903814908616500941gmail_msg">
to put it back into keystone until someone can fix the issues. Also<br class="gmail-m_2903814908616500941gmail_msg">
ideally running the token provider in production.<br class="gmail-m_2903814908616500941gmail_msg">
<span class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
> 2. Has Fernet token improved the performance during these releases? Or any<br class="gmail-m_2903814908616500941gmail_msg">
> road map so that we can make sure Fernet is better than PKI in all side.<br class="gmail-m_2903814908616500941gmail_msg">
> Otherwise, I don't think that remove PKI in Ocata is the right way. Or<br class="gmail-m_2903814908616500941gmail_msg">
> even, we can keep the PKI token in Keystone for more one or two cycles,<br class="gmail-m_2903814908616500941gmail_msg">
> then remove it once Fernet is stable enough.<br class="gmail-m_2903814908616500941gmail_msg">
> 3. Since I'll be in Atalanta next week, if it is possible, I'd like to<br class="gmail-m_2903814908616500941gmail_msg">
> bring this topic to Keystone PTG. can I?<br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
</span>Sure. We have a pretty packed calendar, but I'm sure you could steal a<br class="gmail-m_2903814908616500941gmail_msg">
few minutes somewhere.<br class="gmail-m_2903814908616500941gmail_msg">
<span class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
><br class="gmail-m_2903814908616500941gmail_msg">
> It is a real production problem and I really need your feedback.<br class="gmail-m_2903814908616500941gmail_msg">
><br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
</span>Have you tried playing with the crypt_strength[1]? If the slowness is<br class="gmail-m_2903814908616500941gmail_msg">
the crypto (which it was in the past) then you can tune it a little bit.<br class="gmail-m_2903814908616500941gmail_msg">
Another option might be to keep the same token flow and find a faster<br class="gmail-m_2903814908616500941gmail_msg">
method for hashing a token.<br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
1. <a href="http://git.openstack.org/cgit/openstack/keystone/tree/etc/keystone.conf.sample#n67" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">http://git.openstack.org/cgit/<wbr>openstack/keystone/tree/etc/<wbr>keystone.conf.sample#n67</a><br class="gmail-m_2903814908616500941gmail_msg">
<span class="gmail-m_2903814908616500941m_1400072414767181363HOEnZb gmail-m_2903814908616500941gmail_msg"><font color="#888888" class="gmail-m_2903814908616500941gmail_msg"><br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
--<br class="gmail-m_2903814908616500941gmail_msg">
david stanek<br class="gmail-m_2903814908616500941gmail_msg">
web: <a href="https://dstanek.com" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://dstanek.com</a><br class="gmail-m_2903814908616500941gmail_msg">
twitter: <a href="https://twitter.com/dstanek" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">https://twitter.com/dstanek</a><br class="gmail-m_2903814908616500941gmail_msg">
<br class="gmail-m_2903814908616500941gmail_msg">
______________________________<wbr>______________________________<wbr>______________<br class="gmail-m_2903814908616500941gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail-m_2903814908616500941gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="gmail-m_2903814908616500941gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="gmail-m_2903814908616500941gmail_msg">
</font></span></blockquote></div><br class="gmail-m_2903814908616500941gmail_msg"></div>
______________________________<wbr>______________________________<wbr>______________<br class="gmail-m_2903814908616500941gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail-m_2903814908616500941gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="gmail-m_2903814908616500941gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail-m_2903814908616500941gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="gmail-m_2903814908616500941gmail_msg">
</blockquote></div></div></div><span class="gmail-HOEnZb"><font color="#888888"><div dir="ltr">-- <br></div><div><div dir="ltr">-Dolph</div></div>
</font></span><br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div></div>