<div dir="ltr"><div>Hi, </div><div><br></div><div><br></div>By the way OSprofiler trace shows how this regression impacts on amount of DB queries done by Keystone (during the boot of VM): <div><a href="http://boris-42.github.io/b2.html">http://boris-42.github.io/b2.html</a><br></div><div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 20, 2016 at 3:30 PM, Morgan Fainberg <span dir="ltr"><<a href="mailto:morgan.fainberg@gmail.com" target="_blank">morgan.fainberg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>As promised here are the fixes:<br><br><a href="https://review.openstack.org/#/q/Ifc17c27744dac5ad55e84752ca6f68169c2f5a86,n,z" target="_blank">https://review.openstack.org/#/q/Ifc17c27744dac5ad55e84752ca6f68169c2f5a86,n,z</a><br><br></div>Proposed to both master and liberty.<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 20, 2016 at 12:15 PM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 01/20/2016 02:59 PM, Morgan Fainberg wrote:<br>
> So this was due to a change in keystonemiddleware. We stopped doing<br>
> in-memory caching of tokens per process, per worker by default [1].<br>
> There are a couple of reasons:<br>
><br>
> 1) in-memory caching produced unreliable validation because some<br>
> processed may have a cache, some may not<br>
> 2) in-memory caching was unbounded memory wise per worker.<br>
><br>
> I'll spin up a devstack change to enable memcache and use the memcache<br>
> caching for keystonemiddleware today. This will benefit things in a<br>
> couple ways<br>
><br>
> * All services and all service's workers will share the offload of the<br>
> validation, likely producing a real speedup even over the old in-memory<br>
> caching<br>
> * There will no longer be inconsistent validation offload/responses<br>
> based upon which worker you happen to hit for a given service.<br>
><br>
> I'll post to the ML here with the proposed change later today.<br>
><br>
> [1]<br>
> <a href="https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52" rel="noreferrer" target="_blank">https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52</a><br>
<br>
</span>This seems like a pretty substantial performance impact. Was there a<br>
reno associated with this?<br>
<br>
I think that we should still probably:<br>
<br>
* != the keystone middleware version, it's impacting the ability to land<br>
fixes in the gate<br>
* add devstack memcache code<br>
* find some way to WARN if we are running without memcache config, so<br>
people realize they are in a regressed state<br>
* add back keystone middleware at that version<br>
<br>
-Sean<br>
<span><br>
><br>
> Cheers,<br>
> --Morgan<br>
><br>
> On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a><br>
</span><span>> <mailto:<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>>> wrote:<br>
><br>
><br>
><br>
> On 19 January 2016 at 22:46, Kevin Benton <<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a><br>
</span><div><div>> <mailto:<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>>> wrote:<br>
><br>
> Hi all,<br>
><br>
> We noticed a major jump in the neutron tempest and API test run<br>
> times recently in Neutron. After digging through logstash I<br>
> found out that it first occurred on the requirements bump here:<br>
> <a href="https://review.openstack.org/#/c/265697/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/265697/</a><br>
><br>
> After locally testing each requirements change individually, I<br>
> found that the keystonemiddleware change seems to be the<br>
> culprit. It almost doubles the time it takes to fulfill simple<br>
> port-list requests in Neutron.<br>
><br>
> Armando pushed up a patch here to<br>
> confirm: <a href="https://review.openstack.org/#/c/270024/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/270024/</a><br>
> Once that's verified, we should probably put a cap on the<br>
> middleware because it's causing the tests to run up close to<br>
> their time limits.<br>
><br>
><br>
> Kevin,<br>
><br>
> As usual your analytical skills are to be praised.<br>
><br>
> I wonder if anyone else is aware of the issue/s, because during the<br>
> usual hunting I could see other projects being affected and showing<br>
> abnormally high run times of the dsvm jobs.<br>
><br>
> I am not sure that [1] is the right approach, but it should give us<br>
> some data points if executed successfully.<br>
><br>
> Cheers,<br>
> Armando<br>
><br>
> [1] <a href="https://review.openstack.org/#/c/270024/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/270024/</a><br>
><br>
><br>
> --<br>
> Kevin Benton<br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</div></div>> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<span>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</span>> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<span>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
</span><span><font color="#888888">--<br>
Sean Dague<br>
<a href="http://dague.net" rel="noreferrer" target="_blank">http://dague.net</a><br>
</font></span><div><div><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>