<div dir="ltr"><div><div><div>So this was due to a change in keystonemiddleware. We stopped doing in-memory caching of tokens per process, per worker by default [1]. There are a couple of reasons:<br><br></div>1) in-memory caching produced unreliable validation because some processed may have a cache, some may not<br></div>2) in-memory caching was unbounded memory wise per worker.<br><br></div><div>I'll spin up a devstack change to enable memcache and use the memcache caching for keystonemiddleware today. This will benefit things in a couple ways<br><br></div><div>* All services and all service's workers will share the offload of the validation, likely producing a real speedup even over the old in-memory caching<br></div><div>* There will no longer be inconsistent validation offload/responses based upon which worker you happen to hit for a given service.<br><br></div><div>I'll post to the ML here with the proposed change later today.<br><br>[1] <a href="https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52">https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52</a><br><br></div><div>Cheers,<br></div><div>--Morgan<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <span dir="ltr"><<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On 19 January 2016 at 22:46, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>We noticed a major jump in the neutron tempest and API test run times recently in Neutron. After digging through logstash I found out that it first occurred on the requirements bump here: <a href="https://review.openstack.org/#/c/265697/" target="_blank">https://review.openstack.org/#/c/265697/</a><br clear="all"><div><br></div><div>After locally testing each requirements change individually, I found that the keystonemiddleware change seems to be the culprit. It almost doubles the time it takes to fulfill simple port-list requests in Neutron.</div><div><br></div><div>Armando pushed up a patch here to confirm: <a href="https://review.openstack.org/#/c/270024/" target="_blank">https://review.openstack.org/#/c/270024/</a></div><div>Once that's verified, we should probably put a cap on the middleware because it's causing the tests to run up close to their time limits.</div></div></div></blockquote><div><br></div></span><div>Kevin,</div><div><br></div><div>As usual your analytical skills are to be praised.</div><div><br></div><div>I wonder if anyone else is aware of the issue/s, because during the usual hunting I could see other projects being affected and showing abnormally high run times of the dsvm jobs.</div><div><br></div><div>I am not sure that [1] is the right approach, but it should give us some data points if executed successfully.</div><div><br></div><div>Cheers,</div><div>Armando</div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/270024/" target="_blank">https://review.openstack.org/#/c/270024/</a></div><span class="HOEnZb"><font color="#888888"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><span><font color="#888888"><div><br></div>-- <br><div><div>Kevin Benton</div></div>
</font></span></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></font></span></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>