Open Stack

On 01/20/2016 02:59 PM, Morgan Fainberg wrote:
> So this was due to a change in keystonemiddleware. We stopped doing
> in-memory caching of tokens per process, per worker by default [1].
> There are a couple of reasons:
> 
> 1) in-memory caching produced unreliable validation because some
> processed may have a cache, some may not
> 2) in-memory caching was unbounded memory wise per worker.
> 
> I'll spin up a devstack change to enable memcache and use the memcache
> caching for keystonemiddleware today. This will benefit things in a
> couple ways
> 
> * All services and all service's workers will share the offload of the
> validation, likely producing a real speedup even over the old in-memory
> caching
> * There will no longer be inconsistent validation offload/responses
> based upon which worker you happen to hit for a given service.
> 
> I'll post to the ML here with the proposed change later today.
> 
> [1]
> https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52

This seems like a pretty substantial performance impact. Was there a
reno associated with this?

I think that we should still probably:

* != the keystone middleware version, it's impacting the ability to land
fixes in the gate
* add devstack memcache code
* find some way to WARN if we are running without memcache config, so
people realize they are in a regressed state
* add back keystone middleware at that version

	-Sean

> 
> Cheers,
> --Morgan
> 
> On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <armamig at gmail.com
> <mailto:armamig at gmail.com>> wrote:
> 
> 
> 
>     On 19 January 2016 at 22:46, Kevin Benton <blak111 at gmail.com
>     <mailto:blak111 at gmail.com>> wrote:
> 
>         Hi all,
> 
>         We noticed a major jump in the neutron tempest and API test run
>         times recently in Neutron. After digging through logstash I
>         found out that it first occurred on the requirements bump here:
>         https://review.openstack.org/#/c/265697/
> 
>         After locally testing each requirements change individually, I
>         found that the keystonemiddleware change seems to be the
>         culprit. It almost doubles the time it takes to fulfill simple
>         port-list requests in Neutron.
> 
>         Armando pushed up a patch here to
>         confirm: https://review.openstack.org/#/c/270024/
>         Once that's verified, we should probably put a cap on the
>         middleware because it's causing the tests to run up close to
>         their time limits.
> 
> 
>     Kevin,
> 
>     As usual your analytical skills are to be praised.
> 
>     I wonder if anyone else is aware of the issue/s, because during the
>     usual hunting I could see other projects being affected and showing
>     abnormally high run times of the dsvm jobs.
> 
>     I am not sure that [1] is the right approach, but it should give us
>     some data points if executed successfully.
> 
>     Cheers,
>     Armando
> 
>     [1]  https://review.openstack.org/#/c/270024/
> 
> 
>         -- 
>         Kevin Benton
> 
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
http://dague.net