Open Stack

On 1/20/2016 2:15 PM, Sean Dague wrote:
> On 01/20/2016 02:59 PM, Morgan Fainberg wrote:
>> So this was due to a change in keystonemiddleware. We stopped doing
>> in-memory caching of tokens per process, per worker by default [1].
>> There are a couple of reasons:
>>
>> 1) in-memory caching produced unreliable validation because some
>> processed may have a cache, some may not
>> 2) in-memory caching was unbounded memory wise per worker.
>>
>> I'll spin up a devstack change to enable memcache and use the memcache
>> caching for keystonemiddleware today. This will benefit things in a
>> couple ways
>>
>> * All services and all service's workers will share the offload of the
>> validation, likely producing a real speedup even over the old in-memory
>> caching
>> * There will no longer be inconsistent validation offload/responses
>> based upon which worker you happen to hit for a given service.
>>
>> I'll post to the ML here with the proposed change later today.
>>
>> [1]
>> https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52
>
> This seems like a pretty substantial performance impact. Was there a
> reno associated with this?
>
> I think that we should still probably:
>
> * != the keystone middleware version, it's impacting the ability to land
> fixes in the gate
> * add devstack memcache code
> * find some way to WARN if we are running without memcache config, so
> people realize they are in a regressed state
> * add back keystone middleware at that version
>
> 	-Sean
>
>>
>> Cheers,
>> --Morgan
>>
>> On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <armamig at gmail.com
>> <mailto:armamig at gmail.com>> wrote:
>>
>>
>>
>>      On 19 January 2016 at 22:46, Kevin Benton <blak111 at gmail.com
>>      <mailto:blak111 at gmail.com>> wrote:
>>
>>          Hi all,
>>
>>          We noticed a major jump in the neutron tempest and API test run
>>          times recently in Neutron. After digging through logstash I
>>          found out that it first occurred on the requirements bump here:
>>          https://review.openstack.org/#/c/265697/
>>
>>          After locally testing each requirements change individually, I
>>          found that the keystonemiddleware change seems to be the
>>          culprit. It almost doubles the time it takes to fulfill simple
>>          port-list requests in Neutron.
>>
>>          Armando pushed up a patch here to
>>          confirm: https://review.openstack.org/#/c/270024/
>>          Once that's verified, we should probably put a cap on the
>>          middleware because it's causing the tests to run up close to
>>          their time limits.
>>
>>
>>      Kevin,
>>
>>      As usual your analytical skills are to be praised.
>>
>>      I wonder if anyone else is aware of the issue/s, because during the
>>      usual hunting I could see other projects being affected and showing
>>      abnormally high run times of the dsvm jobs.
>>
>>      I am not sure that [1] is the right approach, but it should give us
>>      some data points if executed successfully.
>>
>>      Cheers,
>>      Armando
>>
>>      [1]  https://review.openstack.org/#/c/270024/
>>
>>
>>          --
>>          Kevin Benton
>>
>>          __________________________________________________________________________
>>          OpenStack Development Mailing List (not for usage questions)
>>          Unsubscribe:
>>          OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>          <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>          http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>      __________________________________________________________________________
>>      OpenStack Development Mailing List (not for usage questions)
>>      Unsubscribe:
>>      OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>      <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>      http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>

The blacklist on 4.1.0 went into g-r:

https://review.openstack.org/#/c/270417/

But that will just be back once a new keystonemiddleware is released.

IMO this should have been reverted, blacklist 4.1.0, and then deprecated 
at least until newton with warnings and release notes saying the 
automatic caching is going away so people have time to prepare.

I'm cross-posting to the ops list as an FYI.

-- 

Thanks,

Matt Riedemann