[openstack-dev] [infra] [gate] [all] openstack services footprint lead to oom-kill in the gate

Clint Byrum clint at fewbar.com
Sun Feb 5 19:19:31 UTC 2017


Excerpts from Matt Riedemann's message of 2017-02-04 16:09:56 -0600:
> On 2/2/2017 4:01 PM, Sean Dague wrote:
> >
> > The only services that are running on Apache in standard gate jobs are
> > keystone and the placement api. Everything else is still the
> > oslo.service stack (which is basically run eventlet as a preforking
> > static worker count webserver).
> >
> > The ways in which OpenStack and oslo.service uses eventlet are known to
> > have scaling bottle necks. The Keystone team saw substantial throughput
> > gains going over to apache hosting.
> >
> >     -Sean
> >
> 
> FWIW, coincidentally the nova team is going to work on running nova-api 
> under apache in some select jobs in Pike because it turns out that 
> TripleO was running that configuration in Newton which is considered 
> experimental in nova (we don't do some things when running in that mode 
> which are actually pretty critical to how the code functions for 
> upgrades). So if Apache/eventlet is related, maybe we'll see some 
> differences after making that change.
> 
> But I also wouldn't be surprised if Nova is creating more versioned 
> objects which reference other full versioned objects (rather than just 
> an id reference) and maybe some of those are hanging around longer than 
> they should be.
> 

Has there ever been an effort to profile memory usage of oslo versioned
objects, including the actual objects defined in each major project? I
would be willing to wager there are enough circular references that
they're pretty sticky.

Also I wonder if there's ever been any serious consideration given to
switching to protobuf? Feels like one could make oslo.versionedobjects
a wrapper around protobuf relatively easily, but perhaps that's already
been explored in a forum that I wasn't paying attention to. I feel like
that's another example of something I hear in the hallways that doesn't
get any forward progress.



More information about the OpenStack-dev mailing list