[openstack-dev] [infra] [gate] [all] openstack services footprint lead to oom-kill in the gate
clint at fewbar.com
Sun Feb 5 19:19:31 UTC 2017
Excerpts from Matt Riedemann's message of 2017-02-04 16:09:56 -0600:
> On 2/2/2017 4:01 PM, Sean Dague wrote:
> > The only services that are running on Apache in standard gate jobs are
> > keystone and the placement api. Everything else is still the
> > oslo.service stack (which is basically run eventlet as a preforking
> > static worker count webserver).
> > The ways in which OpenStack and oslo.service uses eventlet are known to
> > have scaling bottle necks. The Keystone team saw substantial throughput
> > gains going over to apache hosting.
> > -Sean
> FWIW, coincidentally the nova team is going to work on running nova-api
> under apache in some select jobs in Pike because it turns out that
> TripleO was running that configuration in Newton which is considered
> experimental in nova (we don't do some things when running in that mode
> which are actually pretty critical to how the code functions for
> upgrades). So if Apache/eventlet is related, maybe we'll see some
> differences after making that change.
> But I also wouldn't be surprised if Nova is creating more versioned
> objects which reference other full versioned objects (rather than just
> an id reference) and maybe some of those are hanging around longer than
> they should be.
Has there ever been an effort to profile memory usage of oslo versioned
objects, including the actual objects defined in each major project? I
would be willing to wager there are enough circular references that
they're pretty sticky.
Also I wonder if there's ever been any serious consideration given to
switching to protobuf? Feels like one could make oslo.versionedobjects
a wrapper around protobuf relatively easily, but perhaps that's already
been explored in a forum that I wasn't paying attention to. I feel like
that's another example of something I hear in the hallways that doesn't
get any forward progress.
More information about the OpenStack-dev