Open Stack

On 2 February 2017 at 10:08, Sean Dague <sean at dague.net> wrote:

> On 02/02/2017 12:49 PM, Armando M. wrote:
> >
> >
> > On 2 February 2017 at 08:40, Sean Dague <sean at dague.net
> > <mailto:sean at dague.net>> wrote:
> >
> >     On 02/02/2017 11:16 AM, Matthew Treinish wrote:
> >     <snip>
> >     > <oops, forgot to finish my though>
> >     >
> >     > We definitely aren't saying running a single worker is how we
> recommend people
> >     > run OpenStack by doing this. But it just adds on to the
> differences between the
> >     > gate and what we expect things actually look like.
> >
> >     I'm all for actually getting to the bottom of this, but honestly real
> >     memory profiling is needed here. The growth across projects probably
> >     means that some common libraries are some part of this. The ever
> growing
> >     requirements list is demonstrative of that. Code reuse is good, but
> if
> >     we are importing much of a library to get access to a couple of
> >     functions, we're going to take a bunch of memory weight on that
> >     (especially if that library has friendly auto imports in top level
> >     __init__.py so we can't get only the parts we want).
> >
> >     Changing the worker count is just shuffling around deck chairs.
> >
> >     I'm not familiar enough with memory profiling tools in python to know
> >     the right approach we should take there to get this down to
> individual
> >     libraries / objects that are containing all our memory. Anyone more
> >     skilled here able to help lead the way?
> >
> >
> > From what I hear, the overall consensus on this matter is to determine
> > what actually caused the memory consumption bump and how to address it,
> > but that's more of a medium to long term action. In fact, to me this is
> > one of the top priority matters we should talk about at the imminent PTG.
> >
> > For the time being, and to provide relief to the gate, should we want to
> > lock the API_WORKERS to 1? I'll post something for review and see how
> > many people shoot it down :)
>
> I don't think we want to do that. It's going to force down the eventlet
> API workers to being a single process, and it's not super clear that
> eventlet handles backups on the inbound socket well. I honestly would
> expect that creates different hard to debug issues, especially with high
> chatter rates between services.
>

I must admit I share your fear, but out of the tests that I have executed
so far in [1,2,3], the house didn't burn in a fire. I am looking for other
ways to have a substantial memory saving with a relatively quick and dirty
fix, but coming up empty handed thus far.

[1] https://review.openstack.org/#/c/428303/
[2] https://review.openstack.org/#/c/427919/
[3] https://review.openstack.org/#/c/427921/


>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170202/20ec79ef/attachment.html>