[openstack-dev] [infra] [gate] [all] openstack services footprint lead to oom-kill in the gate

Kevin Benton kevin at benton.pub
Thu Feb 2 23:09:10 UTC 2017


I'm referring to Apache sitting in between the services now as a TLS
terminator and connection proxy. That was not the configuration before but
it is now the default devstack behavior.

See this example from Newton:
http://logs.openstack.org/73/428073/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/028ea38/logs/apache_config/
Then this from master:
http://logs.openstack.org/32/421832/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5af5c7c/logs/apache_config/


>The ways in which OpenStack and oslo.service uses eventlet are known to
>have scaling bottle necks. The Keystone team saw substantial throughput
>gains going over to apache hosting.


Right, but there is a difference between scaling issues and a single worker
not being able to handle the peak 5 concurrent requests or so that the gate
jobs experience. The eventlet wsgi server should have no issues with our
gate load.

On Thu, Feb 2, 2017 at 3:01 PM, Sean Dague <sean at dague.net> wrote:

> On 02/02/2017 04:07 PM, Kevin Benton wrote:
> > This error seems to be new in the ocata cycle. It's either related to a
> > dependency change or the fact that we put Apache in between the services
> > now. Handling more concurrent requests than workers wasn't an issue
> > before.
> >
> > It seems that you are suggesting that eventlet can't handle concurrent
> > connections, which is the entire purpose of the library, no?
>
> The only services that are running on Apache in standard gate jobs are
> keystone and the placement api. Everything else is still the
> oslo.service stack (which is basically run eventlet as a preforking
> static worker count webserver).
>
> The ways in which OpenStack and oslo.service uses eventlet are known to
> have scaling bottle necks. The Keystone team saw substantial throughput
> gains going over to apache hosting.
>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170202/14de367c/attachment.html>


More information about the OpenStack-dev mailing list