[openstack-dev] [all] devstack default worker changes

Sean Dague sean at dague.net
Mon Sep 28 20:02:10 UTC 2015


We used to default to using the mysqldb driver in devstack, which is a C
binding that is not eventlet aware. This means that any time a db query
is executed the entire greenlet is blocked. This behavior meant that in
devstack (and in oslo.concurrency) default number of workers for things
like API servers and conductors was = num of cores on the machine so
that everything wouldn't be deadlocked all the time when touching the
database.

The downside is we fixed concurrency with memory, which means you have a
lot of idle python eating all your memory all the time. Devstack in 4GB
isn't really an option any more with more than a small number of
services turned on.

We changed the default mysql driver to the pure python one back early in
Liberty. It shook out some bugs, but things got good quickly on that.
Just recently we decided to see if we could drop the worker count as
well in upstream devstack and gate jobs.

The new math will give you numproc / 4 workers for these services (min
of 2). This seems to be running fine. More interestingly it ends up
keeping an extra 1GB+ of memory in page cache by the end of the gate
runs, which seems to be making things a bit more stable, and under less
load. ( https://review.openstack.org/#/c/226831/ )

We've not seen any down side of this change yet, and it should make it
easier to get devstack into a smaller footprint VM. However, I wanted to
make sure people knew it was a change in case they see some new edge
failure condition that could be related to it.

So, it should mostly all be flowers and unicorns. But, keep an eye out
in case a troll shows up under a bridge somewhere.

	-Sean

-- 
Sean Dague
http://dague.net




More information about the OpenStack-dev mailing list