Open Stack

On 9/28/2015 3:02 PM, Sean Dague wrote:
> We used to default to using the mysqldb driver in devstack, which is a C
> binding that is not eventlet aware. This means that any time a db query
> is executed the entire greenlet is blocked. This behavior meant that in
> devstack (and in oslo.concurrency) default number of workers for things
> like API servers and conductors was = num of cores on the machine so
> that everything wouldn't be deadlocked all the time when touching the
> database.
>
> The downside is we fixed concurrency with memory, which means you have a
> lot of idle python eating all your memory all the time. Devstack in 4GB
> isn't really an option any more with more than a small number of
> services turned on.
>
> We changed the default mysql driver to the pure python one back early in
> Liberty. It shook out some bugs, but things got good quickly on that.
> Just recently we decided to see if we could drop the worker count as
> well in upstream devstack and gate jobs.
>
> The new math will give you numproc / 4 workers for these services (min
> of 2). This seems to be running fine. More interestingly it ends up
> keeping an extra 1GB+ of memory in page cache by the end of the gate
> runs, which seems to be making things a bit more stable, and under less
> load. ( https://review.openstack.org/#/c/226831/ )
>
> We've not seen any down side of this change yet, and it should make it
> easier to get devstack into a smaller footprint VM. However, I wanted to
> make sure people knew it was a change in case they see some new edge
> failure condition that could be related to it.
>
> So, it should mostly all be flowers and unicorns. But, keep an eye out
> in case a troll shows up under a bridge somewhere.
>
> 	-Sean
>

The large ops job appears to be throwing up on itself a bit because of 
this change.  Tracking with bug 
https://bugs.launchpad.net/nova/+bug/1500615.

-- 

Thanks,

Matt Riedemann