Open Stack

Excerpts from Chris Friesen's message of 2015-10-09 23:16:43 -0700:
> On 10/09/2015 07:29 PM, Clint Byrum wrote:
> 
> > Even if you figured out how to make the in-memory scheduler crazy fast,
> > There's still value in concurrency for other reasons. No matter how
> > fast you make the scheduler, you'll be slave to the response time of
> > a single scheduling request. If you take 1ms to schedule each node
> > (including just reading the request and pushing out your scheduling
> > result!) you will never achieve greater than 1000/s. 1ms is way lower
> > than it's going to take just to shove a tiny message into RabbitMQ or
> > even 0mq. So I'm pretty sure this is o-k for small clouds, but would be
> > a disaster for a large, busy cloud.
> >
> > If, however, you can have 20 schedulers that all take 10ms on average,
> > and have the occasional lock contention for a resource counter resulting
> > in 100ms, now you're at 2000/s minus the lock contention rate. This
> > strategy would scale better with the number of compute nodes, since
> > more nodes means more distinct locks, so you can scale out the number
> > of running servers separate from the number of scheduling requests.
> 
> As far as I can see, moving to an in-memory scheduler is essentially orthogonal 
> to allowing multiple schedulers to run concurrently.  We can do both.
> 

Agreed, and I want to make sure we continue to be able to run concurrent
schedulers.

Going in memory won't reduce contention for the same resources. So it
will definitely schedule faster, but it may also serialize with concurrent
schedulers sooner, and thus turn into a situation where scaling out more
nodes means the same, or even less throughput.

Keep in mind, I actually think we give our users _WAY_ too much power
over our clouds, and I actually think we should simply have flavor based
scheduling and let compute nodes grab node reservation requests directly
out of flavor based queues based on their own current observation of
their ability to service it.

But I understand that there are quite a few clouds now that have been
given shiny dynamic scheduling tools and now we have to engineer for
those.