Open Stack

Thanks Chris,

Yep we do have the scheduler_host_subset_size set (I think that was one of our contributions), and we are getting the kick back from the compute nodes when the resource tracker kicks in, so all of that is working as it should do.   I've been a bit wary of bumping the retry count too high as we've also seen a number of errors bouncing thought host due to other errors (such as the quantum port quota issue), but I might look at bumping it up a tad.

I'm also thinking about a small mod to the  scheduler_host_subset_size code that adds a scheduler_host_subset_offest, so that you could for example have one scheduler picking from the top 10 hosts, and another picking from 11-20.    That won't guarantee there's never an overlap, but I think it would reduce it considerably.    It would also mean that if you do loose a scheduler the most hosts that are no longer scheduled to becomes scheduler_host_subset_size.

Phil

From: Chris Behrens [mailto:cbehrens at codestud.com]
Sent: 22 May 2013 04:44
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Running multiple filter schedulers in parallel


On May 21, 2013, at 3:27 PM, "Day, Phil" <philip.day at hp.com<mailto:philip.day at hp.com>> wrote:


Hi Folks,

I wondered if anyone else has managed to run multiple filter-schedulers concurrently under a high load ?

I'd thought that the race conditions that we had in the past (where multiple schedulers pick the same host) been eliminated through the reworking of the resource tracker / retry mechanism, but whilst it is much better I still see the odd case where a request gets rejected multiple times (and eventually fails) because on each successive host it fails to get the resources the scheduler thought were there.

I guess on reflection its implicit in any solution which relies on fail / retry approach to cover the race condition that under a large load the number of retries for any specific request is effectively unlimited, and so no value of max_retries is ever going to be quite big enough - but before I do some more head scratching about how (if) to try and make this more robust under load I'd see if others have approach this I thought I'd ask if others were also trying to rune more than one active scheduler.


Yeah, multiple schedulers are a problem (heck, even a single one is under load :).   There's a config item that may help you:

scheduler_host_subset_size --  It defaults to '1'... but if you set it higher than 1, it'll randomize the top 'x' hosts.  This can help reduce races by introducing a bit of randomization.

Also, I feel like when we have conductor managing the retries, things can get a little better.  Perhaps we can bump the retries, I dunno.  Are you finding your computes kicking the messages back to the scheduler quickly...Ie, nova-compute is detecting quickly that an instance doesn't fit?  The resource tracker is supposed to be good about that.  If that is working well, you can probably safely bump the # of retries now... and be sure to use that above conf item.

- Chris





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130522/22747a79/attachment.html>