Open Stack

On May 21, 2013, at 3:27 PM, "Day, Phil" <philip.day at hp.com> wrote:

> Hi Folks,
>  
> I wondered if anyone else has managed to run multiple filter-schedulers concurrently under a high load ?
>  
> I’d thought that the race conditions that we had in the past (where multiple schedulers pick the same host) been eliminated through the reworking of the resource tracker / retry mechanism, but whilst it is much better I still see the odd case where a request gets rejected multiple times (and eventually fails) because on each successive host it fails to get the resources the scheduler thought were there.
>  
> I guess on reflection its implicit in any solution which relies on fail / retry approach to cover the race condition that under a large load the number of retries for any specific request is effectively unlimited, and so no value of max_retries is ever going to be quite big enough – but before I do some more head scratching about how (if) to try and make this more robust under load I’d see if others have approach this I thought I’d ask if others were also trying to rune more than one active scheduler.


Yeah, multiple schedulers are a problem (heck, even a single one is under load :).   There's a config item that may help you:

scheduler_host_subset_size --  It defaults to '1'… but if you set it higher than 1, it'll randomize the top 'x' hosts.  This can help reduce races by introducing a bit of randomization.

Also, I feel like when we have conductor managing the retries, things can get a little better.  Perhaps we can bump the retries, I dunno.  Are you finding your computes kicking the messages back to the scheduler quickly…Ie, nova-compute is detecting quickly that an instance doesn't fit?  The resource tracker is supposed to be good about that.  If that is working well, you can probably safely bump the # of retries now… and be sure to use that above conf item.

- Chris





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130521/3c12b6a3/attachment.html>