<div dir="ltr"><div>2Everybody,</div><div><br></div><div>Just curios why we need such complexity. </div><div><br></div><div><br></div><div>Let's take a look from other side: </div><div>1) Information about all hosts (even in case of 100k hosts) will be less then 1 GB</div><div>2) Usually servers that runs scheduler service have at least 64GB RAM and more on the board</div><div>3) math.log(100000) < 12 (binary search per rule)</div>
<div>4) We have less then 20 rules for scheduling </div><div>5) Information about hosts is updated every 60 seconds (no updates host is dead) </div><div><br></div><div><br></div><div>According to this information: </div><div>1) We can store everything in RAM of single server</div><div>2) We can use Python </div><div>3) Information about hosts is temporary data and shouldn't be stored in persistence storage</div><div><br></div><div><br></div><div>Simplest architecture to cover this: </div><div>1) Single RPC service that has two methods: find_host(rules), update_host(host, data)</div><div>2) Store information about hosts like a dict (host_name->data)</div><div>3) Create for each rule binary tree and update it on each host update</div><div>4) Make a algorithm that will use binary trees to find host based on rules </div><div>5) Each service like compute node, volume node, or neutron will send updates about host</div><div> that they managed (cross service scheduling)</div><div>6) Make a algorithm that will sync host stats in memory between different schedulers </div><div>7) ...</div><div>8) PROFIT! </div><div><br></div><div>It's: </div><div>1) Simple to manage</div><div>2) Simple to understand</div><div>3) Simple to calc scalability limits</div><div>4) Simple to integrate in current OpenStack architecture<br></div><div><br></div><div><br></div><div>As a future bonus, we can implement scheduler-per-az functionality, so each scheduler will store information </div><div>only about his AZ, and separated AZ can have own rabbit servers for example which will allows us to get </div><div>horizontal scalability in terms of AZ. </div><div><br></div><div><br></div><div>So do we really need Cassandra, Mongo, ... and other Web-scale solution for such simple task? </div><div><br></div><div><br></div><div><div><div>Best regards,</div><div>Boris Pavlovic</div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Oct 10, 2015 at 11:19 PM, Clint Byrum <span dir="ltr"><<a href="mailto:clint@fewbar.com" target="_blank">clint@fewbar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Excerpts from Chris Friesen's message of 2015-10-09 23:16:43 -0700:<br>
<span class="">> On 10/09/2015 07:29 PM, Clint Byrum wrote:<br>
><br>
> > Even if you figured out how to make the in-memory scheduler crazy fast,<br>
> > There's still value in concurrency for other reasons. No matter how<br>
> > fast you make the scheduler, you'll be slave to the response time of<br>
> > a single scheduling request. If you take 1ms to schedule each node<br>
> > (including just reading the request and pushing out your scheduling<br>
> > result!) you will never achieve greater than 1000/s. 1ms is way lower<br>
> > than it's going to take just to shove a tiny message into RabbitMQ or<br>
> > even 0mq. So I'm pretty sure this is o-k for small clouds, but would be<br>
> > a disaster for a large, busy cloud.<br>
> ><br>
> > If, however, you can have 20 schedulers that all take 10ms on average,<br>
> > and have the occasional lock contention for a resource counter resulting<br>
> > in 100ms, now you're at 2000/s minus the lock contention rate. This<br>
> > strategy would scale better with the number of compute nodes, since<br>
> > more nodes means more distinct locks, so you can scale out the number<br>
> > of running servers separate from the number of scheduling requests.<br>
><br>
> As far as I can see, moving to an in-memory scheduler is essentially orthogonal<br>
> to allowing multiple schedulers to run concurrently. We can do both.<br>
><br>
<br>
</span>Agreed, and I want to make sure we continue to be able to run concurrent<br>
schedulers.<br>
<br>
Going in memory won't reduce contention for the same resources. So it<br>
will definitely schedule faster, but it may also serialize with concurrent<br>
schedulers sooner, and thus turn into a situation where scaling out more<br>
nodes means the same, or even less throughput.<br>
<br>
Keep in mind, I actually think we give our users _WAY_ too much power<br>
over our clouds, and I actually think we should simply have flavor based<br>
scheduling and let compute nodes grab node reservation requests directly<br>
out of flavor based queues based on their own current observation of<br>
their ability to service it.<br>
<br>
But I understand that there are quite a few clouds now that have been<br>
given shiny dynamic scheduling tools and now we have to engineer for<br>
those.<br>
<div class="HOEnZb"><div class="h5"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>