<div dir="ltr">Wow, I missed this thread completely, sorry. I just went over the meeting notes and I'd like to add what I can from our own experience with the scheduler at Bluehost.<div><br></div><div style>The first issue we had was dealing with the fanout_cast to schedulers from the compute nodes. With a large number of nodes all of the scheduler's processing time is just getting these updates and processing them. I wasn't the one who dug into this and tore it out, but I think we determined that for us it was sufficient to get the information from the DB and rely on that. In any case, we need to have one way to report instead of reporting to the DB and to the individual schedulers as was discussed in the meeting. Personally, I think the fanout_cast needs to go away. If updating capabilities using RPC is desired that's fine, but it shouldn't be a broadcast type communication. It would be better to have the schedulers share a host state and one of them at a time can get an update and apply it to the shared store. That way we can just spin up more schedulers when your current set are not keeping up.</div>
<div style><br></div><div style>Second issue is something Phil brought up which is the filtering stuff. This, to me, was the larger issue and why we stuck our own scheduler in instead of trying to fix the problem. There are a few filters that you don't need to spin through the whole list to apply. For example filters that select or exclude specific hosts should be applied to a collection instead of each item of the collection. Btw, I'm geekinutah on IRC, feel free to msg me about Bluehost stuff anytime.</div>
<div style><br></div><div style>-Mike Wilson</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 12, 2013 at 11:31 AM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="im">On Mon, Jun 10, 2013 at 3:11 PM, Dugger, Donald D <span dir="ltr"><<a href="mailto:donald.d.dugger@intel.com" target="_blank">donald.d.dugger@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Current list of topics we're going over is:<br>
<br>
1) Extending data in host state<br>
2) Utilization based scheduling<br>
3) Whole host allocation capability<br>
4) Coexistence of different schedulers<br>
5) Rack aware scheduling<br>
6) List scheduler hints via API<br>
7) Host directory service<br>
8) The future of the scheduler<br>
9) Network bandwisth aware scheduling (and wider aspects)<br>
10) ensembles/vclusters<br>
<br>
We've done a first pass over all of these so next will be follow ups to see where we are. But first, a new issue was raised at the last meeting:<br>
<br>
11) Scheduler scalability<br>
<br>
The assertion was that BlueHost has created an OpenStack cluster with ~16,000 nodes and the scheduler didn't scale, they had to throw it out completely and just put in a simple random selection scheduler. Obviously scalability of the scheduler is a concern so I'd like to spend this meeting discussing this topic. (If someone from BlueHost could attend that would be great).<br>
</blockquote><div><br></div><div><br></div></div><div>This is what I am basing my information on (<a href="http://www.openstack.org/summit/portland-2013/session-videos/presentation/using-openstack-in-a-traditional-hosting-environment" target="_blank">http://www.openstack.org/summit/portland-2013/session-videos/presentation/using-openstack-in-a-traditional-hosting-environment</a> starting at 9:45). Compute nodes broadcast updates to the schedulers every minute which for 16k nodes is 266 messages a second (on average). And with the scheduler being single threaded, processing these messages will keep the scheduler(s) very busy just processing compute broadcasts. </div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
--<br>
Don Dugger<br>
"Censeo Toto nos in Kansa esse decisse." - D. Gale<br>
Ph: <a href="tel:303%2F443-3786" value="+13034433786" target="_blank">303/443-3786</a><br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>