Open Stack

Ed Leafe wrote:
> Wow, I seem to have unleashed a bunch of pent-up frustration in the
> community! It's great to see everyone coming forward with their ideas
> and insights for improving the way Nova (and, by extension, all of
> OpenStack) can potentially scale.
>
> I do have a few comments on the discussion:
>
> 1) This isn't a proposal to simply add some sort of DLM to Nova as a
> magic cure-all. The concerns about Nova's ability to scale have to do
> a lot more with the overall internal communication design.
>
> 2) I really liked the comment about "made-up numbers". It's so true:
> we are all impressed by such examples of speed that we sometimes
> forget whether speeding up X will improve the overall process to any
> significant degree. The purpose of my original email back in July,
> and the question I asked at the Nova midcycle, is if we could get
> some numbers that would be a target to shoot for with any of these
> experiments. Sure, I could come up with a test that shows a zillion
> transactions per second, but if that doesn't result in a cloud being
> able to schedule more efficiently, what's the point?
>
> 3) I like the idea of something like ZooKeeper, but my concern is how
> to efficiently query the data. If, for example, we had records for
> 100K compute nodes, would it be possible to do the equivalent of
> "SELECT * FROM resources WHERE resource_type = 'compute' AND
> free_ram_mb>= 2048 AND …" - well, you get the idea. Are complex data
> queries possible in ZK? I haven't been able to find that information
> anywhere.

The idea is that u wouldn't do these queries against any remote source 
in the first place. Instead a scheduler would get notified (via a 
concept like 
http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#sc_zkDataMode_watches) 
when a hypervisor updates its data in zookeeper (or other equivalent 
system); when that notification happens the scheduler then reads the 
data then updates some *local* data source with that information (this 
could be a in-memory dict, or a local sqlite, or something else better 
optimized for searching fast) and then from that point on that local 
source is used to do queries on. This way a hypervisor (compute-node) is 
performing *nearly* the equivalent of a push notification (like on your 
phone) to schedulers.

>
> 4) It is true that even in a very large deployment, it is possible to
> keep all the relevant data needed for scheduling in memory. My
> concern is how to efficiently search that data, much like in the ZK
> scenario.

See above.

>
> 5) Concerns about Cassandra running with OpenJDK instead of the
> Oracle JVM are troubling. I sent an email about this to one of the
> people I know at DataStax, but so far have not received a response.
> And while it would be great to have people contribute to OpenJDK to
> make it compatible, keep in mind that that would be an ongoing
> commitment, not just a one-time effort.
>
> 6) I remember discussions back in the Austin-Bexar time frame about
> what Thierry referred to as 'flavor-based schedulers', and they were
> immediately discounted as not sophisticated enough to handle the sort
> of complex scheduling requests that were expected. I'd be interested
> in finding out from the big cloud providers what percentage of their
> requests would fall into this simple structure, and what percent are
> more complicated than that. Having hosts listening to queues that
> they know they can satisfy removes the raciness from the process,
> although it would require some additional handling for the situation
> where no host accepts the request. Still, it has the advantage of
> being dead simple. Unfortunately, this would probably require a
> bigger architectural change than integrating Cassandra into the
> Scheduler would.

Another discussion that also should get talked about, but is again much 
larger in scope: https://review.openstack.org/#/c/210549/ (still WIP but 
the idea/problem/issue hopefully is clear).

>
> I hope that those of us who will be at the Tokyo Summit and are
> interested in these ideas can get together for an informal
> discussion, and come up with some ideas for grand experiments and
> reality checks. ;-)
>
> BTW, I started playing around with some ideas, and thought that if
> anyone wanted to also try Cassandra, I'd write up a quick how-to for
> setting up a small cluster:
> http://blog.leafe.com/small-scale-cassandra/. Using docker images
> makes it a breeze!
>
>
> -- Ed Leafe
>
>
>
>
>
>
> __________________________________________________________________________
>
>
OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev