Open Stack

Excerpts from Boris Pavlovic's message of 2015-11-03 14:20:10 -0800:
> Hi stackers,
> 
> Usually such projects like Heat, Tempest, Rally, Scalar, and other tool
> that works with OpenStack are working with resources (e.g. VM, Volumes,
> Images, ..) in the next way:
> 
> >>> resource = api.resouce_do_some_stuff()
> >>> while api.resource_get(resource["uuid"]) != expected_status
> >>>    sleep(a_bit)
> 
> For each async operation they are polling and call many times
> resource_get() which creates significant load on API and DB layers due the
> nature of this request. (Usually getting full information about resources
> produces SQL requests that contains multiple JOINs, e,g for nova vm it's 6
> joins).
> 
> What if we add new API method that will just resturn resource status by
> UUID? Or even just extend get request with the new argument that returns
> only status?

I like the idea of being able pass in the set of fields you want to
see with each get. In SQL, often times only passing in indexed fields
will allow a query to be entirely serviced by a brief range scan in
the B-tree. For instance, if you have an index on '(UUID, status)',
then this lookup will be a single read from an index in MySQL/MariaDB:

SELECT status FROM instances WHERE UUID='foo';

The explain on this will say 'Using index' and basically you'll just do
a range scan on the UUID portion, and only find one entry, which will
be lightning fast, and return only status since it already has it there
in the index. Maintaining the index is not free, but probably worth it
if your users really do poll this way a lot.

That said, this is optimizing for polling, and I'm not a huge fan. I'd
much rather see a pub/sub model added to the API, so that users can
simply subscribe to changes in resources, and poll only when a very long
timeout has passed. This will reduce load on API services, databases,
caches, etc. There was a thread some time ago about using Nova's built
in notifications to produce an Atom feed per-project. That seems like
a much more scalable model, as even polling just that super fast query
will still incur quite a bit more cost than a GET with If-Modified-Since
on a single xml file.