Open Stack

On 08/11/2015 09:42 AM, Brian Rosmaita wrote:
> On 8/7/15, 1:07 PM, "Jay Pipes" <jaypipes at gmail.com> wrote:
>
>> So, here's the crux of the issue. Nova and Cinder **do not want to speak
>> the Glance REST API** to either upload or download image bits from
>> storage. Streaming image bits through the Glance API endpoint is a
>> needless and inefficient step, and Nova and Cinder would like to
>> communicate directly with the backend storage systems.
>
> Exactly why do you want to communicate directly with the backend storage
> systems?  Streaming image bits through Glance appears to be "needless and
> inefficient", but if an end-user is booting 1K instances from some custom
> image, Glance's image cache makes an enormous difference in delivery time.

Nova's image cache makes a bigger difference overall in those cases, 
since the image will most likely be cached on compute nodes themselves 
and won't need to be copied at all.

Having the image bits streaming through an unrelated endpoint (the 
Glance API server) is just not required if the logic for grabbing the 
"closest" image from a set of locations is all in the glance_store 
library and different nova-compute daemons can just efficiently grab the 
image from multiple locations if the image is stored in multiple 
locations in backend storage.

In addition, Glance's image cache, while excellent for improving 
performance of first-pull of images from slower backend storage like 
Swift, also requires the operator to have a bunch of disk space used on 
the controller nodes that run glance-api. In many deployments that I 
know of, the controller nodes do not run stateful services (DB and MQ 
are on separate node clusters entirely from nova-api, glance-api, 
cinder-api, etc), and because of this, don't have a large root disk 
(sometimes nothing more than a small SSD for the OS kernel and some 
swap). Setting up Glance's image cache on these types of nodes means you 
need to be careful not to run out of local disk space, since a single 
popular Windows image can easily be 20-40+ GB. In addition to that, each 
glance-api server is going to have its own image cache, not all with the 
same images in them, since different requests will be routed to 
different glance-api servers, and each image cache is its own LRU layout.

Having the image cache local to the compute nodes themselves gives the 
best performance overall, and with glance_store, means that glance-api 
isn't needed at all, and Glance can become just a metadata repository, 
which would be awesome, IMHO.

Best,
-jay

> So I'm curious about what exactly the use cases for direct backend storage
> communication are, and why Glance can't meet them.