Open Stack

> -----Original Message-----
> From: Jay Pipes [mailto:jaypipes at gmail.com]
> Sent: Tuesday, August 11, 2015 3:10 PM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [Nova] [Cinder] [Glance] glance_store and
> glance
> 
> On 08/11/2015 09:42 AM, Brian Rosmaita wrote:
> > On 8/7/15, 1:07 PM, "Jay Pipes" <jaypipes at gmail.com> wrote:
> >
> >> So, here's the crux of the issue. Nova and Cinder **do not want to
> >> speak the Glance REST API** to either upload or download image bits
> >> from storage. Streaming image bits through the Glance API endpoint is
> >> a needless and inefficient step, and Nova and Cinder would like to
> >> communicate directly with the backend storage systems.
> >
> > Exactly why do you want to communicate directly with the backend
> > storage systems?  Streaming image bits through Glance appears to be
> > "needless and inefficient", but if an end-user is booting 1K instances
> > from some custom image, Glance's image cache makes an enormous
> difference in delivery time.
> 
> Nova's image cache makes a bigger difference overall in those cases, since
> the image will most likely be cached on compute nodes themselves and
> won't need to be copied at all.

If that would be the case we wouldn't have this problem. If nova image caching solves all those boot time issues it probably doesn't matter that the image comes from glance?
> 
> Having the image bits streaming through an unrelated endpoint (the Glance
> API server) is just not required if the logic for grabbing the "closest" image
> from a set of locations is all in the glance_store library and different nova-
> compute daemons can just efficiently grab the image from multiple locations
> if the image is stored in multiple locations in backend storage.

I think the _if_ there is really relevant. So what you are actually asking is not to merge glance_store to glance but merge glance to glance_store so you get all the acl controls, policies, data integrity promise, etc. to glance_store? It's not just where the bits are. We could then access it all just through nova's image API. Perhaps provide Nova Images API V2 for cinder and Horizon; V3 for the future artifacts consumers. Jay, if you miss glance so much, we happily take your contribution to the project, you don't need to be ranting there that you want it back to part of nova to contribute. ;)
> 
> In addition, Glance's image cache, while excellent for improving performance
> of first-pull of images from slower backend storage like Swift, also requires
> the operator to have a bunch of disk space used on the controller nodes that
> run glance-api. In many deployments that I know of, the controller nodes do
> not run stateful services (DB and MQ are on separate node clusters entirely
> from nova-api, glance-api, cinder-api, etc), and because of this, don't have a
> large root disk (sometimes nothing more than a small SSD for the OS kernel
> and some swap). Setting up Glance's image cache on these types of nodes
> means you need to be careful not to run out of local disk space, since a single
> popular Windows image can easily be 20-40+ GB. In addition to that, each
> glance-api server is going to have its own image cache, not all with the same
> images in them, since different requests will be routed to different glance-
> api servers, and each image cache is its own LRU layout.

This explains a lot.
> 
> Having the image cache local to the compute nodes themselves gives the
> best performance overall, and with glance_store, means that glance-api isn't
> needed at all, and Glance can become just a metadata repository, which
> would be awesome, IMHO.

You have any figures to back this up in scale? We've heard similar claims for quite a while and as soon as people starts to actually look into how the environments behaves, they quite quickly turn back. As you're not the first one, I'd like to make the same request as to everyone before, show your data to back this claim up! Until that it is just like you say it is, opinion.;)

- Erno
> 
> Best,
> -jay
> 
> > So I'm curious about what exactly the use cases for direct backend
> > storage communication are, and why Glance can't meet them.
> 
> __________________________________________________________
> ________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev