Open Stack

> -----Original Message-----
> From: Mike Perez [mailto:thingee at gmail.com]
> Sent: Wednesday, August 12, 2015 4:45 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Nova] [Cinder] [Glance] glance_store and
> glance
> 
> On Wed, Aug 12, 2015 at 2:23 AM, Kuvaja, Erno <kuvaja at hp.com> wrote:
> >> -----Original Message-----
> >> From: Mike Perez [mailto:thingee at gmail.com]
> >> Sent: 11 August 2015 19:04
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> Subject: Re: [openstack-dev] [Nova] [Cinder] [Glance] glance_store
> >> and glance
> >>
> >> On 15:06 Aug 11, Kuvaja, Erno wrote:
> >> > > -----Original Message-----
> >> > > From: Jay Pipes [mailto:jaypipes at gmail.com]
> >>
> >> <snip>
> >>
> >> > > Having the image cache local to the compute nodes themselves
> >> > > gives the best performance overall, and with glance_store, means
> >> > > that glance-api isn't needed at all, and Glance can become just a
> >> > > metadata repository, which would be awesome, IMHO.
> >> >
> >> > You have any figures to back this up in scale? We've heard similar
> >> > claims for quite a while and as soon as people starts to actually
> >> > look into how the environments behaves, they quite quickly turn
> >> > back. As you're not the first one, I'd like to make the same
> >> > request as to everyone before, show your data to back this claim
> >> > up! Until that it is just like you say it is, opinion.;)
> >>
> >> The claims I make with Cinder doing caching on its own versus just
> >> using Glance with rally with an 8G image:
> >>
> >> Creating/deleting 50 volumes w/ Cinder image cache: 324 seconds
> >> Creating/delete 50 volumes w/o Cinder image cache: 3952 seconds
> >>
> >> http://thing.ee/x/cache_results/
> >>
> >> Thanks to Patrick East for pulling these results together.
> >>
> >> Keep in mind, this is using a block storage backend that is
> >> completely separate from the OpenStack nodes. It's *not* using a
> >> local LVM all in one OpenStack contraption. This is important because
> >> even if you have Glance caching enabled, and there was no cache miss,
> >> you still have to dd the bits to the block device, which is still
> >> going over the network. Unless Glance is going to cache on the storage
> array itself, forget about it.
> >>
> >> Glance should be focusing on other issues, rather than trying to make
> >> copying image bits over the network and dd'ing to a block device faster.
> >>
> >> --
> >> Mike Perez
> >>
> > Thanks Mike,
> >
> > So without cinder cache your times averaged roughly 150+second marks.
> > The couple of first volumes with the cache took roughly 170+seconds.
> > What the data does not tell, was cinder pulling the images directly
> > from glance backend rather than through glance on either of these cases?
> 
> Oh but I did, and that's the beauty of this, the files marked cinder-cache-
> x.html are avoiding Glance as soon as it can, using the Cinder generic image
> cache solution [1]. Please reread my when I say Glance is unable to do
> caching in a storage array, so we don't rely on Glance. It's too slow otherwise.
> 
> Take this example with 50 volumes created from image with Cinder's image
> cache
> [2]:
> 
> * Is using Glance cache (oh no cache miss)
> * Downloads the image from whatever glance store
> * dd's the bits to the exported block device.
> * the bits travel to the storage array that the block device was exported
> from.
> * [2nd-50th] request of that same image comes, Cinder instead just
> references
>   a cinder:// endpoint which has the storage array do a copy on write. ZERO
>   COPYING since we can clone the image. Just a reference pointer and done,
> move
>   on.
> 
> > Somehow you need to seed those caches and that seeding
> time/mechanism
> > is where the debate seems to be. Can you afford keeping every image in
> > cache so that they are all local or if you need to pull the image to
> > seed your cache how much you will benefit that your 100 cinder nodes
> > are pulling it directly from backend X versus glance caching/sitting
> > in between. How block storage backend handles that 100 concurrent
> > reads by different client when you are seeding it between different
> > arrays? The scale starts matter here because it makes a lot of
> > difference on backend if it's couple of cinder or nova nodes
> > requesting the image vs. 100s of them. Lots of backends tends to not
> > like such loads or we outperform them due not having to fight for the
> bandwidth with other consumers of that backend.
> 
> Are you seriously asking if a backend is going to be with stand concurrent
> reads compared to Glance cache?
> 
> All storage backends do is I/O, unlike Glance which is trying to do a million
> things and just pissing off the community.

Thanks, I'm so happy to hear that it's not just couple of us who thinks that the project is lacking focus.
> 
> They do it pretty darn well and are a lot more sophisticated than Glance
> cache.
> I'd pick Ceph w/ Cinder generic image cache doing copy on writes over
> Glance cache any day.
> 
> As it stands Cinder will be recommending in documentation for users to use
> the generic image cache solution over Glance Cache.
> 

That looks interesting, in a very good way.

>From the commit message of that review you referred:
"""
These image-volumes are host specific, so each backend may end up with
its very own image-volume to do clones from.
"""
Does that mean that each cinder host will still need to pull the image from glance rather than sharing the cache across the deployment? No pun intended, but if so, this is the exact point we are referring here. It doesn't matter if it's 100 nova or 100 cinder hosts, they will get the stuff for their local caches faster via glance than talking directly to for example swift. And that is the step we're talking here all the time. I doubt anyone is willing to question the performance benefit of local cache in either nova or cinder case. 

My question was sincere regarding multiple hosts accessing the same cinder volume (ref. having glance using cinder backend and having the glance by pass), my perhaps false understanding has been that block devices rarely like multiple hosts poking them. Not questioning the raw I/O they provide. And if my assumption of 100 hosts accessing same volume concurrently causing issues was false, then I definitely see the benefit here being able to just give cinder reference pointer if cinder backend is in use, but in that case I do not see any benefit cinder consuming the glance_store to do those actions. I'm pretty sure ye guys know bit better how to optimize your operations than we do. And that location data is available via V2 API already if deployer allows.

This means using cinder backend in glance and the caching ye guys are working on, you do not need glance_store, you can just go and request glance to hand over the metadata including the location of the image instead of ask glance to provide the image data. Now, last update of cinder driver condition in glance_store I've heard wasn't great so that might need bit more work to be viable option in production use.

I really appreciate you taking the time to bend the railroad for me(/us),
Erno
> 
> [1] - https://review.openstack.org/#/c/195795/
> [2] - http://thing.ee/x/cache_results/cinder-cache-
> 50.html#/CinderVolumes.create_and_delete_volume
> 
> --
> Mike Perez
> 
> __________________________________________________________
> ________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev