Open Stack

On 13/08/15 11:11 +0000, Kuvaja, Erno wrote:
>
>
>> -----Original Message-----
>> From: Mike Perez [mailto:thingee at gmail.com]
>> Sent: Wednesday, August 12, 2015 4:45 PM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [Nova] [Cinder] [Glance] glance_store and
>> glance
>>
>> On Wed, Aug 12, 2015 at 2:23 AM, Kuvaja, Erno <kuvaja at hp.com> wrote:
>> >> -----Original Message-----
>> >> From: Mike Perez [mailto:thingee at gmail.com]
>> >> Sent: 11 August 2015 19:04
>> >> To: OpenStack Development Mailing List (not for usage questions)
>> >> Subject: Re: [openstack-dev] [Nova] [Cinder] [Glance] glance_store
>> >> and glance
>> >>
>> >> On 15:06 Aug 11, Kuvaja, Erno wrote:
>> >> > > -----Original Message-----
>> >> > > From: Jay Pipes [mailto:jaypipes at gmail.com]
>> >>
>> >> <snip>
>> >>
>> >> > > Having the image cache local to the compute nodes themselves
>> >> > > gives the best performance overall, and with glance_store, means
>> >> > > that glance-api isn't needed at all, and Glance can become just a
>> >> > > metadata repository, which would be awesome, IMHO.
>> >> >
>> >> > You have any figures to back this up in scale? We've heard similar
>> >> > claims for quite a while and as soon as people starts to actually
>> >> > look into how the environments behaves, they quite quickly turn
>> >> > back. As you're not the first one, I'd like to make the same
>> >> > request as to everyone before, show your data to back this claim
>> >> > up! Until that it is just like you say it is, opinion.;)
>> >>
>> >> The claims I make with Cinder doing caching on its own versus just
>> >> using Glance with rally with an 8G image:
>> >>
>> >> Creating/deleting 50 volumes w/ Cinder image cache: 324 seconds
>> >> Creating/delete 50 volumes w/o Cinder image cache: 3952 seconds
>> >>
>> >> http://thing.ee/x/cache_results/
>> >>
>> >> Thanks to Patrick East for pulling these results together.
>> >>
>> >> Keep in mind, this is using a block storage backend that is
>> >> completely separate from the OpenStack nodes. It's *not* using a
>> >> local LVM all in one OpenStack contraption. This is important because
>> >> even if you have Glance caching enabled, and there was no cache miss,
>> >> you still have to dd the bits to the block device, which is still
>> >> going over the network. Unless Glance is going to cache on the storage
>> array itself, forget about it.
>> >>
>> >> Glance should be focusing on other issues, rather than trying to make
>> >> copying image bits over the network and dd'ing to a block device faster.
>> >>
>> >> --
>> >> Mike Perez
>> >>
>> > Thanks Mike,
>> >
>> > So without cinder cache your times averaged roughly 150+second marks.
>> > The couple of first volumes with the cache took roughly 170+seconds.
>> > What the data does not tell, was cinder pulling the images directly
>> > from glance backend rather than through glance on either of these cases?
>>
>> Oh but I did, and that's the beauty of this, the files marked cinder-cache-
>> x.html are avoiding Glance as soon as it can, using the Cinder generic image
>> cache solution [1]. Please reread my when I say Glance is unable to do
>> caching in a storage array, so we don't rely on Glance. It's too slow otherwise.
>>
>> Take this example with 50 volumes created from image with Cinder's image
>> cache
>> [2]:
>>
>> * Is using Glance cache (oh no cache miss)
>> * Downloads the image from whatever glance store
>> * dd's the bits to the exported block device.
>> * the bits travel to the storage array that the block device was exported
>> from.
>> * [2nd-50th] request of that same image comes, Cinder instead just
>> references
>>   a cinder:// endpoint which has the storage array do a copy on write. ZERO
>>   COPYING since we can clone the image. Just a reference pointer and done,
>> move
>>   on.
>>
>> > Somehow you need to seed those caches and that seeding
>> time/mechanism
>> > is where the debate seems to be. Can you afford keeping every image in
>> > cache so that they are all local or if you need to pull the image to
>> > seed your cache how much you will benefit that your 100 cinder nodes
>> > are pulling it directly from backend X versus glance caching/sitting
>> > in between. How block storage backend handles that 100 concurrent
>> > reads by different client when you are seeding it between different
>> > arrays? The scale starts matter here because it makes a lot of
>> > difference on backend if it's couple of cinder or nova nodes
>> > requesting the image vs. 100s of them. Lots of backends tends to not
>> > like such loads or we outperform them due not having to fight for the
>> bandwidth with other consumers of that backend.
>>
>> Are you seriously asking if a backend is going to be with stand concurrent
>> reads compared to Glance cache?
>>
>> All storage backends do is I/O, unlike Glance which is trying to do a million
>> things and just pissing off the community.
>
>Thanks, I'm so happy to hear that it's not just couple of us who thinks that the project is lacking focus.

This is one of the reasons why it was asked for this email to be sent
rather than making decisions based on assumptions (you're free to read
- or not - the meeting logs).

Saying "Glance is just pissing off the community" helps spreading a
rumor that has been floating around which causes more harm than good
and I'm not willing to accept that.

If people have problems with were Glance is headed then please, speak
up. I've done that for Glance, Zaqar, Trove and several other
projects. I expect the same courtesy from other folks in this
community, especially folks in leadership positions.

As hard as it can be, in OpenStack it's never too late to correct
issues and change directions if needed, hence this email.

>>
>> They do it pretty darn well and are a lot more sophisticated than Glance
>> cache.
>> I'd pick Ceph w/ Cinder generic image cache doing copy on writes over
>> Glance cache any day.
>>
>> As it stands Cinder will be recommending in documentation for users to use
>> the generic image cache solution over Glance Cache.
>>
>
>That looks interesting, in a very good way.
>
>From the commit message of that review you referred:
>"""
>These image-volumes are host specific, so each backend may end up with
>its very own image-volume to do clones from.
>"""
>Does that mean that each cinder host will still need to pull the image from glance rather than sharing the cache across the deployment? No pun intended, but if so, this is the exact point we are referring here. It doesn't matter if it's 100 nova or 100 cinder hosts, they will get the stuff for their local caches faster via glance than talking directly to for example swift. And that is the step we're talking here all the time. I doubt anyone is willing to question the performance benefit of local cache in either nova or cinder case.
>
>My question was sincere regarding multiple hosts accessing the same cinder volume (ref. having glance using cinder backend and having the glance by pass), my perhaps false understanding has been that block devices rarely like multiple hosts poking them. Not questioning the raw I/O they provide. And if my assumption of 100 hosts accessing same volume concurrently causing issues was false, then I definitely see the benefit here being able to just give cinder reference pointer if cinder backend is in use, but in that case I do not see any benefit cinder consuming the glance_store to do those actions. I'm pretty sure ye guys know bit better how to optimize your operations than we do. And that location data is available via V2 API already if deployer allows.
>
>This means using cinder backend in glance and the caching ye guys are working on, you do not need glance_store, you can just go and request glance to hand over the metadata including the location of the image instead of ask glance to provide the image data. Now, last update of cinder driver condition in glance_store I've heard wasn't great so that might need bit more work to be viable option in production use.
>
>I really appreciate you taking the time to bend the railroad for me(/us),

Let me try to give a different perspective to this thread. We keep
arguing on whether people have a use-case to access the image data
directly or not without thinking that, regardless of the existence of
glance_store, consumers of Glance's API *can* already do that.

The above, to me, translates to "Glance doesn't own the image data"
because it's possible to alter it from outside. There, I said it.

As long as this is true, I'd rather have a library that provides a
common interface to these stores than having everyone implementing
their own abstraction to access such data. Let's not even talk about
whether glance_store can be specialized or not. I believe there's a
use case for glance_store as long as we allow users to access the
store directly.

I'm intentionally skipping the performance discussion above because I
believe that, once again, the Glance team is loosing/changing focus by
simply ignoring what Glance is today. That is, a bookstore with a very
old librarian - no offense intended to anyone - that can either tell
younger librarians where to find the books or slooooowly do it
herself/himself everytime they need them.

Flavio

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150813/5b1a7dc8/attachment.pgp>