Open Stack

On 04/06/15 11:46 -0600, Chris Friesen wrote:
>On 06/04/2015 03:01 AM, Flavio Percoco wrote:
>>On 03/06/15 16:46 -0600, Chris Friesen wrote:
>>>We recently ran into an issue where nova couldn't write an image file due to
>>>lack of space and so just quit reading from glance.
>>>
>>>This caused glance to be stuck with an open file descriptor, which meant that
>>>the image consumed space even after it was deleted.
>>>
>>>I have a crude fix for nova at "https://review.openstack.org/#/c/188179/"
>>>which basically continues to read the image even though it can't write it.
>>>That seems less than ideal for large images though.
>>>
>>>Is there a better way to do this?  Is there a way for nova to indicate to
>>>glance that it's no longer interested in that image and glance can close the
>>>file?
>>>
>>>If I've followed this correctly, on the glance side I think the code in
>>>question is ultimately glance_store._drivers.filesystem.ChunkedFile.__iter__().
>>
>>Actually, to be honest, I was quite confused by the email :P
>>
>>Correct me if I still didn't understand what you're asking.
>>
>>You ran out of space on the Nova side while downloading the image and
>>there's a file descriptor leak somewhere either in that lovely (sarcasm)
>>glance wrapper or in glanceclient.
>
>The first part is correct, but the file descriptor is actually held by glance-api.
>
>>Just by reading your email and glancing your patch, I believe the bug
>>might be in glanceclient but I'd need to five into this. The piece of
>>code you'll need to look into is[0].
>>
>>glance_store is just used server side. If that's what you meant -
>>glance is keeping the request and the ChunkedFile around - then yes,
>>glance_store is the place to look into.
>>
>>[0]
>>https://github.com/openstack/python-glanceclient/blob/master/glanceclient/v1/images.py#L152
>
>I believe what's happening is that the ChunkedFile code opens the file 
>and creates the iterator.  Nova then starts iterating through the 
>file.
>
>If nova (or any other user of glance) iterates all the way through the 
>file then the ChunkedFile code will hit the "finally" clause in 
>__iter__() and close the file descriptor.
>
>If nova starts iterating through the file and then stops (due to 
>running out of room, for example), the ChunkedFile.__iter__() routine 
>is left with an open file descriptor.  At this point deleting the 
>image will not actually free up any space.
>
>I'm not a glance guy so I could be wrong about the code.  The 
>externally-visible data are:
>1) glance-api is holding an open file descriptor to a deleted image file
>2) If I kill glance-api the disk space is freed up.
>3) If I modify nova to always finish iterating through the file the 
>problem doesn't occur in the first place.

Gotcha, thanks for explaining. I think the problem is that there might
be a reference leak and therefore the FD is kept opened. Probably the
request interruption is not getting to the driver. I've filed this
bug[0] so we can look into it.

[0] https://bugs.launchpad.net/glance-store/+bug/1462235

Flavio

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150605/95cd36eb/attachment.pgp>