Open Stack

Perhaps someone needs to use (or forgot to use) contextlib.closing?

https://docs.python.org/2/library/contextlib.html#contextlib.closing

Or contextlib2 also may be useful:

http://contextlib2.readthedocs.org/en/latest/#contextlib2.ExitStack

-Josh

Chris Friesen wrote:
> On 06/04/2015 03:01 AM, Flavio Percoco wrote:
>> On 03/06/15 16:46 -0600, Chris Friesen wrote:
>>> We recently ran into an issue where nova couldn't write an image file
>>> due to
>>> lack of space and so just quit reading from glance.
>>>
>>> This caused glance to be stuck with an open file descriptor, which
>>> meant that
>>> the image consumed space even after it was deleted.
>>>
>>> I have a crude fix for nova at
>>> "https://review.openstack.org/#/c/188179/"
>>> which basically continues to read the image even though it can't
>>> write it.
>>> That seems less than ideal for large images though.
>>>
>>> Is there a better way to do this? Is there a way for nova to indicate to
>>> glance that it's no longer interested in that image and glance can
>>> close the
>>> file?
>>>
>>> If I've followed this correctly, on the glance side I think the code in
>>> question is ultimately
>>> glance_store._drivers.filesystem.ChunkedFile.__iter__().
>>
>> Actually, to be honest, I was quite confused by the email :P
>>
>> Correct me if I still didn't understand what you're asking.
>>
>> You ran out of space on the Nova side while downloading the image and
>> there's a file descriptor leak somewhere either in that lovely (sarcasm)
>> glance wrapper or in glanceclient.
>
> The first part is correct, but the file descriptor is actually held by
> glance-api.
>
>> Just by reading your email and glancing your patch, I believe the bug
>> might be in glanceclient but I'd need to five into this. The piece of
>> code you'll need to look into is[0].
>>
>> glance_store is just used server side. If that's what you meant -
>> glance is keeping the request and the ChunkedFile around - then yes,
>> glance_store is the place to look into.
>>
>> [0]
>> https://github.com/openstack/python-glanceclient/blob/master/glanceclient/v1/images.py#L152
>>
>
> I believe what's happening is that the ChunkedFile code opens the file
> and creates the iterator. Nova then starts iterating through the file.
>
> If nova (or any other user of glance) iterates all the way through the
> file then the ChunkedFile code will hit the "finally" clause in
> __iter__() and close the file descriptor.
>
> If nova starts iterating through the file and then stops (due to running
> out of room, for example), the ChunkedFile.__iter__() routine is left
> with an open file descriptor. At this point deleting the image will not
> actually free up any space.
>
> I'm not a glance guy so I could be wrong about the code. The
> externally-visible data are:
> 1) glance-api is holding an open file descriptor to a deleted image file
> 2) If I kill glance-api the disk space is freed up.
> 3) If I modify nova to always finish iterating through the file the
> problem doesn't occur in the first place.
>
> Chris
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev