Open Stack

On 9 June 2015 at 03:50, Chris Friesen <chris.friesen at windriver.com> wrote:
> On 06/07/2015 04:22 PM, Robert Collins wrote:

> Hi, original reporter here.
>
> There's no LB involved.  The issue was noticed in a test lab that is tight
> on disk space.  When an instance failed to boot the person using the lab
> tried to delete some images to free up space, at which point it was noticed
> that space didn't actually free up.  (For at least half an hour, exact time
> unknown.)
>
> I'm more of a nova guy, so could you elaborate a bit on the GC?  Is
> something going to delete the ChunkedFile object after a certain amount of
> inactivity? What triggers the GC to run?

Ok, updating my theory...  I'm not super familiar with glances
innards, so I'm going to call out my assumptions.

The ChunkedFile object is in the Nova process. It reads from a local
file, so its the issue - and it has a handle onto the image because
glance arranged for it to read from it.

Nova runs out of space and throws in the middle of iterating the ChunkedFile.

the code in https://review.openstack.org/#/c/188179/1/nova/image/glance.py
(base) propogates that exception, hitting the
 finally:
     if close_file:
        data.close()

on the way out.

the image_chunks iterator, which is being iterated by the for loop,
will be closed when the for loop is left because there are no other
references to the iterator object.

So the in-nova ChunkedFile object will have its close method called
due to the close() in the finally block of the __iter__ method
automatically when you except out of that exception. (demo below).
Assumption: glance_store/_drivers/filesystem.py is the ChunkedFile in
question; it might not be. If e.g. you are reading from the glance
HTTP service, then it might be a different class.

Anyhow - to answer your question: the iterator is only referenced by
the for loop, nothing else *can* hold a reference to it (without nasty
introspection hacks) and so as long as the iterator has an appropriate
try:finally:, which the filesystem ChunkedFile one does- the file will
be closed automatically.

So things to verify:
 - it is filesystem.ChunkedFile that you're seeing when you reproduce
it (within Nova). If not, what class is it :).

If its an HTTP handle of some sort, then we just need to make sure it
has an appropriate try:finally: within its generator. Assuming that is
that the contract with glance client is 'and you get an iterator' not
a 'file like object'.

>>> class f:
...   def __iter__(self):
...     try:
...       yield 1
...       1/0
...     finally:
...       print('yay')
...
>>> for x in f():
...   pass
...
yay
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __iter__
ZeroDivisionError: integer division or modulo by zero
>>> for x in f():
...   raise Exception()
...
yay
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud