[openstack-dev] [nova] [glance] How to deal with aborted image read?
robertc at robertcollins.net
Tue Jun 9 00:26:29 UTC 2015
On 9 June 2015 at 07:53, Chris Friesen <chris.friesen at windriver.com> wrote:
> On 06/08/2015 12:30 PM, Robert Collins wrote:
>> On 9 June 2015 at 03:50, Chris Friesen <chris.friesen at windriver.com>
>>> On 06/07/2015 04:22 PM, Robert Collins wrote:
>>> Hi, original reporter here.
>>> There's no LB involved. The issue was noticed in a test lab that is
>>> on disk space. When an instance failed to boot the person using the lab
>>> tried to delete some images to free up space, at which point it was
>>> that space didn't actually free up. (For at least half an hour, exact
>>> I'm more of a nova guy, so could you elaborate a bit on the GC? Is
>>> something going to delete the ChunkedFile object after a certain amount
>>> inactivity? What triggers the GC to run?
>> Ok, updating my theory... I'm not super familiar with glances
>> innards, so I'm going to call out my assumptions.
>> The ChunkedFile object is in the Nova process. It reads from a local
>> file, so its the issue - and it has a handle onto the image because
>> glance arranged for it to read from it.
> As I understand it, the ChunkedFile object is in the glance-api process.
> (At least, it's the glance-api process that is holding the open file
>> Anyhow - to answer your question: the iterator is only referenced by
>> the for loop, nothing else *can* hold a reference to it (without nasty
>> introspection hacks) and so as long as the iterator has an appropriate
>> try:finally:, which the filesystem ChunkedFile one does- the file will
>> be closed automatically.
> From what I understand, the iterator (in the glance-api process) normally
> breaks out of the while loop once the whole file has been read and the
> read() call returns an empty string.
> It's not clear to me how an error in the nova process (which causes it to
> stop reading the file from glance-api) will cause glance-api to break out
> of the while loop in ChunkedFile.__iter__().
AIUI the conclusion of our IRC investigation was:
- with caching middleware, the fd is freed, just after ~4m.
- without caching middleware, the fd is freed after ~90s.
Robert Collins <rbtcollins at hp.com>
HP Converged Cloud
More information about the OpenStack-dev