Open Stack

On 11/03/2012 01:47 PM, Joshua Harlow wrote:
> Hi all,
> 
> I was looking at the image cache code and since we are hitting hashing
> taking up the whole nova-compute process I had a few questions.

Sorry for the slow reply to this, I was on an air plane returning from UDS.

> Is it meant for the image cache process to run inside nova-compute, the
> fact that it does by default seems rather odd (since anyone having large
> image files will be hit when a sha1 of that whole file happens, this
> will most likely stop that nova-compute instance from responding until
> its done). This then happens every X 'ticks', which means if that tick
> count isn't set 'smartly' u basically end up turning nova-compute into a
> 'hash calculation engine' (for lack of better words).

I wrote this image management code, so I guess I should try and explain
what I was thinking at the time. The intention here is to checksum
images to ensure that they haven't been corrupted in some way -- we
weren't necessarily thinking of malicious changes here, but more file
corruption. This was a concern because once an image is cached we start
instances off it from that point, so if something did go wrong we'd want
to warn you about it instead of returning instances in an invalid state
from then on.

However, I can't come up with a good example of why a base image would
be corrupt, so perhaps we're being overly conservative here.

That said -- you can disable the checksumming by setting
checksum_base_images to false. Additionally, it isn't intended that you
run the image management code particularly frequently -- once an hour or
something along those lines is the default.

> Should that cache process be run out of process? I think it might have
> been like this a while ago (I think I remember seeing a separate binary,
> but don't see one now, any reason for that?) 

No, this has always happened in nova-compute. There was a proposal to
add a binary which pre-fetched images to compute nodes, but that code
never landed.

> Anyways if others are hitting this, the following might
> help: https://review.openstack.org/#/c/15328

There have been some comments on that review I see. I'd be interested in
knowing some more about your images before commenting though. What is
the average size of your base images? What is the largest? That would
help me think more about these performance problems.

Cheers,
Mikal

PS: this email is brought to you by jetlag and red bull, so sorry if its
incoherent.