Open Stack

Hi everyone,

A while ago I reported some issues with Glance where image files went
missing from the file system. Since then we implemented a monitoring script
that regularly verifies the state of every active image in the Glance
database, checking that:
1) it has a corresponding record in the "image_locations" table;
2) it has an image file on the disk;
3) the size of the image file matches size information in the database.

As it turns out, over the last 5 weeks we observed several cases (55 images
across 7 different clusters) where an image was successfully uploaded to
the database and reached "active" state, although there was no record in
the "images_location" table. The impact of such inconsistencies is that any
attempt to boot a VM with these images result in obscure 500 errors to
clients as Glance fails to retrieve the image from the backend.

All these cluster are running Glance Icehouse (2014.1.3-2.el6) on CentOS
6.5, using the "file" backend store.

I haven't found any bug reports or anyone complaining of similar issues in
the community, so I'm curious to hear if anyone has ever experienced this
before. Looking at the occurrences in our clusters, it appears that the
problem is more likely to manifest itself when a high number of images are
being uploaded at the same time -- for instance, we recently had to upload
around 1700 images to 2 new clusters in one go and ended up with 23 images
missing "image_locations" in one cluster, and 25 in the other.

Thanks,

Sergio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161129/26521fe9/attachment.html>