[openstack-dev] [glance] Missing records in "image_locations" table
Sergio A. de Carvalho Jr.
scarvalhojr at gmail.com
Wed Nov 30 10:24:11 UTC 2016
A while ago I reported some issues with Glance where image files went
missing from the file system. Since then we implemented a monitoring script
that regularly verifies the state of every active image in the Glance
database, checking that:
1) it has a corresponding record in the "image_locations" table;
2) it has an image file on the disk;
3) the size of the image file matches size information in the database.
As it turns out, over the last 5 weeks we observed several cases (55 images
across 7 different clusters) where an image was successfully uploaded to
the database and reached "active" state, although there was no record in
the "images_location" table. The impact of such inconsistencies is that any
attempt to boot a VM with these images result in obscure 500 errors to
clients as Glance fails to retrieve the image from the backend.
It appears that the problem is more likely to manifest itself when a high
number of images are being uploaded at the same time -- for instance, we
recently had to upload around 1700 images to 2 new clusters in one go and
ended up with 23 images missing "image_locations" in one cluster, and 25 in
All these cluster are running Glance Icehouse (2014.1.3-2.el6) on CentOS
6.5, using the "file" backend store.
I haven't found any bug reports or anyone complaining of similar issues in
the community, so I'm curious to hear if anyone has ever experienced this
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev