Open Stack

Hi,

We've been seeing an issue in one of our (semi-)production systems running
Icehouse on CentOS 6.5 in which VMs fail to spawn due to a "corrupted
image". Nova compute log shows this:


2015-09-16 18:38:17.439 8279 ERROR nova.compute.manager
[req-a77819b1-7137-4c1d-9465-3f6d55c64c4c ******
27076b1b035c4f278a581c3d938360a7] [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1] Instance failed to spawn
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1] Traceback (most recent call last):
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1737, in
_spawn
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     block_device_info)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2287,
in spawn
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     admin_pass=admin_password)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2656,
in _create_image
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     project_id=instance['project_id'])
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line
192, in cache
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     *args, **kwargs)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line
383, in create_image
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     prepare_template(target=base,
max_size=size, *args, **kwargs)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line
249, in inner
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     return f(*args, **kwargs)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line
182, in fetch_func_sync
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     fetch_func(target=target, *args,
**kwargs)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 653, in
fetch_image
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     max_size=max_size)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/images.py", line 78, in
fetch_to_raw
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     max_size=max_size)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/virt/images.py", line 72, in fetch
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     image_service.download(context,
image_id, dst_path=path)
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/nova/image/glance.py", line 342, in
download
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     for chunk in image_chunks:
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]   File
"/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 504,
in __iter__
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]     'expected %s' % (md5sum,
self._checksum))
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1] IOError: [Errno 32] Corrupted image.
Checksum was 43525c5261f518e96831b93ffa530afe expected
60589b89b66722a4e41c1607a4385882
2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance:
3cd644f2-c392-4306-8463-4fe2708d31e1]
2015-09-16 18:38:17.441 8279 DEBUG nova.compute.claims
[req-a77819b1-7137-4c1d-9465-3f6d55c64c4c ******
27076b1b035c4f278a581c3d938360a7]
[instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] Aborting claim: [Claim:
4096 MB memory, 40 GB disk, 2 VCPUS] abort
/usr/lib/python2.6/site-packages/nova/compute/claims.py:113
2015-09-16 18:38:17.473 8279 DEBUG nova.compute.utils
[req-a77819b1-7137-4c1d-9465-3f6d55c64c4c ******
27076b1b035c4f278a581c3d938360a7]
[instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] [Errno 32] Corrupted
image. Checksum was 43525c5261f518e96831b93ffa530afe expected
60589b89b66722a4e41c1607a4385882 notify_about_instance_usage
/usr/lib/python2.6/site-packages/nova/compute/utils.py:336


This bug looks related: https://bugs.launchpad.net/nova/+bug/1350766

However, after a few patches were proposed to the above, it seems that the
bug was classified as a qemu issue:
* https://bugs.launchpad.net/qemu/+bug/1368815
* https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1292234

It's not clear, however, if these bugs are what exactly what we're seeing
and I could not replicate the problem with the manual steps described in
them. Also, I couldn't figure out what versions of qemu have the fix and if
the particular version of qemu-img is broken or not. This is the package we
have installed:

Name        : qemu-img
Arch        : x86_64
Epoch       : 2
Version     : 0.12.1.2
Release     : 2.415.el6_5.14
Size        : 969 k
Repo        : installed
>From repo   : C65-Updates

We're using the default ext3 filesystems.

Any help in solving the mystery here would be hugely appreciated.

Thanks,

-Sergio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150924/793dbcab/attachment.html>