[Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image

Wangpan hzwangpan at corp.netease.com
Wed Sep 26 10:14:27 UTC 2012


Thanks for your reminders, I found the reason now, it's my own fault!
I tried to optimize the snapshot_instance process like this blueprint:
https://blueprints.launchpad.net/glance/+spec/hierarchical-images
and I copied the call_if_no_exists & copy_and_extend methods, and I forgot the locks @utils.synchronized!
I think it's really OK now.

What a stupid mistake!

2012-09-26



Wangpan



发件人:Boris Filippov
发送时间:2012-09-26 03:25
主题:Re: Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image
收件人:"Wangpan"<hzwangpan at corp.netease.com>
抄送:"Razique Mahroua"<razique.mahroua at gmail.com>,"Scott Moser"<smoser at ubuntu.com>,"openstack"<openstack at lists.launchpad.net>

Actually, this is unlikely because there was locking for image fetching already. This patch only adds it for case when several compute processes share same $instances_dir. 
- disk full really might be the case. Simple check is to extend available space and try to reproduce it again.


2012/9/25 Wangpan <hzwangpan at corp.netease.com>

Thanks for your clue.
yes, the occured condition of my issue is the same as BUG 1014227, but the difference is that my instance directory is NOT shared between compute nodes,
the instances' images are saved in local disk on compute nodes.
I believe the bug Boris found may occur in NOT shared storage circumstance as mine, and my issue may be resloved by this patch which adds a lock to cache image method.
Thanks again.


2012-09-25



Wangpan



发件人:Boris Filippov
发送时间:2012-09-24 19:59
主题:Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image
收件人:"Razique Mahroua"<razique.mahroua at gmail.com>
抄送:"Wangpan"<hzwangpan at corp.netease.com>,"openstack"<openstack at lists.launchpad.net>

Your instance directory is shared between compute nodes?
Similar issue was fixed for folsom-rc1 
https://review.openstack.org/#/c/12024/

https://bugs.launchpad.net/nova/+bug/1014227


2012/9/24 Razique Mahroua <razique.mahroua at gmail.com>

Here is what I found 
http://comments.gmane.org/gmane.comp.emulators.qemu/153254
http://osdir.com/ml/qemu-devel/2012-05/msg04234.html


What do you think ?


Nuage & Co - Razique Mahroua  
razique.mahroua at gmail.com

 


Le 24 sept. 2012 à 10:43, Wangpan <hzwangpan at corp.netease.com> a écrit :


qemu-img check -f qcow2 test.qcow2
No errors were found on the image.

2012-09-24



Wangpan



发件人:Razique Mahroua
发送时间:2012-09-24 16:30
主题:Re: Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image
收件人:"Wangpan"<hzwangpan at corp.netease.com>
抄送:"openstack"<openstack at lists.launchpad.net>

hi,  
what about a 
qemu-img check -f qcow2 test.qcow2




Nuage & Co - Razique Mahroua 
razique.mahroua at gmail.com

<NUAGECO-LOGO-Fblan_petit(1).jpg>


Le 24 sept. 2012 à 10:19, "Wangpan"<hzwangpan at corp.netease.com> a écrit :


Thanks for your attention!

- disk full  ----# I believe there is enough space, because I have tried to boot serveral instances after this error with the same image, they are all booted successfully.
- permission issues    ----# Not this reason, too. explain as above.

I still believe this is a race problem, if we convert to image from qcow2 to raw at a short interval, one of the convertion may fail, 
I have verified this by running two same commands continuously such as:
qemu-img convert -f qcow2 -O raw  test.qcow2 test.part &
and one of them give me the same error:
qemu-img: error while writing sector 1052672: Input/output error



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack at lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20120926/335e6046/attachment.html>


More information about the Openstack mailing list