[Openstack-operators] I/O errors on RBD after hypervisor crash.

Jonathan Proulx jon at csail.mit.edu
Mon Apr 30 16:58:16 UTC 2018


Hi All,

I have a VM with ephemeral root on RBD spewing I/O erros on boot after
hypervisor crash.  I've (unfortunately) seen a lot of hypervisors go
down badly with lots of VMs on them and this is a new one on me.

I can 'rbd export' the volume and I get a clean filesystem.

version details

OpenStack: Mitaka
Host OS:   Ubuntu 16.04
Ceph:      Luminous (12.2.4)

after booting to initrd VM shows:

end_request: I/O error, dev vda, sector <lots of sectors>

Tried hard reboot, tried rescue (in which case vdb shows same
issue) tried migrating to different hypervisor and all have consistent
failure.

I do have writeback caching enable on the crashed hypervisor so I can
imaging filesystem corruption, but not this type of I/O error.

Also if the rbd volume doesn't seem to be dammaged since I could dump
it to an iamge and see correct partioning and filesystems.

Anyone seen this before? I have the bits since the export worked but
concerned about possibility of recurrence.

Thanks,
-Jon

-- 



More information about the OpenStack-operators mailing list