Open Stack

Sometimes I’ve had similar problems that can by fixed by running fsck against the rbd device in bare meta oob via rbd-nbd.  I’ve been thinking it’s related to trim/discard and some sort of disk geometry mismatch.

> On Apr 30, 2018, at 1:22 PM, Jonathan Proulx <jon at csail.mit.edu> wrote:
> 
> 
> In Proulx's Corollary to Murphy's Law, just after hitting send I tried
> something that "worked".
> 
> I noticed the volume shared nothing with the image it was based on
> so tried "flattening" it just to try something.
> 
> Oddly that worked, that or just having waited in power off state for
> an hour wile I was at lunch.
> 
> Still have no theory on why it broke or how that could be a fix...if
> anyone else does please do tell :)
> 
> Thanks,
> -JOn
> 
> On Mon, Apr 30, 2018 at 12:58:16PM -0400, Jonathan Proulx wrote:
> :Hi All,
> :
> :I have a VM with ephemeral root on RBD spewing I/O erros on boot after
> :hypervisor crash.  I've (unfortunately) seen a lot of hypervisors go
> :down badly with lots of VMs on them and this is a new one on me.
> :
> :I can 'rbd export' the volume and I get a clean filesystem.
> :
> :version details
> :
> :OpenStack: Mitaka
> :Host OS:   Ubuntu 16.04
> :Ceph:      Luminous (12.2.4)
> :
> :after booting to initrd VM shows:
> :
> :end_request: I/O error, dev vda, sector <lots of sectors>
> :
> :Tried hard reboot, tried rescue (in which case vdb shows same
> :issue) tried migrating to different hypervisor and all have consistent
> :failure.
> :
> :I do have writeback caching enable on the crashed hypervisor so I can
> :imaging filesystem corruption, but not this type of I/O error.
> :
> :Also if the rbd volume doesn't seem to be dammaged since I could dump
> :it to an iamge and see correct partioning and filesystems.
> :
> :Anyone seen this before? I have the bits since the export worked but
> :concerned about possibility of recurrence.
> :
> :Thanks,
> :-Jon
> :
> :-- 
> 
> -- 
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators