<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Hi,</div><div class=""><br class=""></div><div class="">I've fixed the issue.</div><div class=""><br class=""></div><div class="">First, I would like to thank to all Ceph developers for making it bulletproof.</div><div class=""><br class=""></div><div class="">The root cause was "force_config_drive"[1] option of Nova that I had enabled few weeks ago. When you enable this option, Nova creates a new disk with the same name ending with ".config". The reason why I had enabled this option is, I am facing dhcp related issues sometimes.</div><div class=""><br class=""></div><div class="">Temporary disabling this option fixed the issue.</div><div class=""><br class=""></div><div class="">Regards</div><div class=""><br class=""></div><div class="">[1] <a href="https://docs.openstack.org/nova/stein/configuration/config.html#DEFAULT.force_config_drive" class="">https://docs.openstack.org/nova/stein/configuration/config.html#DEFAULT.force_config_drive</a></div><div class=""><br class=""></div><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>/* Please encrypt every message you can. Privacy is your right, don't let anyone take it from you. */</div><div><br class=""></div><div>/* My fingerprint is: 5E50 ABB0 F108 24DA 10CC BD43 D2AE DD2A 7893 0EAA */</div></div></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 18 Oct 2019, at 14:29, Eugen Block <<a href="mailto:eblock@nde.ag" class="">eblock@nde.ag</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">I assumed the header was missing because of this message:<br class=""><br class=""><blockquote type="cite" class="">error reading header from c2b54eac-179b-4907-9d61-8e075edc21cf_disk.config: No such file or directory<br class=""></blockquote><br class="">If you can stat the header file can you share the output of<br class=""><br class="">rados -p vms listomapvals rbd_header.<BLOCK_PREFIX><br class=""><br class="">Are there rbd_data objects left in the pool from that config drive?<br class=""><br class="">rados -p images ls | grep <BLOCK_PREFIX><br class="">rbd_object_map.1cbc666b8b4567<br class="">rbd_data.1cbc666b8b4567.0000000000000000<br class="">rbd_header.1cbc666b8b4567<br class=""><br class="">If yes, maybe there's a way to set things back together, which I haven't done yet. Are all affected VMs referring to a config drive and is it always the config drive object that's missing?<br class=""><br class=""><br class="">Zitat von Dinçer Çelik <<a href="mailto:hello@dincercelik.com" class="">hello@dincercelik.com</a>>:<br class=""><br class=""><blockquote type="cite" class="">Hi Eugen,<br class=""><br class="">I think this is not the same situation with I’m facing because I can get rbd headers.<br class=""><br class="">Regards<br class=""><br class="">/* Please encrypt every message you can. Privacy is your right, don't let anyone take it from you. */<br class=""><br class="">/* My fingerprint is: 5E50 ABB0 F108 24DA 10CC BD43 D2AE DD2A 7893 0EAA */<br class=""><br class=""><blockquote type="cite" class="">On 18 Oct 2019, at 09:44, Eugen Block <<a href="mailto:eblock@nde.ag" class="">eblock@nde.ag</a>> wrote:<br class=""><br class="">Hi,<br class=""><br class="">I've recently found this post [1] to recover a failing header, but I haven't tried it myself. I'm curios if it works though.<br class=""><br class="">Regards,<br class="">Eugen<br class=""><br class=""><a href="https://fnordahl.com/2017/04/17/ceph-rbd-volume-header-recovery/" class="">https://fnordahl.com/2017/04/17/ceph-rbd-volume-header-recovery/</a><br class=""><br class=""><br class="">Zitat von Dinçer Çelik <hello@dincercelik.com>:<br class=""><br class=""><blockquote type="cite" class="">Greetings,<br class=""><br class="">Today I had a data center power outage, and the OpenStack cluster went down. After taking the cluster up again, I cannot start some VMs due to error below. I've tried "rbd object-map rebuild" but it didn't work. What's the proper way to re-create the missing "_disk.config" files?<br class=""><br class="">Thanks.<br class=""><br class="">[instance: c2b54eac-179b-4907-9d61-8e075edc21cf] Failed to start libvirt guest: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-10-17T23:19:41.103720Z qemu-system-x86_64: -drive file=rbd:vms/c2b54eac-179b-4907-9d61-8e075edc21cf_disk.config:id=nova:auth_supported=cephx\;none:mon_host=10.250.129.10\:6789\;10.250.129.11\:6789\;10.250.129.12\:6789\;10.250.129.15\:6789,file.password-secret=ide0-0-0-secret0,format=raw,if=none,id=drive-ide0-0-0,readonly=on,cache=writeback,discard=unmap: error reading header from c2b54eac-179b-4907-9d61-8e075edc21cf_disk.config: No such file or directory<br class=""><br class="">/* Please encrypt every message you can. Privacy is your right, don't let anyone take it from you. */<br class=""><br class="">/* My fingerprint is: 5E50 ABB0 F108 24DA 10CC BD43 D2AE DD2A 7893 0EAA */<br class=""></blockquote><br class=""><br class=""><br class=""><br class=""></blockquote></blockquote><br class=""><br class=""><br class=""></div></div></blockquote></div><br class=""></body></html>