<html><head></head><body><div>Hello,</div><div><br></div><div>I honestly dont know if it's a bug or a security. I faced several power outages now and Ceph won't remove any lock even after few days. </div><div><br></div><div>Maybe because Ceph MONs were down too?</div><div><br></div><div>Anyway, your thread motivate me to end my unlock script. It still needs improvement but it does the job (used many times now!).</div><div><br></div><div><a href="https://github.com/RomainLyon1/cephunlock">https://github.com/RomainLyon1/cephunlock</a></div><div><br></div><div>Best Regards,<br>Romain</div><div><br></div><div><br></div><div>On Fri, 2023-02-17 at 10:45 -0500, Satish Patel wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr">Hi Eugen,<div><br></div><div>I have a few questions before we close this thread. </div><div><br></div><div>- Is it normal that ceph locks images during power failure or disaster? </div><div>- Shouldn't ceph should release locks automatically when VMs shutdown? </div><div>- Is this a bug or natural behavior of ceph? I am worried what if i have 100s of VMs and remove lock of all of them</div><div><br></div><div><br></div></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 17, 2023 at 10:28 AM Satish Patel <<a href="mailto:satish.txt@gmail.com">satish.txt@gmail.com</a>> wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr">Hi Eugen,<div><br></div><div>You saved my life!!!!!! all my vms up without any filesystem error :) </div><div><br></div><div>This is the correct command to remove the lock. </div><div><br></div><div>$ rbd lock rm -p vms ec6044e6-2231-4906-9e30-1e2e72573e64_disk "auto 139643345791728" client.1211875</div><div><br></div><div><br></div></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 17, 2023 at 10:06 AM Satish Patel <<a href="mailto:satish.txt@gmail.com" target="_blank">satish.txt@gmail.com</a>> wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr">Hi Eugen,<div><br></div><div>I am playing with less important machine and i did following</div><div><br></div><div>I shutdown VM but still down following lock</div><div><br></div><div>root@ceph1:~# rbd lock list --image ec6044e6-2231-4906-9e30-1e2e72573e64_disk -p vms<br>There is 1 exclusive lock on this image.<br>Locker ID Address<br>client.1211875 auto 139643345791728 <a href="http://192.168.3.12:0/2259335316" target="_blank">192.168.3.12:0/2259335316</a><br></div><div><br></div><div>root@ceph1:~# ceph osd blacklist add <a href="http://192.168.3.12:0/2259335316" target="_blank">192.168.3.12:0/2259335316</a><br>blocklisting <a href="http://192.168.3.12:0/2259335316" target="_blank">192.168.3.12:0/2259335316</a> until 2023-02-17T16:00:59.399775+0000 (3600 sec)<br></div><div><br></div><div>Still I can see it in the following lock list. Am I missing something?</div><div><br></div><div>root@ceph1:~# rbd lock list --image ec6044e6-2231-4906-9e30-1e2e72573e64_disk -p vms<br>There is 1 exclusive lock on this image.<br>Locker ID Address<br>client.1211875 auto 139643345791728 <a href="http://192.168.3.12:0/2259335316" target="_blank">192.168.3.12:0/2259335316</a><br></div><div><br></div><div><br></div></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 17, 2023 at 2:39 AM Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>The lock is aquired automatically, you don't need to create one. I'm <br>curious why you have that many blacklist entries, maybe that is indeed <br>the issue here (locks are not removed). I would shutdown the corrupted <br>VM and see if the compute node still has a lock on that image, because <br>after shutdown it should remove the lock (automatically). If there's <br>still a watcher or lock on that image after shutdown (rbd status <br>vms/55dbf40b-0a6a-4bab-b3a5-b4bb74e963af_disk) you can try to <br>blacklist the client with:<br></div><div><br># ceph osd blacklist add client.<ID><br></div><div><br>Then check the status again, if no watchers are present, boot the VM.<br></div><div><br></div><div><br>Zitat von Satish Patel <<a href="mailto:satish.txt@gmail.com" target="_blank">satish.txt@gmail.com</a>>:<br></div><div><br>> Hi Eugen,<br>><br>> This is what I did, let me know if I missed anything.<br>><br>> root@ceph1:~# ceph osd blacklist ls<br>> <a href="http://192.168.3.12:0/0" rel="noreferrer" target="_blank">192.168.3.12:0/0</a> 2023-02-17T04:48:54.381763+0000<br>> <a href="http://192.168.3.22:0/753370860" rel="noreferrer" target="_blank">192.168.3.22:0/753370860</a> 2023-02-17T04:47:08.185434+0000<br>> <a href="http://192.168.3.22:0/2833179066" rel="noreferrer" target="_blank">192.168.3.22:0/2833179066</a> 2023-02-17T04:47:08.185434+0000<br>> <a href="http://192.168.3.22:0/1812968936" rel="noreferrer" target="_blank">192.168.3.22:0/1812968936</a> 2023-02-17T04:47:08.185434+0000<br>> <a href="http://192.168.3.22:6824/2057987683" rel="noreferrer" target="_blank">192.168.3.22:6824/2057987683</a> 2023-02-17T04:47:08.185434+0000<br>> <a href="http://192.168.3.21:0/2756666482" rel="noreferrer" target="_blank">192.168.3.21:0/2756666482</a> 2023-02-17T05:16:23.939511+0000<br>> <a href="http://192.168.3.21:0/1646520197" rel="noreferrer" target="_blank">192.168.3.21:0/1646520197</a> 2023-02-17T05:16:23.939511+0000<br>> <a href="http://192.168.3.22:6825/2057987683" rel="noreferrer" target="_blank">192.168.3.22:6825/2057987683</a> 2023-02-17T04:47:08.185434+0000<br>> <a href="http://192.168.3.21:0/526748613" rel="noreferrer" target="_blank">192.168.3.21:0/526748613</a> 2023-02-17T05:16:23.939511+0000<br>> <a href="http://192.168.3.21:6815/2454821797" rel="noreferrer" target="_blank">192.168.3.21:6815/2454821797</a> 2023-02-17T05:16:23.939511+0000<br>> <a href="http://192.168.3.22:0/288537807" rel="noreferrer" target="_blank">192.168.3.22:0/288537807</a> 2023-02-17T04:47:08.185434+0000<br>> <a href="http://192.168.3.21:0/4161448504" rel="noreferrer" target="_blank">192.168.3.21:0/4161448504</a> 2023-02-17T05:16:23.939511+0000<br>> <a href="http://192.168.3.21:6824/2454821797" rel="noreferrer" target="_blank">192.168.3.21:6824/2454821797</a> 2023-02-17T05:16:23.939511+0000<br>> listed 13 entries<br>><br>> root@ceph1:~# rbd lock list --image<br>> 55dbf40b-0a6a-4bab-b3a5-b4bb74e963af_disk -p vms<br>> There is 1 exclusive lock on this image.<br>> Locker ID Address<br>> client.268212 auto 139971105131968 <a href="http://192.168.3.12:0/1649312807" rel="noreferrer" target="_blank">192.168.3.12:0/1649312807</a><br>><br>> root@ceph1:~# ceph osd blacklist rm <a href="http://192.168.3.12:0/1649312807" rel="noreferrer" target="_blank">192.168.3.12:0/1649312807</a><br>> <a href="http://192.168.3.12:0/1649312807" rel="noreferrer" target="_blank">192.168.3.12:0/1649312807</a> isn't blocklisted<br>><br>> How do I create a lock?<br>><br>><br>> On Thu, Feb 16, 2023 at 10:45 AM Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br>><br>>> In addition to Sean's response, this has been asked multiple times,<br>>> e.g. [1]. You could check if your hypervisors gave up the lock on the<br>>> RBDs or if they are still locked (rbd status <pool>/<image>), in that<br>>> case you might need to blacklist the clients and see if that resolves<br>>> anything. Do you have regular snapshots (or backups) to be able to<br>>> rollback in case of a curruption?<br>>><br>>> [1] <a href="https://www.spinics.net/lists/ceph-users/msg45937.html" rel="noreferrer" target="_blank">https://www.spinics.net/lists/ceph-users/msg45937.html</a><br>>><br>>><br>>> Zitat von Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>>:<br>>><br>>> > On Thu, 2023-02-16 at 09:56 -0500, Satish Patel wrote:<br>>> >> Folks,<br>>> >><br>>> >> I am running a small 3 node compute/controller with 3 node ceph storage<br>>> in<br>>> >> my lab. Yesterday, because of a power outage all my nodes went down.<br>>> After<br>>> >> reboot of all nodes ceph seems to show good health and no error (in ceph<br>>> >> -s).<br>>> >><br>>> >> When I started using the existing VM I noticed the following errors.<br>>> Seems<br>>> >> like data loss. This is a lab machine and has zero activity on vms but<br>>> >> still loses data and the file system corrupt. Is this normal ?<br>>> > if the vm/cluster hard crashes due to the power cut yes it can.<br>>> > personally i have hit this more often with XFS then ext4 but i have<br>>> > seen it with both.<br>>> >><br>>> >> I am not using eraser coding, does that help in this matter?<br>>> >><br>>> >> blk_update_request: I/O error, dev sda, sector 233000 op 0x1: (WRITE)<br>>> flags<br>>> >> 0x800 phys_seg 8 prio class 0<br>>> ><br>>> > you will proably need to rescue the isntance and repair the<br>>> > filesystem of each vm with fsck<br>>> > or similar. so boot with recue image -> repair filestem -> unrescue<br>>> > -> hardreboot/start vm if needed<br>>> ><br>>> > you might be able to mitigate this somewhat by disableing disk<br>>> > cacheing at teh qemu level but<br>>> > that will reduce performance. ceph recommenes that you use<br>>> > virtio-scis fo the device model and<br>>> > writeback cach mode. we generally recommend that too however you can<br>>> > use the disk_cachemodes option to<br>>> > chage that.<br>>> ><br>>> <a href="https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes</a><br>>> ><br>>> > [libvirt]<br>>> > disk_cachemodes=file=none,block=none,network=none<br>>> ><br>>> > this curreption may also have happend on the cecph cluter side.<br>>> > they have some options that can help prevent that via journaling wirtes<br>>> ><br>>> > if you can afford it i would get even a small UPS to allow a<br>>> > graceful shutdown if you have future powercuts<br>>> > to aovid dataloss issues.<br>>><br>>><br>>><br>>><br>>><br></div><div><br></div><div><br></div><div><br></div></blockquote></div></blockquote></div></blockquote></div></blockquote><div><br></div><div><span></span></div></body></html>