<div dir="ltr">How is the recovery coming along Gökhan? I am curious to hear. </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 12, 2019 at 3:46 AM Gökhan IŞIK <<a href="mailto:skylightcoder@gmail.com">skylightcoder@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Awesome, thanks! Donny,</div><div>I followed below steps and rescue my instance.<br></div><div><ol style="margin:10px 0px 0px;list-style-type:decimal;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><li><p style="margin:0px"> Find instance id and compute host<br></p><table class="gmail-m_-3500946902272678134gmail-wysiwyg-macro" style="background-color:rgb(240,240,240);background-position:0px 0px;background-repeat:no-repeat;border:1px solid rgb(221,221,221);margin-top:10px;padding:24px 2px 2px;width:1918.23px;border-collapse:separate;background-image:url("")"><tbody><tr><td class="gmail-m_-3500946902272678134gmail-wysiwyg-macro-body" style="white-space:pre-wrap;background-color:rgb(255,255,255);border:1px solid rgb(221,221,221);margin:0px;padding:10px"><pre style="margin:0px;white-space:pre-wrap">root@infra1-utility-container-50bcf920:~# openstack server show 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e -c id -c OS-EXT-SRV-ATTR:hypervisor_hostname
+-------------------------------------+--------------------------------------+
| Field | Value |
+-------------------------------------+--------------------------------------+
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute06 |
| id | 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e |
+-------------------------------------+--------------------------------------+
<br></pre></td></tr></tbody></table></li><li><p style="margin:0px"> Find image and backing image file on compute host</p><table class="gmail-m_-3500946902272678134gmail-wysiwyg-macro" style="background-color:rgb(240,240,240);background-position:0px 0px;background-repeat:no-repeat;border:1px solid rgb(221,221,221);margin-top:10px;padding:24px 2px 2px;width:1918.23px;border-collapse:separate;background-image:url("")"><tbody><tr><td class="gmail-m_-3500946902272678134gmail-wysiwyg-macro-body" style="white-space:pre-wrap;background-color:rgb(255,255,255);border:1px solid rgb(221,221,221);margin:0px;padding:10px"><pre style="margin:0px;white-space:pre-wrap">root@compute06:~# qemu-img info -U --backing-chain /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk
image: /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk
file format: qcow2
virtual size: 160G (171798691840 bytes)
disk size: 32G
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
image: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323
file format: raw
virtual size: 160G (171798691840 bytes)
disk size: 18G
</pre></td></tr></tbody></table><p style="margin:10px 0px 0px"><br></p></li><li>Copy image and backing image file <br><p style="margin:0px"><br></p><table class="gmail-m_-3500946902272678134gmail-wysiwyg-macro" style="background-color:rgb(240,240,240);background-position:0px 0px;background-repeat:no-repeat;border:1px solid rgb(221,221,221);margin-top:10px;padding:24px 2px 2px;width:1918.23px;border-collapse:separate;background-image:url("")"><tbody><tr><td class="gmail-m_-3500946902272678134gmail-wysiwyg-macro-body" style="white-space:pre-wrap;background-color:rgb(255,255,255);border:1px solid rgb(221,221,221);margin:0px;padding:10px"><pre style="margin:0px;white-space:pre-wrap">root@compute06:~# cp /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk master
root@compute06:~# cp /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 new-master</pre></td></tr></tbody></table><p style="margin:10px 0px 0px"><br></p></li><li><p style="margin:0px"> <span style="color:rgba(0,0,0,0.84);font-family:medium-content-serif-font,Georgia,Cambria,"Times New Roman",Times,serif;font-size:21px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.084px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">Rebase the image file that was backed off the original file so that it uses the new file i.e new-master then <span style="color:rgba(0,0,0,0.84);font-family:medium-content-serif-font,Georgia,Cambria,"Times New Roman",Times,serif;font-size:21px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.084px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">commit those changes back to original file<span> </span></span><code class="gmail-m_-3500946902272678134gmail-iq gmail-m_-3500946902272678134gmail-ir gmail-m_-3500946902272678134gmail-is gmail-m_-3500946902272678134gmail-ih gmail-m_-3500946902272678134gmail-b" style="box-sizing:inherit;font-weight:400;font-family:Menlo,Monaco,"Courier New",Courier,monospace;background-color:rgba(0,0,0,0.05);padding:2px 4px;font-size:15.75px;color:rgba(0,0,0,0.84);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:-0.084px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">master</code><span style="color:rgba(0,0,0,0.84);font-family:medium-content-serif-font,Georgia,Cambria,"Times New Roman",Times,serif;font-size:21px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.084px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none"><span> </span>back into the new base<span> </span></span><code class="gmail-m_-3500946902272678134gmail-iq gmail-m_-3500946902272678134gmail-ir gmail-m_-3500946902272678134gmail-is gmail-m_-3500946902272678134gmail-ih gmail-m_-3500946902272678134gmail-b" style="box-sizing:inherit;font-weight:400;font-family:Menlo,Monaco,"Courier New",Courier,monospace;background-color:rgba(0,0,0,0.05);padding:2px 4px;font-size:15.75px;color:rgba(0,0,0,0.84);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:-0.084px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">new-master</code></span></p><table class="gmail-m_-3500946902272678134gmail-wysiwyg-macro" style="background-color:rgb(240,240,240);background-position:0px 0px;background-repeat:no-repeat;border:1px solid rgb(221,221,221);margin-top:10px;padding:24px 2px 2px;width:1918.23px;border-collapse:separate;background-image:url("")"><tbody><tr><td class="gmail-m_-3500946902272678134gmail-wysiwyg-macro-body" style="white-space:pre-wrap;background-color:rgb(255,255,255);border:1px solid rgb(221,221,221);margin:0px;padding:10px"><pre style="margin:0px;white-space:pre-wrap">root@compute06:~# qemu-img rebase -b new-master -U master</pre><pre style="margin:0px;white-space:pre-wrap">root@compute06:~# qemu-img commit master</pre><pre style="margin:0px;white-space:pre-wrap">root@compute06:~# qemu-img info new-master</pre><pre style="margin:0px;white-space:pre-wrap"><br></pre><pre style="margin:0px;white-space:pre-wrap"><br></pre></td></tr></tbody></table><p style="margin:10px 0px 0px"><br></p></li><li><p style="margin:0px"> Convert raw image to qcow2 <br></p><table class="gmail-m_-3500946902272678134gmail-wysiwyg-macro" style="background-color:rgb(240,240,240);background-position:0px 0px;background-repeat:no-repeat;border:1px solid rgb(221,221,221);margin-top:10px;padding:24px 2px 2px;width:1918.23px;border-collapse:separate;background-image:url("")"><tbody><tr><td class="gmail-m_-3500946902272678134gmail-wysiwyg-macro-body" style="white-space:pre-wrap;background-color:rgb(255,255,255);border:1px solid rgb(221,221,221);margin:0px;padding:10px"><pre style="margin:0px;white-space:pre-wrap">root@compute06:~# qemu-img convert -f raw -O qcow2 new-master new-master.qcow2</pre></td></tr></tbody></table><p style="margin:10px 0px 0px"><br></p></li><li> Time to upload glance and then launch instance from this image :)</li></ol><div><br></div><div>Thanks,</div><div>Gökhan.<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Donny Davis <<a href="mailto:donny@fortnebula.com" target="_blank">donny@fortnebula.com</a>>, 12 Tem 2019 Cum, 00:56 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Of course you can also always just pull the disk images from the vm folders, merge them back with the base file, upload to glance and then relaunch the instances. <div><br></div><div><div>You can give this method a spin with the lowest risk to your instances</div><div><br></div><div><a href="https://medium.com/@kumar_pravin/qemu-merge-snapshot-and-backing-file-into-standalone-disk-c8d3a2b17c0e" target="_blank">https://medium.com/@kumar_pravin/qemu-merge-snapshot-and-backing-file-into-standalone-disk-c8d3a2b17c0e</a><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 11, 2019 at 4:10 PM Donny Davis <<a href="mailto:donny@fortnebula.com" target="_blank">donny@fortnebula.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You surely want to leave locking turned on. <div><br></div><div>You may want to ask qemu-devel about the locking of a image file and how it works. This isn't really an Openstack issue, seems to be a layer below. <br></div><div><br></div><div>Depending on how mission critical your VM's are, you could probably work around it by just passing in --force-share into the command openstack is trying to run. </div><div><br></div><div>I cannot recommend this path, the best way is to find out how you remove the lock. </div><div><br></div><div><br></div><br class="gmail-m_-3500946902272678134gmail-m_-5326723840756313392gmail-m_-6764668941712628219gmail-Apple-interchange-newline"><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 11, 2019 at 3:23 PM Gökhan IŞIK <<a href="mailto:skylightcoder@gmail.com" target="_blank">skylightcoder@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In [1] it says "<span style="color:rgb(37,37,37);font-family:sans-serif;font-size:14px">Image locking is added and enabled by default. Multiple QEMU processes cannot write to the same image as long as the host supports OFD or posix locking, unless options are specified otherwise." May be need to do something on nova side. </span><div><font face="sans-serif" color="#252525"><span style="font-size:14px"><br></span></font><div>I run this command and get same error. Output is in
<a href="http://paste.openstack.org/show/754311/" target="_blank">http://paste.openstack.org/show/754311/</a></div></div><div><br></div><div>İf I run qemu-img info instance-0000219b with -U , it doesn't give any errors.</div><div><br></div><div>[1] <a href="https://wiki.qemu.org/ChangeLog/2.10" target="_blank">https://wiki.qemu.org/ChangeLog/2.10</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Donny Davis <<a href="mailto:donny@fortnebula.com" target="_blank">donny@fortnebula.com</a>>, 11 Tem 2019 Per, 22:11 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Well that is interesting. If you look in your libvirt config directory (/etc/libvirt on Centos) you can get a little more info on what is being used for locking. <div><br></div><div>Maybe strace can shed some light on it. Try something like </div><div><br></div><div>strace -ttt -f qemu-img info /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk<br></div><div><br></div><div><br></div><div><br><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK <<a href="mailto:skylightcoder@gmail.com" target="_blank">skylightcoder@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I run virsh list --all command and output is below:<div><br><div>root@compute06:~# virsh list --all<br> Id Name State<br>----------------------------------------------------<br> - instance-000012f9 shut off<br> - instance-000013b6 shut off<br> - instance-000016fb shut off<br> - instance-0000190a shut off<br> - instance-00001a8a shut off<br> - instance-00001e05 shut off<br> - instance-0000202a shut off<br> - instance-00002135 shut off<br> - instance-00002141 shut off<br> - instance-000021b6 shut off<br> - instance-000021ec shut off<br> - instance-000023db shut off<br> - instance-00002ad7 shut off<br></div><div><br></div><div>And also when I try start instances with virsh , output is below:</div><div><br></div><div>root@compute06:~# virsh start instance-0000219b<br>error: Failed to start domain
instance-000012f9 <br>error: internal error: process exited while connecting to monitor: 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device redirected to /dev/pts/3 (label charserial0)<br>2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: Failed to get "write" lock<br>Is another process using the image?<br></div><div><br></div><div>Thanks,</div><div>Gökhan</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Donny Davis <<a href="mailto:donny@fortnebula.com" target="_blank">donny@fortnebula.com</a>>, 11 Tem 2019 Per, 21:06 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Can you ssh to the hypervisor and run virsh list to make sure your instances are in fact down?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK <<a href="mailto:skylightcoder@gmail.com" target="_blank">skylightcoder@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Can anyone help me please ? I can no't rescue my instances yet :(</div><div><br></div><div><div>Thanks,</div><div>Gökhan</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Gökhan IŞIK <<a href="mailto:skylightcoder@gmail.com" target="_blank">skylightcoder@gmail.com</a>>, 9 Tem 2019 Sal, 15:46 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi folks,</div><div> Because of power outage, Most of our compute
nodes unexpectedly shut down and now I can not start our instances. Error message is "Failed to get "write" lock another process using the image?".
Instances Power status is No State. Full error log is <a href="http://paste.openstack.org/show/754107/" rel="nofollow" target="_blank">http://paste.openstack.org/show/754107/</a>.
My environment is OpenStack Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs shared
storage. Nova version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is 3.6.0. I saw a commit [1], but it doesn't solve this problem.</div><div>There are important instances
on my environment. How can I rescue my instances? What would you suggest
?</div><div><br></div><div>Thanks,</div><div>Gökhan</div><div><br></div><div>[1] <a href="https://review.opendev.org/#/c/509774/" target="_blank">https://review.opendev.org/#/c/509774/</a></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>