<div dir="ltr">By the way... there's something we can learn from this miss understanding. The error message:<div><br></div><div>2014-05-27 15:23:45.002 ERROR nova.compute.manager [req-a76d922e-4aaa-4357-83cb-5e5a1869b5cc 31020076174943bdb7486c330a298d93 d1e3aae242f14c488d2225dcbf1e96d6] [instance: b17bfae2-27b4-49a4-9d1b-bd739b400347] Cannot reboot instance: Image 39baad54-6ce1-4f42-b431-1bac4fd6df28 could not be found.<br>
</div><div><br></div><div>is wrong and even useless because it points to the wrong direction. The problem here wasn't glance image at all, there was a part of the qcow drive missing. Should I fill a bug with this problem? I'm on fire today hahaha.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-27 18:19 GMT-03:00 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm not proud of this... somehow George was right. Last week we migrated our instances from gfs2 volume to ocfs2 one and we copied "all" the files from one volume to the other, we mounted the new one and started the VMs. BUT... it seems that a few files were lost during the last node failure and the files that were supposed to be in _base dir weren't there (this is an awkward answer I'll have to improve before telling my boss about this). You can see it here:<div>
<br></div><div> root@cebolla:/var/lib/nova# ll instances/_base/ instances_17_05_2014/_base/</div><div>instances_17_05_2014/_base/:</div><div>total 6572308</div><div>drwxr-xr-x 2 nova nova 4096 may 17 20:50 ./</div>
<div>drwxr-xr-x 27 root root 4096 may 17 20:57 ../</div><div>-rw-r--r-- 1 nova kvm 2147483648 may 17 20:50 1cfaaa19259a9538efb89dd674645af7ad334322</div><div>-rw-r--r-- 1 nova kvm 2147483648 may 17 20:50 6a861f8328e7fd0b4bd80bf95dbb7fd2b782e0bd</div>
<div>-rw-r--r-- 1 nova kvm 2147483648 may 17 20:50 99edbbef0de23ac4ed20015ded60887690444661</div><div>-rw-r--r-- 1 nova kvm 2147483648 may 17 20:50 d04d963a4efa93ecacaadc272ab841c1dd901c9f</div><div class="">
<div>-rw-r--r-- 1 nova nova 8589934592 nov 18 2013 swap</div>
<div>-rw-r--r-- 1 libvirt-qemu kvm 536870912 nov 15 2013 swap_512</div><div><br></div></div><div>instances/_base/:</div><div>total 2424832</div><div>drwxr-xr-x 2 nova nova 3896 may 27 18:02 ./</div><div>
drwxr-xr-x 28 nova nova 3896 may 27 17:45 ../</div>
<div>-rw-r--r-- 1 nova nova 2147483648 may 27 17:34 1cfaaa19259a9538efb89dd674645af7ad334322</div><div class=""><div>-rw-r--r-- 1 nova nova 8589934592 nov 18 2013 swap</div><div>-rw-r--r-- 1 libvirt-qemu kvm 536870912 nov 15 2013 swap_512</div>
</div><div>root@cebolla:/var/lib/nova#</div><div><br></div><div>Before that I checked that the qcow disk of the instances were being backed up by a file that didn't exist at all!!!:</div><div><br></div><div><div>root@cebolla:/var/lib/nova/instances/b17bfae2-27b4-49a4-9d1b-bd739b400347# qemu-img info disk</div>
<div>image: disk</div><div>file format: qcow2</div><div>virtual size: 10G (10737418240 bytes)</div><div>disk size: 2.6G</div><div>cluster_size: 65536</div><div>backing file: <b>/var/lib/nova/instances/_base/99edbbef0de23ac4ed20015ded60887690444661</b> (actual path: /var/lib/nova/instances/_base/99edbbef0de23ac4ed20015ded60887690444661)</div>
<div>root@cebolla:/var/lib/nova/instances/b17bfae2-27b4-49a4-9d1b-bd739b400347#</div></div><div><br></div><div>Basically, I copied the missing files from the older volume (6a861f8328e7fd0b4bd80bf95dbb7fd2b782e0bd, 99edbbef0de23ac4ed20015ded60887690444661 and d04d963a4efa93ecacaadc272ab841c1dd901c9f) and started the VMs. Everything is up and running again, sorry about the incovenients and thanks!!!</div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-27 17:35 GMT-03:00 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span>:<div>
<div class="h5"><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">What if I change Image ID in glance DB for an existing image's ID? As far as I see, if you delete an image you can't reboot the instances that were created with that image, doesn't sound fine. I must be loosing something here...</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-27 16:56 GMT-03:00 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span>:<div>
<div><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Great, now I understand that, new thing learned hahah! But this problem doesn't seem to be related with the _base files, the log says it couldn't found the Image file, that's why I'm confused and don't see the point. I'll try spying the code a bit, maybe it's a simple check and there's no real need of the image file. </div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-27 16:29 GMT-03:00 George Shuklin <span dir="ltr"><<a href="mailto:george.shuklin@gmail.com" target="_blank">george.shuklin@gmail.com</a>></span>:<div>
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
_base contains 'base' copy of disk, if disk is in qcow format.<br>
<br>
Qcow consists from basic (unmodified) image and file with changes.
If instance never write to some area, it will be read from base
copy. As soon it write something there, new data will be read from
disk, not from _base.<div><div><br>
<br>
<br>
<div>On 05/27/2014 10:18 PM, Juan José
Pavlik Salles wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi George, I don't really understand the
relationship between _base and the <span style="font-family:arial,sans-serif;font-size:13px">b17bfae2-27b4-49a4-</span><span style="font-family:arial,sans-serif;font-size:13px">9d1b-bd739b400347
(</span>instance directory, where the disks are), this is what
_base contains
<div>
<br>
</div>
<div>
<div>root@cebolla:/var/lib/nova/instances# ll _base/</div>
<div>total 2424832</div>
<div>drwxr-xr-x 2 nova nova 3896 may 27 15:23
./</div>
<div>drwxr-xr-x 28 nova nova 3896 may 27 14:36
../</div>
<div>-rw-r--r-- 1 nova kvm 2147483648 may 27 15:52
1cfaaa19259a9538efb89dd674645af7ad334322</div>
<div>-rw-r--r-- 1 nova nova 8589934592 nov 18 2013
swap</div>
<div>-rw-r--r-- 1 libvirt-qemu kvm 536870912 nov 15 2013
swap_512</div>
<div>root@cebolla:/var/lib/nova/instances# </div>
</div>
<div><br>
</div>
<div>And I've checked glance DB and
the 39baad54-6ce1-4f42-b431-1bac4fd6df28 register is indeed
marked as deleted and the file is gone:</div>
<div><br>
</div>
<div>
<div>root@acelga:/var/lib/glance# ls images</div>
<div>37a88684-f1d8-472a-8681-65eb047c2100
c94ee2f6-fae5-451c-9633-18c33ec512de
d21dd4db-389c-4f4c-a749-91acc1262652</div>
<div>root@acelga:/var/lib/glance# </div>
</div>
<div><br>
</div>
<div>Is there any healthy way to start the instances without
this lost image? Do I really need the image to start the
instances?</div>
<div><br>
</div>
<div>Thanks</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-05-27 15:58 GMT-03:00 George
Shuklin <span dir="ltr"><<a href="mailto:george.shuklin@gmail.com" target="_blank">george.shuklin@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> I think nova checking
if image is in place and available to restore image _base
(if it missing). But if _base is fine, I think it's
strange to complain about glance images...
<div>
<div><br>
<br>
<div>On 05/27/2014 09:32 PM, Juan José Pavlik Salles
wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">Hi guys, today we found out that one
of our compute nodes had rebooted durning the
night, so when i got to the office I started
rebooting the instances but... they never started.
After a quite a few reboots I saw the light at the
end of the tunnel...
<div> <br>
</div>
<div>
<div>2014-05-27 15:23:45.002 ERROR
nova.compute.manager
[req-a76d922e-4aaa-4357-83cb-5e5a1869b5cc
31020076174943bdb7486c330a298d93
d1e3aae242f14c488d2225dcbf1e96d6] [instance:
b17bfae2-27b4-49a4-9d1b-bd739b400347] Cannot
reboot instance: Image
39baad54-6ce1-4f42-b431-1bac4fd6df28 could not
be found.</div>
<div><br>
</div>
<div>I've got 3 instances with this same error,
all of them were created from the same glance
image which is not longer among us (replaced
for a new one). My question is, why do the
instance need the image to start? The instance
disks are there</div>
<div><br>
</div>
<div> root@cebolla:/var/lib/nova# ll
instances/b17bfae2-27b4-49a4-9d1b-bd739b400347/</div>
<div>total 3233792</div>
<div>drwxr-xr-x 2 nova nova 3896 feb 20
12:49 ./</div>
<div>drwxr-xr-x 28 nova nova 3896 may 27
14:36 ../</div>
<div>-rw-rw---- 1 root root 0 may 27
15:23 console.log</div>
<div>-rw-r--r-- 1 root root 2773155840 may 24
20:23 disk</div>
<div>-rw-r--r-- 1 root root 537198592 may 16
16:14 disk.swap</div>
<div>-rw-r--r-- 1 nova nova 1782 may 27
15:23 libvirt.xml</div>
<div>root@cebolla:/var/lib/nova# </div>
<div><br>
</div>
<div>Any ideas will be more than apreciated. </div>
<div><br>
</div>
<div>Thanks guys!</div>
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik Salles Juan José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
OpenStack-operators mailing list
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a>
</pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik Salles Juan José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div></div></div><div><div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div></div></div>
</blockquote></div></div></div><div><div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div></div></div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div></div>
</div>