<div dir="ltr">Well... now you mention that "Nova is really smart and will attempt to redownload image from glance if _base part is deleted." maybe that was actually happing, but since the image had been deleted there was no way it could get it back from glance and this would perfectly explain the ERROR message. I really learned a few things today hahaha.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-27 19:07 GMT-03:00 George Shuklin <span dir="ltr"><<a href="mailto:george.shuklin@gmail.com" target="_blank">george.shuklin@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Em... I think it's not bug. May be some kind of inconvenience, and
better error logging is always welcome, but error message is
correct: image is not found. Not in glance, but inside of nova
internals.<br>
<br>
For everyone, who know how nova works it perfectly clear. For new
users is unclear, but this is 'wishlist', not a bug.<br>
<br>
P. S. As far as I hear, Nova is really smart and will attempt to
redownload image from glance if _base part is deleted.<div><div class="h5"><br>
<br>
<br>
On 05/28/2014 12:25 AM, Juan José Pavlik Salles wrote:<br>
<blockquote type="cite">
<div dir="ltr">By the way... there's something we can learn from
this miss understanding. The error message:
<div><br>
</div>
<div>2014-05-27 15:23:45.002 ERROR nova.compute.manager
[req-a76d922e-4aaa-4357-83cb-5e5a1869b5cc
31020076174943bdb7486c330a298d93
d1e3aae242f14c488d2225dcbf1e96d6] [instance:
b17bfae2-27b4-49a4-9d1b-bd739b400347] Cannot reboot instance:
Image 39baad54-6ce1-4f42-b431-1bac4fd6df28 could not be found.<br>
</div>
<div><br>
</div>
<div>is wrong and even useless because it points to the wrong
direction. The problem here wasn't glance image at all, there
was a part of the qcow drive missing. Should I fill a bug with
this problem? I'm on fire today hahaha.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-05-27 18:19 GMT-03:00 Juan José
Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I'm not proud of this... somehow George was
right. Last week we migrated our instances from gfs2
volume to ocfs2 one and we copied "all" the files from one
volume to the other, we mounted the new one and started
the VMs. BUT... it seems that a few files were lost during
the last node failure and the files that were supposed to
be in _base dir weren't there (this is an awkward answer
I'll have to improve before telling my boss about this).
You can see it here:
<div>
<br>
</div>
<div> root@cebolla:/var/lib/nova# ll instances/_base/
instances_17_05_2014/_base/</div>
<div>instances_17_05_2014/_base/:</div>
<div>total 6572308</div>
<div>drwxr-xr-x 2 nova nova 4096 may 17
20:50 ./</div>
<div>drwxr-xr-x 27 root root 4096 may 17
20:57 ../</div>
<div>-rw-r--r-- 1 nova kvm 2147483648 may 17
20:50 1cfaaa19259a9538efb89dd674645af7ad334322</div>
<div>-rw-r--r-- 1 nova kvm 2147483648 may 17
20:50 6a861f8328e7fd0b4bd80bf95dbb7fd2b782e0bd</div>
<div>-rw-r--r-- 1 nova kvm 2147483648 may 17
20:50 99edbbef0de23ac4ed20015ded60887690444661</div>
<div>-rw-r--r-- 1 nova kvm 2147483648 may 17
20:50 d04d963a4efa93ecacaadc272ab841c1dd901c9f</div>
<div>
<div>-rw-r--r-- 1 nova nova 8589934592 nov 18
2013 swap</div>
<div>-rw-r--r-- 1 libvirt-qemu kvm 536870912 nov 15
2013 swap_512</div>
<div><br>
</div>
</div>
<div>instances/_base/:</div>
<div>total 2424832</div>
<div>drwxr-xr-x 2 nova nova 3896 may 27
18:02 ./</div>
<div>
drwxr-xr-x 28 nova nova 3896 may 27 17:45
../</div>
<div>-rw-r--r-- 1 nova nova 2147483648 may 27
17:34 1cfaaa19259a9538efb89dd674645af7ad334322</div>
<div>
<div>-rw-r--r-- 1 nova nova 8589934592 nov 18
2013 swap</div>
<div>-rw-r--r-- 1 libvirt-qemu kvm 536870912 nov 15
2013 swap_512</div>
</div>
<div>root@cebolla:/var/lib/nova#</div>
<div><br>
</div>
<div>Before that I checked that the qcow disk of the
instances were being backed up by a file that didn't
exist at all!!!:</div>
<div><br>
</div>
<div>
<div>root@cebolla:/var/lib/nova/instances/b17bfae2-27b4-49a4-9d1b-bd739b400347#
qemu-img info disk</div>
<div>image: disk</div>
<div>file format: qcow2</div>
<div>virtual size: 10G (10737418240 bytes)</div>
<div>disk size: 2.6G</div>
<div>cluster_size: 65536</div>
<div>backing file: <b>/var/lib/nova/instances/_base/99edbbef0de23ac4ed20015ded60887690444661</b>
(actual path:
/var/lib/nova/instances/_base/99edbbef0de23ac4ed20015ded60887690444661)</div>
<div>root@cebolla:/var/lib/nova/instances/b17bfae2-27b4-49a4-9d1b-bd739b400347#</div>
</div>
<div><br>
</div>
<div>Basically, I copied the missing files from the older
volume (6a861f8328e7fd0b4bd80bf95dbb7fd2b782e0bd,
99edbbef0de23ac4ed20015ded60887690444661 and
d04d963a4efa93ecacaadc272ab841c1dd901c9f) and started
the VMs. Everything is up and running again, sorry about
the incovenients and thanks!!!</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-05-27 17:35 GMT-03:00 Juan
José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span>:
<div>
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">What if I change Image ID in glance
DB for an existing image's ID? As far as I see,
if you delete an image you can't reboot the
instances that were created with that image,
doesn't sound fine. I must be loosing something
here...</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-05-27 16:56
GMT-03:00 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span>:
<div>
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Great, now I understand
that, new thing learned hahah! But
this problem doesn't seem to be
related with the _base files, the log
says it couldn't found the Image file,
that's why I'm confused and don't see
the point. I'll try spying the code a
bit, maybe it's a simple check and
there's no real need of the image
file. </div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-05-27
16:29 GMT-03:00 George Shuklin <span dir="ltr"><<a href="mailto:george.shuklin@gmail.com" target="_blank">george.shuklin@gmail.com</a>></span>:
<div>
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> _base
contains 'base' copy of
disk, if disk is in qcow
format.<br>
<br>
Qcow consists from basic
(unmodified) image and file
with changes. If instance
never write to some area, it
will be read from base copy.
As soon it write something
there, new data will be read
from disk, not from _base.
<div>
<div><br>
<br>
<br>
<div>On 05/27/2014 10:18
PM, Juan José Pavlik
Salles wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi
George, I don't
really understand
the relationship
between _base and
the <span style="font-family:arial,sans-serif;font-size:13px">b17bfae2-27b4-49a4-</span><span style="font-family:arial,sans-serif;font-size:13px">9d1b-bd739b400347 (</span>instance
directory, where the
disks are), this is
what _base contains
<div> <br>
</div>
<div>
<div>root@cebolla:/var/lib/nova/instances#
ll _base/</div>
<div>total 2424832</div>
<div>drwxr-xr-x 2
nova
nova 3896
may 27 15:23 ./</div>
<div>drwxr-xr-x 28
nova
nova 3896
may 27 14:36 ../</div>
<div>-rw-r--r-- 1
nova kvm
2147483648 may
27 15:52
1cfaaa19259a9538efb89dd674645af7ad334322</div>
<div>-rw-r--r-- 1
nova
nova 8589934592
nov 18 2013
swap</div>
<div>-rw-r--r-- 1
libvirt-qemu kvm
536870912 nov
15 2013
swap_512</div>
<div>root@cebolla:/var/lib/nova/instances# </div>
</div>
<div><br>
</div>
<div>And I've
checked glance DB
and
the 39baad54-6ce1-4f42-b431-1bac4fd6df28
register is indeed
marked as deleted
and the file is
gone:</div>
<div><br>
</div>
<div>
<div>root@acelga:/var/lib/glance#
ls images</div>
<div>37a88684-f1d8-472a-8681-65eb047c2100
c94ee2f6-fae5-451c-9633-18c33ec512de
d21dd4db-389c-4f4c-a749-91acc1262652</div>
<div>root@acelga:/var/lib/glance# </div>
</div>
<div><br>
</div>
<div>Is there any
healthy way to
start the
instances without
this lost image?
Do I really need
the image to start
the instances?</div>
<div><br>
</div>
<div>Thanks</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-05-27
15:58 GMT-03:00
George Shuklin <span dir="ltr"><<a href="mailto:george.shuklin@gmail.com" target="_blank">george.shuklin@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
I think nova
checking if
image is in
place and
available to
restore image
_base (if it
missing). But
if _base is
fine, I think
it's strange
to complain
about glance
images...
<div>
<div><br>
<br>
<div>On
05/27/2014
09:32 PM, Juan
José Pavlik
Salles wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">Hi
guys, today we
found out that
one of our
compute nodes
had rebooted
durning the
night, so when
i got to the
office I
started
rebooting the
instances
but... they
never started.
After a quite
a few reboots
I saw the
light at the
end of the
tunnel...
<div> <br>
</div>
<div>
<div>2014-05-27
15:23:45.002
ERROR
nova.compute.manager
[req-a76d922e-4aaa-4357-83cb-5e5a1869b5cc
31020076174943bdb7486c330a298d93
d1e3aae242f14c488d2225dcbf1e96d6]
[instance:
b17bfae2-27b4-49a4-9d1b-bd739b400347]
Cannot reboot
instance:
Image
39baad54-6ce1-4f42-b431-1bac4fd6df28
could not be
found.</div>
<div><br>
</div>
<div>I've got
3 instances
with this same
error, all of
them were
created from
the same
glance image
which is not
longer among
us (replaced
for a new
one). My
question is,
why do the
instance need
the image to
start? The
instance disks
are there</div>
<div><br>
</div>
<div>
root@cebolla:/var/lib/nova#
ll
instances/b17bfae2-27b4-49a4-9d1b-bd739b400347/</div>
<div>total
3233792</div>
<div>drwxr-xr-x
2 nova nova
3896 feb
20 12:49 ./</div>
<div>drwxr-xr-x
28 nova nova
3896 may
27 14:36 ../</div>
<div>-rw-rw----
1 root root
0 may
27 15:23
console.log</div>
<div>-rw-r--r--
1 root root
2773155840 may
24 20:23 disk</div>
<div>-rw-r--r--
1 root root
537198592 may
16 16:14
disk.swap</div>
<div>-rw-r--r--
1 nova nova
1782 may
27 15:23
libvirt.xml</div>
<div>root@cebolla:/var/lib/nova# </div>
<div><br>
</div>
<div>Any ideas
will be more
than
apreciated. </div>
<div><br>
</div>
<div>Thanks
guys!</div>
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik
Salles Juan
José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
OpenStack-operators mailing list
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a>
</pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
OpenStack-operators
mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik
Salles Juan José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div>
<div><br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik Salles Juan
José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div>
<div><br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik Salles Juan José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div>
<div><br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik Salles Juan José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik Salles Juan José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div></div>
</div>