Awesome. Thx for sharing!<br><br><div class="gmail_quote"><div dir="ltr">On Wed, Jun 15, 2016, 11:26 AM Eugen Block <<a href="mailto:eblock@nde.ag">eblock@nde.ag</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
this is no question for you guys, it's more of an information what I<br>
was facing today and I wanted to share it if any of you should be<br>
facing the same or similar issue.<br>
<br>
Due to an incautious upload of several huge images to glance (backed<br>
by Ceph) my colleague crashed our Ceph cluster, the OSDs were full or<br>
near full, so the glance images got stuck in status "saving". But the<br>
more severe impact was that the already running instances with their<br>
root disk in Ceph were not reachable anymore, nova-compute also shut<br>
down.<br>
So we had to get Ceph to a healthy state first, we added some OSDs,<br>
the cluster recovered successfully. Now I wanted to boot the instances<br>
again, basically that worked. Only the instances had no network<br>
interface although Horizon still showed the IP address information in<br>
the dashboard, so I assumed the information still had to be somewhere,<br>
but virsh dumpxml also didn't contain any interface data. A reboot of<br>
my control and compute nodes didn't fix anything, although new<br>
instances were created successfully, even in the respective network<br>
the troublesome instances were supposed to boot in.<br>
<br>
I took a look into the port-list, the ports were still there. So I<br>
tried detaching and reattaching the instance's interface. Detaching<br>
seemed to work, but a refresh of the page or a "nova interface-list<br>
<INSTACE>" still showed the same IP address. As a workaround I<br>
attached a new interface in the same subnet temporarily, detached it<br>
again and then I saw what I wanted to see, NO IP address. Then I<br>
reattached the original port via nova CLI (Horizon doesn't let you<br>
choose a specific port, at least I didn't find it yet), and then my<br>
instances had back their interfaces. What I was also facing was that<br>
some ports seemed to be available, but when I wanted to attach them, I<br>
got an error message saying that port didn't exist. So I created new<br>
ports with the same IP addresses like the original ports, attached<br>
them and now I have a working cloud and a healthy cluster again.<br>
<br>
It took me a couple of hours to figure that out, I hope this helps<br>
anyone! I think it's better than rebuild your instances... ;-)<br>
<br>
Best regards,<br>
Eugen<br>
<br>
--<br>
Eugen Block voice : +49-40-559 51 75<br>
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77<br>
Postfach 61 03 15<br>
D-22423 Hamburg e-mail : <a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a><br>
<br>
Vorsitzende des Aufsichtsrates: Angelika Mozdzen<br>
Sitz und Registergericht: Hamburg, HRB 90934<br>
Vorstand: Jens-U. Mozdzen<br>
USt-IdNr. DE 814 013 983<br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</blockquote></div>