<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div></div><div>Did you set a lock_path in the neutron’s config?</div><div><br>On Jul 3, 2018, at 17:34, Torin Woltjer <<a href="mailto:torin.woltjer@granddial.com">torin.woltjer@granddial.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div style="font-family: arial; font-size: 12px;"><div data-empty="true">The following errors appear in the neutron-linuxbridge-agent.log on both controllers: <a target="_blank" href="http://paste.openstack.org/show/724930/">http://paste.openstack.org/show/724930/</a></div><div data-empty="true"><br></div><div data-empty="true">No such errors are on the compute nodes themselves.</div><div data-empty="true"><br></div><div contenteditable="false"><div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong><span class="ClickablePhoneNumber"><span class="ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></strong></em></span></div></div></div></div></div><div data-empty="true"><br></div><hr id="previousmessagehr"><div><span><strong>From</strong>: "Torin Woltjer" <<a href="mailto:torin.woltjer@granddial.com">torin.woltjer@granddial.com</a>><br><strong>Sent</strong>: 7/3/18 5:14 PM<br><strong>To</strong>: <<a href="mailto:lmihaiescu@gmail.com">lmihaiescu@gmail.com</a>><br><strong>Cc</strong>: "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>>, "<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>" <<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div style="font-family: arial; font-size: 12px;"><div data-empty="true">Running `openstack server reboot` on an instance just causes the instance to be stuck in a rebooting status. Most notable of the logs is neutron-server.log which shows the following:</div><div data-empty="true"><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a target="_blank" href="http://paste.openstack.org/show/724917/">http://paste.openstack.org/show/724917/</a></div><div data-empty="true"><br></div><div data-empty="true">I realized that rabbitmq was in a failed state, so I bootstrapped it, rebooted controllers, and all of the agents show online.</div><div data-empty="true"><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a target="_blank" href="http://paste.openstack.org/show/724921/">http://paste.openstack.org/show/724921/</a></div><div data-empty="true">And all of the instances can be properly started, however I cannot ping any of the instances floating IPs or the neutron router. And when logging into an instance with the console, there is no IP address on any interface.</div><div data-empty="true"><br></div><div><div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong><span class="ClickablePhoneNumber"><span class="ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></strong></em></span></div></div></div></div></div><div data-empty="true"><br></div><hr id="previousmessagehr"><div><span><strong>From</strong>: George Mihaiescu <<a href="mailto:lmihaiescu@gmail.com">lmihaiescu@gmail.com</a>><br><strong>Sent</strong>: 7/3/18 11:50 AM<br><strong>To</strong>: <a href="mailto:torin.woltjer@granddial.com">torin.woltjer@granddial.com</a><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div dir="ltr">Try restarting them using "openstack server reboot" and also check the nova-compute.log and neutron agents logs on the compute nodes.</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 3, 2018 at 11:28 AM, Torin Woltjer <span dir="ltr"><<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="font-family:arial;font-size:12px;"><div>We just suffered a power outage in out data center and I'm having trouble recovering the Openstack cluster. All of the nodes are back online, every instance shows active but `virsh list --all` on the compute nodes show that all of the VMs are actually shut down. Running `ip addr` on any of the nodes shows that none of the bridges are present and `ip netns` shows that all of the network namespaces are missing as well. So despite all of the neutron service running, none of the networking appears to be active, which is concerning. How do I solve this without recreating all of the networks?</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_-5169717044002267889ClickablePhoneNumber"><span class="m_-5169717044002267889ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></strong></em></span></div></div></div></div></div></div><br>______________________________<wbr>_________________<br>Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a><br>Post to     : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a><br><br></blockquote></div><br></div></div></div>
</div></blockquote></body></html>