<div style="font-family: arial; font-size: 12px;"><div data-empty="true">If I run `ip netns exec qrouter netstat -lnp` or `ip netns exec qdhcp netstat -lnp` on the controller, should I see anything listening on the metadata port (8775)? When I run these commands I don't see that listening, but I have no example of a working system to check against. Can anybody verify this?</div><div data-empty="true"><br></div><div data-empty="true">Thanks,</div><div data-empty="true"><br></div><div data-empty="true"><br></div><div contenteditable="false"><div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong><span class="ClickablePhoneNumber"><span class="ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><strong><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div data-empty="true"><br></div><hr id="previousmessagehr"><div><span><strong>From</strong>: "Torin Woltjer" <torin.woltjer@granddial.com><br><strong>Sent</strong>: 7/10/18 2:58 PM<br><strong>To</strong>: <torin.woltjer@granddial.com><br><strong>Cc</strong>: <openstack-operators@lists.openstack.org>, <openstack@lists.openstack.org><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div style="font-family: arial; font-size: 12px;"><div data-empty="true">DHCP is working again so instances are getting their addresses. For some reason cloud-init isn't working correctly. Hostnames aren't getting set, and SSH key pair isn't getting set. The neutron-metadata service is in control of this?</div><div data-empty="true"><br></div><div data-empty="true">neutron-metadata-agent.log:</div><div data-empty="true">2018-07-10 08:01:42.046 5518 INFO eventlet.wsgi.server [-] 109.73.185.195,<local> "GET / HTTP/1.1" status: 404  len: 195 time: 0.0622332<br>2018-07-10 09:49:42.604 5518 INFO eventlet.wsgi.server [-] 197.149.85.150,<local> "GET / HTTP/1.1" status: 404  len: 195 time: 0.0645461<br>2018-07-10 10:52:50.845 5517 INFO eventlet.wsgi.server [-] 88.249.225.204,<local> "GET / HTTP/1.1" status: 404  len: 195 time: 0.0659041<br>2018-07-10 11:43:20.471 5518 INFO eventlet.wsgi.server [-] 143.208.186.168,<local> "GET / HTTP/1.1" status: 404  len: 195 time: 0.0618532<br>2018-07-10 11:53:15.574 5511 INFO eventlet.wsgi.server [-] 194.40.240.254,<local> "GET / HTTP/1.1" status: 404  len: 195 time: 0.0636070<br>2018-07-10 13:26:46.795 5518 INFO eventlet.wsgi.server [-] 109.73.177.149,<local> "GET / HTTP/1.1" status: 404  len: 195 time: 0.0611560<br>2018-07-10 13:27:38.795 5513 INFO eventlet.wsgi.server [-] 125.167.69.238,<local> "GET / HTTP/1.0" status: 404  len: 195 time: 0.0631371<br>2018-07-10 13:30:49.551 5514 INFO eventlet.wsgi.server [-] 155.93.152.111,<local> "GET / HTTP/1.0" status: 404  len: 195 time: 0.0609179<br>2018-07-10 14:12:42.008 5521 INFO eventlet.wsgi.server [-] 190.85.38.173,<local> "GET / HTTP/1.1" status: 404  len: 195 time: 0.0597739</div><div data-empty="true"><br></div><div data-empty="true">No other log files show abnormal behavior.</div><div data-empty="true"><br></div><div><div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong><span class="ClickablePhoneNumber"><span class="ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div data-empty="true"><br></div><hr id="previousmessagehr"><div><span><strong>From</strong>: "Torin Woltjer" <torin.woltjer@granddial.com><br><strong>Sent</strong>: 7/6/18 2:33 PM<br><strong>To</strong>: "lmihaiescu@gmail.com" <lmihaiescu@gmail.com><br><strong>Cc</strong>: openstack-operators@lists.openstack.org, openstack@lists.openstack.org<br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div style="font-family: arial; font-size: 12px;"><div data-empty="true">I explored creating a second "selfservice" vxlan to see if DHCP would work on it as it does on my external "provider" network. The new vxlan network shares the same problems as the old vxlan network. Am I having problems with VXLAN in particular? </div><div data-empty="true"><br></div><div><div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong><span class="ClickablePhoneNumber"><span class="ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div data-empty="true"><br></div><hr id="previousmessagehr"><div><span><strong>From</strong>: "Torin Woltjer" <torin.woltjer@granddial.com><br><strong>Sent</strong>: 7/6/18 12:05 PM<br><strong>To</strong>: <lmihaiescu@gmail.com><br><strong>Cc</strong>: openstack-operators@lists.openstack.org, openstack@lists.openstack.org<br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div style="font-family: arial; font-size: 12px;"><div data-empty="true">Interestingly, I can ping the neutron router at 172.16.1.1 just fine, but DHCP (located at 172.16.1.2 and 172.16.1.3) fails. The instance that I manually added the IP address to has a floating IP, and oddly enough I am able to ping DHCP on the provider network, which suggests that DHCP may be working on other networks but not on my selfservice network. I was able to confirm this by creating a new virtual machine directly on the provider network, I was able to ping to it and SSH into it right off of the bat, as it obtained the proper address on its own. "/var/lib/neutron/dhcp/d85c2a00-a637-4109-83f0-<wbr>7c2949be4cad/leases" is empty. "/var/lib/neutron/dhcp/d85c2a00-a637-4109-83f0-<wbr>7c2949be4cad/leases" contains:</div><div data-empty="true">fa:16:3e:3f:94:17,host-172-16-1-8.openstacklocal,172.16.1.8<br>fa:16:3e:e0:57:e7,host-172-16-1-7.openstacklocal,172.16.1.7<br>fa:16:3e:db:a7:cb,host-172-16-1-12.openstacklocal,172.16.1.12<br>fa:16:3e:f8:10:99,host-172-16-1-10.openstacklocal,172.16.1.10<br>fa:16:3e:a7:82:4c,host-172-16-1-3.openstacklocal,172.16.1.3<br>fa:16:3e:f8:23:1d,host-172-16-1-14.openstacklocal,172.16.1.14<br>fa:16:3e:63:53:a4,host-172-16-1-1.openstacklocal,172.16.1.1<br>fa:16:3e:b7:41:a8,host-172-16-1-2.openstacklocal,172.16.1.2<br>fa:16:3e:5e:25:5f,host-172-16-1-4.openstacklocal,172.16.1.4<br>fa:16:3e:3a:a2:53,host-172-16-1-100.openstacklocal,172.16.1.100<br>fa:16:3e:46:39:e2,host-172-16-1-13.openstacklocal,172.16.1.13<br>fa:16:3e:06:de:e0,host-172-16-1-18.openstacklocal,172.16.1.18</div><div data-empty="true"><br></div><div data-empty="true">I've done system restarts since the power outage and the agent hasn't corrected itself. I've restarted all neutron services as I've done things, I could also try stopping and starting dnsmasq.</div><div data-empty="true"><br></div><div><div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong><span class="ClickablePhoneNumber"><span class="ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div data-empty="true"><br></div><hr id="previousmessagehr"><div><span><strong>From</strong>: George Mihaiescu <lmihaiescu@gmail.com><br><strong>Sent</strong>: 7/6/18 11:15 AM<br><strong>To</strong>: torin.woltjer@granddial.com<br><strong>Cc</strong>: "openstack@lists.openstack.org" <openstack@lists.openstack.org>, "openstack-operators@lists.openstack.org" <openstack-operators@lists.openstack.org>, pgsousa@gmail.com<br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div dir="ltr"><div>Can you manually assign an IP address to a VM and once inside, ping the address of the dhcp server?</div><div>That would confirm if there is connectivity at least.</div><div><br></div><div><br></div><div>Also, on the controller node where the dhcp server for that network is, check the "/var/lib/neutron/dhcp/d85c2a00-a637-4109-83f0-<wbr>7c2949be4cad/leases" and make sure there are entries corresponding to your instances.</div><div><br></div><div>In my experience, if neutron is broken after working fine (so excluding any miss-configuration), then an agent is out-of-sync and restart usually fixes things.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 6, 2018 at 9:38 AM, Torin Woltjer <span dir="ltr"><<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="font-family:arial;font-size:12px;"><div>I have done tcpdumps on both the controllers and on a compute node.</div><div>Controller:</div><div>`ip netns exec qdhcp-d85c2a00-a637-4109-83f0-<wbr>7c2949be4cad tcpdump -vnes0 -i ns-83d68c76-b8 port 67`</div><div>`tcpdump -vnes0 -i any port 67`</div><div>Compute:</div><div>`tcpdump -vnes0 -i brqd85c2a00-a6 port 68`</div><div><br></div><div>For the first command on the controller, there are no packets captured at all. The second command on the controller captures packets, but they don't appear to be relevant to openstack. The dump from the compute node shows constant requests are getting sent by openstack instances. </div><div><br></div><div>In summary; DHCP requests are being sent, but are never received.</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249ClickablePhoneNumber"><span class="m_5383920506812206249ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div><br></div><hr id="m_5383920506812206249previousmessagehr"><div><span><strong>From</strong>: George Mihaiescu <<a href="mailto:lmihaiescu@gmail.com" target="_blank">lmihaiescu@gmail.com</a>><br><strong>Sent</strong>: 7/5/18 4:50 PM<br><strong>To</strong>: <a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div><br></div><div>The cloud-init requires network connectivity by default in order to reach the metadata server for the hostname, ssh-key, etc</div><div><br></div><div>You can configure cloud-init to use the config-drive, but the lack of network connectivity will make the instance useless anyway, even though it will have you ssh-key and hostname...</div><div><br></div><div>Did you check the things I told you? </div><div><br>On Jul 5, 2018, at 16:06, Torin Woltjer <<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div style="font-family:arial;font-size:12px;"><div>Are IP addresses set by cloud-init on boot? I noticed that cloud-init isn't working on my VMs. created a new instance from an ubuntu 18.04 image to test with, the hostname was not set to the name of the instance and could not login as users I had specified in the configuration.</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249ClickablePhoneNumber"><span class="m_5383920506812206249ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div><br></div><hr id="m_5383920506812206249previousmessagehr"><div><span><strong>From</strong>: George Mihaiescu <<a href="mailto:lmihaiescu@gmail.com" target="_blank">lmihaiescu@gmail.com</a>><br><strong>Sent</strong>: 7/5/18 12:57 PM<br><strong>To</strong>: <a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a><br><strong>Cc</strong>: "<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><wbr>" <<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><wbr>>, "<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.<wbr>openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.<wbr>openstack.org</a>><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div dir="ltr"><div>You should tcpdump inside the qdhcp namespace to see if the requests make it there, and also check iptables rules on the compute nodes for the return traffic.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 5, 2018 at 12:39 PM, Torin Woltjer <span dir="ltr"><<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="font-family:arial;font-size:12px;"><div>Yes, I've done this. The VMs hang for awhile waiting for DHCP and eventually come up with no addresses. neutron-dhcp-agent has been restarted on both controllers. The qdhcp netns's were all present; I stopped the service, removed the qdhcp netns's, noted the dhcp agents show offline by `neutron agent-list`, restarted all neutron services, noted the qdhcp netns's were recreated, restarted a VM again and it still fails to pull an IP address.</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249m_-1661399758165171731ClickablePhoneNumber"><span class="m_5383920506812206249m_-1661399758165171731ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div><br></div><hr id="m_5383920506812206249m_-1661399758165171731previousmessagehr"><div><span><strong>From</strong>: George Mihaiescu <<a href="mailto:lmihaiescu@gmail.com" target="_blank">lmihaiescu@gmail.com</a>><br><strong>Sent</strong>: 7/5/18 10:38 AM<br><strong>To</strong>: <a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div dir="ltr">Did you restart the neutron-dhcp-agent  and rebooted the VMs?</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 5, 2018 at 10:30 AM, Torin Woltjer <span dir="ltr"><<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="font-family:arial;font-size:12px;"><div>The qrouter netns appears once the lock_path is specified, the neutron router is pingable as well. However, instances are not pingable. If I log in via console, the instances have not been given IP addresses, if I manually give them an address and route they are pingable and seem to work. So the router is working correctly but dhcp is not working.</div><div><br></div><div>No errors in any of the neutron or nova logs on controllers or compute nodes.</div><div><br></div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber"><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div><br></div><hr id="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872previousmessagehr"><div><span><strong>From</strong>: "Torin Woltjer" <<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>><br><strong>Sent</strong>: 7/5/18 8:53 AM<br><strong>To</strong>: <<a href="mailto:lmihaiescu@gmail.com" target="_blank">lmihaiescu@gmail.com</a>><br><strong>Cc</strong>: <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.open<wbr>stack.org</a>, <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div style="font-family:arial;font-size:12px;"><div>There is no lock path set in my neutron configuration. Does it ultimately matter what it is set to as long as it is consistent? Does it need to be set on compute nodes as well as controllers?</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber"><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div><br></div><hr id="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872previousmessagehr"><div><span><strong>From</strong>: George Mihaiescu <<a href="mailto:lmihaiescu@gmail.com" target="_blank">lmihaiescu@gmail.com</a>><br><strong>Sent</strong>: 7/3/18 7:47 PM<br><strong>To</strong>: <a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a><br><strong>Cc</strong>: <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.open<wbr>stack.org</a>, <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div><br></div><div>Did you set a lock_path in the neutron’s config?</div><div><br>On Jul 3, 2018, at 17:34, Torin Woltjer <<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div style="font-family:arial;font-size:12px;"><div>The following errors appear in the neutron-linuxbridge-agent.log on both controllers: <a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"></a><a href="http://paste.openstack.org/show/724930/" target="_blank"><a target="_blank" href="http://paste.openstack.org/show/724930/">http://paste.openstack.org/sho<wbr>w/724930/</a></a></div><div><br></div><div>No such errors are on the compute nodes themselves.</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber"><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div><br></div><hr id="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872previousmessagehr"><div><span><strong>From</strong>: "Torin Woltjer" <<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>><br><strong>Sent</strong>: 7/3/18 5:14 PM<br><strong>To</strong>: <<a href="mailto:lmihaiescu@gmail.com" target="_blank">lmihaiescu@gmail.com</a>><br><strong>Cc</strong>: "<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.ope<wbr>nstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.ope<wbr>nstack.org</a>>, "<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><wbr>" <<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><wbr>><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div style="font-family:arial;font-size:12px;"><div>Running `openstack server reboot` on an instance just causes the instance to be stuck in a rebooting status. Most notable of the logs is neutron-server.log which shows the following:</div><div><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"></a><a href="http://paste.openstack.org/show/724917/" target="_blank"><a target="_blank" href="http://paste.openstack.org/show/724917/">http://paste.openstack.org/sho<wbr>w/724917/</a></a></div><div><br></div><div>I realized that rabbitmq was in a failed state, so I bootstrapped it, rebooted controllers, and all of the agents show online.</div><div><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"></a><a href="http://paste.openstack.org/show/724921/" target="_blank"><a target="_blank" href="http://paste.openstack.org/show/724921/">http://paste.openstack.org/sho<wbr>w/724921/</a></a></div><div>And all of the instances can be properly started, however I cannot ping any of the instances floating IPs or the neutron router. And when logging into an instance with the console, there is no IP address on any interface.</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber"><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div><br></div><hr id="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872previousmessagehr"><div><span><strong>From</strong>: George Mihaiescu <<a href="mailto:lmihaiescu@gmail.com" target="_blank">lmihaiescu@gmail.com</a>><br><strong>Sent</strong>: 7/3/18 11:50 AM<br><strong>To</strong>: <a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a><br><strong>Subject</strong>: Re: [Openstack] Recovering from full outage</span></div><div dir="ltr">Try restarting them using "openstack server reboot" and also check the nova-compute.log and neutron agents logs on the compute nodes.</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 3, 2018 at 11:28 AM, Torin Woltjer <span dir="ltr"><<a href="mailto:torin.woltjer@granddial.com" target="_blank">torin.woltjer@granddial.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="font-family:arial;font-size:12px;"><div>We just suffered a power outage in out data center and I'm having trouble recovering the Openstack cluster. All of the nodes are back online, every instance shows active but `virsh list --all` on the compute nodes show that all of the VMs are actually shut down. Running `ip addr` on any of the nodes shows that none of the bridges are present and `ip netns` shows that all of the network namespaces are missing as well. So despite all of the neutron service running, none of the networking appears to be active, which is concerning. How do I solve this without recreating all of the networks?</div><div><br></div><div><div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"> </span></div><div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><strong><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872m_-5169717044002267889ClickablePhoneNumber"><span class="m_5383920506812206249m_-1661399758165171731m_-3568879635690894872m_-5169717044002267889ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family:Arial,Helvetica,Sans-Serif;font-size:12px;"><em><strong><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"></a><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div></div><br>______________________________<wbr>_________________<br>Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><a target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a></a><br>Post to     : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"></a><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><a target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a></a><br><br></blockquote></div><br></div></div></div></div></blockquote></div></div></blockquote></div><br></div></div></blockquote></div><br></div></div></div></blockquote></div></blockquote></div><br></div></div></div></div></div>