
<div style="font-family: arial; font-size: 12px;"><div data-empty="true">$ip netns exec qdhcp-87a5200d-057f-475d-953d-17e873a47454 curl <a target="_blank" href="http://169.254.169.254">http://169.254.169.254</a>                                          <br><html><br><head><br> <title>404 Not Found</title><br></head><br><body><br> <h1>404 Not Found</h1><br> The resource could not be found.<br /><br /><br><br><br><br></body><br></html></div><div data-empty="true">$ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e curl <a target="_blank" href="http://169.254.169.254">http://169.254.169.254</a>                                          <br>curl: (7) Couldn't connect to server<br><br></div><div data-empty="true"><br></div><div contenteditable="false"><div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><span style="font-family:lucida sans unicode,lucida grande,sans-serif;"><span style="font-size:16px;"><strong>Torin Woltjer</strong></span></span></em></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong>Grand Dial Communications - A ZK Tech Inc. Company</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"> </span></div><div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><strong><span class="ClickablePhoneNumber"><span class="ClickablePhoneNumber">616.776.1066</span></span> ext. 2006</strong></span></div><div><span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px;"><em><strong><a href="http://www.granddial.com" target="_blank"><a target="_blank" href="http://www.granddial.com">www.granddial.com</a></a></strong></em></span></div></div></div></div></div><div data-empty="true"><br></div><hr id="previousmessagehr"><div><span><strong>From</strong>: "Torin Woltjer" <torin.woltjer@granddial.com><br><strong>Sent</strong>: 7/12/18 11:16 AM<br><strong>To</strong>: <haleyb.dev@gmail.com>, <thangam.arunx@gmail.com>, "jpetrini@coredial.com" <jpetrini@coredial.com><br><strong>Cc</strong>: openstack-operators@lists.openstack.org, openstack@lists.openstack.org<br><strong>Subject</strong>: Re: [Openstack] [Openstack-operators] Recovering from full outage</span></div><div style="font-family: arial; font-size: 12px;"><div>Checking iptables for the metadata-proxy inside of qrouter provides the following:</div><div>$ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e iptables-save -c | grep 169            <br>[0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697<br>[0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff</div><div>Packets:Bytes are both 0, so no traffic is touching this rule?</div><div><br></div><div>Interestingly the command:</div><div>$ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e netstat -anep | grep 9697<br>returns nothing, so there isn't actually anything running on 9697 in the network namespace...</div><div><br></div><div>This is the output without grep:<br>Active Internet connections (servers and established)<br>Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name    <br>raw        0      0 0.0.0.0:112             0.0.0.0:*               7           0          76154      8404/keepalived      <br>raw        0      0 0.0.0.0:112             0.0.0.0:*               7           0          76153      8404/keepalived      <br>Active UNIX domain sockets (servers and established)<br>Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path<br>unix  2      [ ]         DGRAM                    64501    7567/python2          <br>unix  2      [ ]         DGRAM                    79953    8403/keepalived</div><div><br></div><div>Could the reason no traffic touching the rule be that nothing is listening on that port, or is there a second issue down the chain?</div><div><br></div><div>Curl fails even after restarting the neutron-dhcp-agent & neutron-metadata agent.</div><div><br>Thank you for this, and any future help.</div></div></div>