[Openstack-operators] [Openstack] Recovering from full outage

Torin Woltjer torin.woltjer at granddial.com
Mon Jul 16 12:41:16 UTC 2018


$ip netns exec qdhcp-87a5200d-057f-475d-953d-17e873a47454 curl http://169.254.169.254                                          
<html>
<head>
 <title>404 Not Found</title>
</head>
<body>
 <h1>404 Not Found</h1>
 The resource could not be found.<br /><br />

</body>
</html>
$ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e curl http://169.254.169.254                                          
curl: (7) Couldn't connect to server

Torin Woltjer
 
Grand Dial Communications - A ZK Tech Inc. Company
 
616.776.1066 ext. 2006
www.granddial.com

----------------------------------------
From: "Torin Woltjer" <torin.woltjer at granddial.com>
Sent: 7/12/18 11:16 AM
To: <haleyb.dev at gmail.com>, <thangam.arunx at gmail.com>, "jpetrini at coredial.com" <jpetrini at coredial.com>
Cc: openstack-operators at lists.openstack.org, openstack at lists.openstack.org
Subject: Re: [Openstack] [Openstack-operators] Recovering from full outage
Checking iptables for the metadata-proxy inside of qrouter provides the following:
$ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e iptables-save -c | grep 169            
[0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
[0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
Packets:Bytes are both 0, so no traffic is touching this rule?

Interestingly the command:
$ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e netstat -anep | grep 9697
returns nothing, so there isn't actually anything running on 9697 in the network namespace...

This is the output without grep:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name    
raw        0      0 0.0.0.0:112             0.0.0.0:*               7           0          76154      8404/keepalived      
raw        0      0 0.0.0.0:112             0.0.0.0:*               7           0          76153      8404/keepalived      
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ]         DGRAM                    64501    7567/python2          
unix  2      [ ]         DGRAM                    79953    8403/keepalived

Could the reason no traffic touching the rule be that nothing is listening on that port, or is there a second issue down the chain?

Curl fails even after restarting the neutron-dhcp-agent & neutron-metadata agent.

Thank you for this, and any future help.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180716/57a32033/attachment.html>


More information about the OpenStack-operators mailing list