[Openstack] [Openstack-operators] Recovering from full outage

Brian Haley haleyb.dev at gmail.com
Mon Jul 16 20:38:31 UTC 2018 

On 07/16/2018 08:41 AM, Torin Woltjer wrote:
> $ip netns exec qdhcp-87a5200d-057f-475d-953d-17e873a47454 curl 
> http://169.254.169.254
> <html>
> <head>
>   <title>404 Not Found</title>
> </head>
> <body>
>   <h1>404 Not Found</h1>
>   The resource could not be found.<br /><br /> > </body>
> </html>

Strange, don't know where the reply came from for that.

> $ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e curl 
> http://169.254.169.254
> curl: (7) Couldn't connect to server

Based on your iptables output below, I would think the metadata proxy is 
running in the qrouter namespace.  However, a curl from there will not 
work since it is restricted to only work for incoming packets from the 
qr- device(s).  You would have to try curl from a running instance.

Is there an haproxy process running?  And is it listening on port 9697 
in the qrouter namespace?

-Brian


> ------------------------------------------------------------------------
> *From*: "Torin Woltjer" <torin.woltjer at granddial.com>
> *Sent*: 7/12/18 11:16 AM
> *To*: <haleyb.dev at gmail.com>, <thangam.arunx at gmail.com>, 
> "jpetrini at coredial.com" <jpetrini at coredial.com>
> *Cc*: openstack-operators at lists.openstack.org, openstack at lists.openstack.org
> *Subject*: Re: [Openstack] [Openstack-operators] Recovering from full outage
> Checking iptables for the metadata-proxy inside of qrouter provides the 
> following:
> $ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e 
> iptables-save -c | grep 169
> [0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p 
> tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
> [0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p 
> tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
> Packets:Bytes are both 0, so no traffic is touching this rule?
> 
> Interestingly the command:
> $ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e netstat 
> -anep | grep 9697
> returns nothing, so there isn't actually anything running on 9697 in the 
> network namespace...
> 
> This is the output without grep:
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address           Foreign Address         
> State       User       Inode      PID/Program name
> raw        0      0 0.0.0.0:112             0.0.0.0:*               7   
>          0          76154      8404/keepalived
> raw        0      0 0.0.0.0:112             0.0.0.0:*               7   
>          0          76153      8404/keepalived
> Active UNIX domain sockets (servers and established)
> Proto RefCnt Flags       Type       State         I-Node   PID/Program 
> name     Path
> unix  2      [ ]         DGRAM                    64501    7567/python2
> unix  2      [ ]         DGRAM                    79953    8403/keepalived
> 
> Could the reason no traffic touching the rule be that nothing is 
> listening on that port, or is there a second issue down the chain?
> 
> Curl fails even after restarting the neutron-dhcp-agent & 
> neutron-metadata agent.
> 
> Thank you for this, and any future help.

More information about the Openstack mailing list