[Openstack] [Openstack-operators] Recovering from full outage

Brian Haley haleyb.dev at gmail.com
Thu Jul 12 14:29:54 UTC 2018


On 07/12/2018 08:20 AM, Torin Woltjer wrote:
> The neutron-metadata-agent service is running, the the agent is alive, 
> and it is listening on port 8775. However, new instances still do not 
> get any information like hostname or keypair. If I run `curl 
> 192.168.116.22:8775` from the compute nodes, I do get a response. The 
> metadata agent is running, listening, and accessible from the compute 
> nodes; and it worked previously.
> 
> I'm stumped.

There is also a metadata proxy that runs in the qrouter namespace, you 
can verify it's running and getting requests by looking at both iptables 
and netstat output.

$ sudo ip netns exec qrouter-$ID iptables-save -c | grep 169
[16:960] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p 
tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
[96:7968] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ 
-p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff

The numbers inside [] represent packets:bytes, so non-zero is good.

$ sudo ip netns exec qrouter-$ID netstat -anep | grep 9697
tcp        0      0 0.0.0.0:9697            0.0.0.0:* 
LISTEN      0          294339      4867/haproxy

If you have a running instance you can log into, running curl to the 
metadata IP would be helpful to try and diagnose since it would go 
through this entire path.

-Brian


> /*Torin Woltjer*/
> *Grand Dial Communications - A ZK Tech Inc. Company*
> *616.776.1066 ext. 2006*
> /*<http://www.granddial.com>www.granddial.com <http://www.granddial.com>*/
> 
> ------------------------------------------------------------------------
> *From*: அருண் குமார் (Arun Kumar) <thangam.arunx at gmail.com>
> *Sent*: 7/12/18 12:01 AM
> *To*: torin.woltjer at granddial.com
> *Cc*: "openstack at lists.openstack.org" <openstack at lists.openstack.org>, 
> openstack-operators at lists.openstack.org
> *Subject*: Re: [Openstack-operators] [Openstack] Recovering from full outage
> Hi Torin,
> 
>     If I run `ip netns exec qrouter netstat -lnp` or `ip netns exec
>     qdhcp netstat -lnp` on the controller, should I see anything
>     listening on the metadata port (8775)? When I run these commands I
>     don't see that listening, but I have no example of a working system
>     to check against. Can anybody verify this?
> 
> 
> Either on qrouter/qdhcp namespaces, you won't see port 8775, instead 
> check whether meta-data service is running on the neutron controller 
> node(s) and listening on port 8775? Aslo, you can verify metadata and 
> neturon services using following commands
> 
> service neutron-metadata-agent status
> neutron agent-list
> netstat -ntplua | grep :8775
> 
> 
> Thanks & Regards
> Arun
> 
> ஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃ
> அன்புடன்
> அருண்
> நுட்பம் நம்மொழியில் தழைக்கச் செய்வோம்
> <http://thangamaniarun.wordpress.com><http://thangamaniarun.wordpress.com>http://thangamaniarun.wordpress.com
> ஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃஃ
> 
> 
> 
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 



More information about the Openstack mailing list