[Openstack] [IceHouse][Trusty] iptables rules disappeared from within Tenant Namespace (qdhcp-XXX), all tenants affected! Metadata not working anymore...

Martinx - ジェームズ thiagocmartinsc at gmail.com
Sat Aug 16 05:57:21 UTC 2014


I restarted the network node a few times, before trying to
`neutron-ovs-cleanup` it manually.
It is running during boot time, weird...
And... Talking with myself!     lol     :-P
Nice weekend for everybody!

On 16 August 2014 00:48, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:

> Fixed!
>
> I just did on Neutron Node:
>
> ---
> # stop neutron-* services
> service neutron-ovs-cleanup start
> reboot
> ---
>
> ---
> ubuntu at linux-builder-1:~$ curl http://169.254.169.254/
> 1.0
> 2007-01-19
> 2007-03-01
> 2007-08-29
> 2007-10-10
> 2007-12-15
> 2008-02-01
> 2008-09-01
> 2009-04-04
> ---
>
> Knowledge base...    :-P
>
>
> On 16 August 2014 00:40, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
> wrote:
>
>> Guys,
>>
>> I'm seeing the following error on Neutron Node:
>>
>> --
>> 2014-08-15 20:36:59.596 12685 ERROR neutron.agent.dhcp_agent
>> [req-17198f16-4149-47f8-8647-0b381df7d888 None] Unable to enable dhcp for
>> f0076840-43f3-4b2e-aa15-d6b2422e3795.
>> --
>>
>> I'm getting there...   :-)
>>
>>
>> On 15 August 2014 21:10, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>> wrote:
>>
>>> Guys,
>>>
>>> Today I'm facing a old/new problem... Metadata doesn't work anymore
>>> (again)... From an already running instance, I'm seeing:
>>>
>>>
>>> ---
>>> ubuntu at linux-builder-1:~$ curl http://169.254.169.254/
>>> curl: (7) Failed to connect to 169.254.169.254 port 80: Connection
>>> refused
>>> ---
>>>
>>>
>>> Then, I looked at my Neutron Node, at this tenant Namespace, there are
>>> no iptables rules there, look:
>>>
>>> ---
>>> root at neutron-node-1:/var/log/neutron# ip netns exec
>>> qdhcp-f0076840-43f3-4b2e-aa15-d6b2422e3795 iptables -L -nv -t nat
>>> Chain PREROUTING (policy ACCEPT 4 packets, 776 bytes)
>>>  pkts bytes target     prot opt in     out     source
>>> destination
>>>
>>> Chain INPUT (policy ACCEPT 4 packets, 776 bytes)
>>>  pkts bytes target     prot opt in     out     source
>>> destination
>>>
>>> Chain OUTPUT (policy ACCEPT 1 packets, 328 bytes)
>>>  pkts bytes target     prot opt in     out     source
>>> destination
>>>
>>> Chain POSTROUTING (policy ACCEPT 1 packets, 328 bytes)
>>>  pkts bytes target     prot opt in     out     source
>>> destination
>>>
>>>
>>>  root at neutron-node-1:/var/log/neutron# ip netns exec
>>> qdhcp-f0076840-43f3-4b2e-aa15-d6b2422e3795 ip -4 r
>>> default via 10.192.0.1 dev tap216bf57d-e8
>>> 10.192.0.0/20 dev tap216bf57d-e8  proto kernel  scope link  src
>>> 10.192.0.3
>>> 169.254.0.0/16 dev tap216bf57d-e8  proto kernel  scope link  src
>>> 169.254.169.254
>>> ---
>>>
>>>
>>> I can see that "curl request" within the Namespace, with tcpdump:
>>> ---
>>> root at neutron-node-1:~# ip netns exec
>>> qdhcp-f0076840-43f3-4b2e-aa15-d6b2422e3795 tcpdump -ni tap216bf57d-e8
>>>
>>>
>>>
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on tap216bf57d-e8, link-type EN10MB (Ethernet), capture size
>>> 65535 bytes
>>> 21:01:29.582960 IP 10.192.0.90.55635 > 169.254.169.254.80: Flags [S],
>>> seq 2521313833, win 29200, options [mss 1460,sackOK,TS val 85649833 ecr
>>> 0,nop,wscale 7], length 0
>>> 21:01:29.583140 IP 169.254.169.254.80 > 10.192.0.90.55635: Flags [R.],
>>> seq 0, ack 2521313834, win 0, length 0
>>> ---
>>>
>>>
>>> If I'm not wrong, there was some iptables NAT rules there, to redirect
>>> the Metadata traffic to the Nova API (controller-node-1) at TCP port 8775,
>>> right?!
>>>
>>> I'm using `VLAN Provider Networks` (No L3 Router), my Instances have a
>>> route to the 169.254.169.254 IP, via their Namespace IP (10.192.0.3), look:
>>>
>>> ---
>>> ubuntu at linux-builder-1:~$ ip r
>>> default via 10.192.0.1 dev eth0
>>> 10.192.0.0/20 dev eth0  proto kernel  scope link  src 10.192.0.90
>>> 169.254.169.254 via 10.192.0.3 dev eth0
>>>
>>> ubuntu at linux-builder-1:~$ ping -c 1 10.192.0.3
>>> PING 10.192.0.3 (10.192.0.3) 56(84) bytes of data.
>>> 64 bytes from 10.192.0.3: icmp_seq=1 ttl=64 time=4.55 ms
>>>  ---
>>>
>>> It crashed today, it was okay yesterday... This time, I did nothing
>>> wrong (I think)...   :-(
>>>
>>> BTW, I just upgraded all nodes (apt-get update / dist-upgrade), still
>>> doesn't work... Proposed repo enabled.
>>>
>>> I really appreciate any help!
>>>
>>> Tks!
>>> Thiago
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140816/626ead60/attachment.html>


More information about the Openstack mailing list