<div dir="ltr"><div></div>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">Hi all,</span></p>
<p class="inbox-inbox-p2"><span style="line-height:1.5">I’m running Mitaka on CentOS 7.2 with Neutron in dvr_snat mode. </span><br><span class="inbox-inbox-s1"></span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"># uname -msr</span></p><p class="inbox-inbox-p1"><span style="line-height:1.5">Linux 3.10.0-327.22.2.el7.x86_64 x86_64</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">I’m using vlans, not vxlans, but I don’t think that matters either way. So basically, I have one NIC “eth2” which is in vlan trunk mode, and on my switch side, I have every neutron-defined vlan trunked there. Whether it’s a tenant network vlan, or an external vlan for floating IPs, it all comes back to that same NIC.</span></p>
<p class="inbox-inbox-p2"><span style="line-height:1.5">So here’s a compute node “node1”. It has a successfully booted VM, which has fixed IP 10.97.8.103 and floating IP 10.96.8.107. As seen from the compute node:</span><br><span class="inbox-inbox-s1"></span></p>
<p class="inbox-inbox-p2"><span style="line-height:1.5"><br></span></p><p class="inbox-inbox-p2"><span style="line-height:1.5"># ip netns</span><br><span class="inbox-inbox-s1"></span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">fip-cbe55dc5-c4e4-4ec0-aa52-b4713f1279ee</span></p><p class="inbox-inbox-p1"><span style="line-height:1.5">qrouter-efc60192-97ad-49ef-bab7-cda42ca6bc29</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">snat-efc60192-97ad-49ef-bab7-cda42ca6bc29</span></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"># ip netns exec fip-cbe55dc5-c4e4-4ec0-aa52-b4713f1279ee ip addr</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet <a href="http://127.0.0.1/8">127.0.0.1/8</a> scope host lo</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet6 ::1/128 scope host</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2: fpr-efc60192-9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> link/ether 32:06:67:df:53:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet <a href="http://169.254.109.47/31">169.254.109.47/31</a> scope global fpr-efc60192-9</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet6 fe80::3006:67ff:fedf:53c6/64 scope link</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">19: fg-152dc56a-c1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> link/ether fa:16:3e:40:9f:5b brd ff:ff:ff:ff:ff:ff</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet <a href="http://10.96.8.101/23">10.96.8.101/23</a> brd 10.96.9.255 scope global fg-152dc56a-c1</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet6 fe80::f816:3eff:fe40:9f5b/64 scope link</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"># ip netns exec qrouter-efc60192-97ad-49ef-bab7-cda42ca6bc29 ip addr</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet <a href="http://127.0.0.1/8">127.0.0.1/8</a> scope host lo</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet6 ::1/128 scope host</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2: rfp-efc60192-9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> link/ether 72:49:e7:78:48:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet <a href="http://169.254.109.46/31">169.254.109.46/31</a> scope global rfp-efc60192-9</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet <a href="http://10.96.8.107/32">10.96.8.107/32</a> brd 10.96.8.107 scope global rfp-efc60192-9</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet6 fe80::7049:e7ff:fe78:485d/64 scope link</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">17: qr-ffc302ba-82: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> link/ether fa:16:3e:8d:7c:62 brd ff:ff:ff:ff:ff:ff</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet <a href="http://10.97.8.1/23">10.97.8.1/23</a> brd 10.97.9.255 scope global qr-ffc302ba-82</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> inet6 fe80::f816:3eff:fe8d:7c62/64 scope link</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1"> valid_lft forever preferred_lft forever</span></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">So you can see that I have both the ‘fpr’ and ‘rfp’ namespaces, which is a good indicator I didn’t totally flub the dvr_snat neutron config. From within either namespace, I can ping the floating IP 10.96.8.107, which makes sense. However, for the floating IP to be useful, it would need to be generally reachable by any other system in its designated vlan, and that is not the case. In my real-world use case, I would be running the vlan of this floating IP network back over to my bastion host, to allow users to ssh into their VMs via the floating IP. I can’t reach the floating IPs though from anywhere outside the namespace on the compute node.</span></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">One more clue, in the l3-agent log on the compute node in question:</span></p>
<p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span><br></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 10.96.8.107 on fg-152dc56a-c1 in namespace fip-cbe55dc5-c4e4-4ec0-aa52-b4713f1279ee</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib Traceback (most recent call last):</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1040, in _arping</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib ip_wrapper.netns.execute(arping_cmd, check_exit_code=True)</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 927, in execute</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib log_fail_as_error=log_fail_as_error, **kwargs)</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib raise RuntimeError(msg)</span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">2016-08-03 11:14:09.665 6041 ERROR neutron.agent.linux.ip_lib RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: bind: Cannot assign requested address</span></p>
<p class="inbox-inbox-p2"><br><span class="inbox-inbox-s1"></span></p>
<p class="inbox-inbox-p1"><span class="inbox-inbox-s1">After a little Googling, I think I may be seeing the same behavior as this user:</span></p>
<p class="inbox-inbox-p2"><a href="https://bugs.centos.org/view.php?id=11238" style="line-height:1.5">https://bugs.centos.org/view.php?id=11238</a><br><span class="inbox-inbox-s1"></span></p>
<p class="inbox-inbox-p2"><span style="line-height:1.5">I’m reaching out to see if anyone else has witnessed this, or has any sage advice for me.</span><br><span class="inbox-inbox-s1"></span></p><p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span></p>
<p class="inbox-inbox-p2"><span style="line-height:1.5"><br></span></p><p class="inbox-inbox-p2"><span style="line-height:1.5">Jonathan</span><br><span class="inbox-inbox-s1"></span></p><p class="inbox-inbox-p2"><span class="inbox-inbox-s1"></span></p></div>