<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Brian,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
thank you very much for pointing to those bugs. It is exactly what we are experiencing in our deployment. I will follow-up in those bugs then.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Kamil<br>
</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Brian Haley <haleyb.dev@gmail.com><br>
<b>Sent:</b> Monday, January 3, 2022 2:35 AM<br>
<b>To:</b> Kamil Madáč <kamil.madac@slovenskoit.sk>; openstack-discuss <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> Re: [neutron] Dadfailed of ipv6 metadata IP in qdhcp namespace and disappearing dhcp namespaces</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Hi,<br>
<br>
On 1/2/22 10:51 AM, Kamil Madáč wrote:<br>
> Hello,<br>
> <br>
> In our small cloud environment, we started to see weird behavior during <br>
> last 2 months. Dhcp namespaces started to disappear randomly, which <br>
> caused that VMs losed connectivity once dhcp lease expired.<br>
> After the investigation I found out following issue/bug:<br>
> <br>
> 1. ipv6 metadata address of tap interface in some qdhcp-xxxx namespaces<br>
> are stucked in "dadfailed tentative" state (i do not know why yet)<br>
<br>
This issue was reported about a month ago:<br>
<br>
<a href="https://bugs.launchpad.net/neutron/+bug/1953165">https://bugs.launchpad.net/neutron/+bug/1953165</a><br>
<br>
And Bence marked it a duplicate of:<br>
<br>
<a href="https://bugs.launchpad.net/neutron/+bug/1930414">https://bugs.launchpad.net/neutron/+bug/1930414</a><br>
<br>
Seems to be a bug in a flow based on the title - "Traffic leaked from <br>
dhcp port before vlan tag is applied".<br>
<br>
I would follow-up in that second bug.<br>
<br>
Thanks,<br>
<br>
-Brian<br>
<br>
> 3. root@cloud01:~# ip netns exec<br>
> qdhcp-3094b264-829b-4381-9ca2-59b3a3fc1ea1 ip a<br>
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN<br>
> group default qlen 1000<br>
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>
> inet 127.0.0.1/8 scope host lo<br>
> valid_lft forever preferred_lft forever<br>
> inet6 ::1/128 scope host<br>
> valid_lft forever preferred_lft forever<br>
> 2585: tap1797d9b1-e1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
> qdisc noqueue state UNKNOWN group default qlen 1000<br>
> link/ether fa:16:3e:77:64:0d brd ff:ff:ff:ff:ff:ff<br>
> inet 169.254.169.254/32 brd 169.254.169.254 scope global<br>
> tap1797d9b1-e1<br>
> valid_lft forever preferred_lft forever<br>
> inet 192.168.0.2/24 brd 192.168.0.255 scope global tap1797d9b1-e1<br>
> valid_lft forever preferred_lft forever<br>
> inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative<br>
> valid_lft forever preferred_lft forever<br>
> inet6 fe80::f816:3eff:fe77:640d/64 scope link<br>
> valid_lft forever preferred_lft forever<br>
> 4.<br>
> <br>
> 5. This blocked dhcp agent to finish sync_state function, and<br>
> NetworkCache was not updated with subnets of such neutron network<br>
> 6. During creation of VM assigned to such network, agent does not<br>
> detect any subnets (see point 2), so he thinks<br>
> (reload_allocations()) there is no dhcp needed and deletes<br>
> qdhcp-xxxx namespace, so no DHCP and no Metadata are working on such<br>
> network since that moment, and after 24h we see connectivity issues.<br>
> 7. Restart of DHCP agent recreates missing qdhcp-xxxx namespaces, but<br>
> NetworkCache in dhcp agent is again empty, so creation of VM<br>
> deletes the qdhcp-xxxx namespace again 🙁<br>
> <br>
> Workaround is to remove dhcp agent from that network and add it again. <br>
> Interestingly, sometimes I need to do it multiple times, because in few <br>
> cases tap interface in new qdhcp finishes again in dadfailed tentative <br>
> state. After year in production we have 20 networks out of 60 in such state.<br>
> <br>
> We are using kolla-ansible deployment on Ubuntu 20.04, kernel <br>
> 5.4.0-65-generic. Openstack version Victoria and neutron is in version <br>
> 17.2.2.dev70.<br>
> <br>
> <br>
> Is that bug in neutron, or is it misconfiguration of OS on our side?<br>
> <br>
> I'm locally testing patch which disables ipv6 dad in qdhcp-xxxx <br>
> namespace (net.ipv6.conf.default.accept_dad = 1), but I'm not sure it is <br>
> good solution when it comes to other neutron features?<br>
> <br>
> <br>
> Kamil Madáč<br>
> /Slovensko IT a.s./<br>
> <br>
</div>
</span></font></div>
</body>
</html>