<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In our small cloud environment, we started to see weird behavior during last 2 months. Dhcp namespaces started to disappear randomly, which caused that VMs losed connectivity once dhcp lease expired.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
After the investigation I found out following issue/bug:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<ol>
<li><span>ipv6 metadata address of tap interface in some qdhcp-xxxx namespaces are stucked in "dadfailed tentative" state (i do not know why yet)</span></li><li style="display:block"><span><br>
</span></li><li style="display:block"><span>root@cloud01:~# ip netns exec qdhcp-3094b264-829b-4381-9ca2-59b3a3fc1ea1 ip a
<div>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000</div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div> inet 127.0.0.1/8 scope host lo</div>
<div> valid_lft forever preferred_lft forever</div>
<div> inet6 ::1/128 scope host </div>
<div> valid_lft forever preferred_lft forever</div>
<div>2585: tap1797d9b1-e1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000</div>
<div> link/ether fa:16:3e:77:64:0d brd ff:ff:ff:ff:ff:ff</div>
<div> inet 169.254.169.254/32 brd 169.254.169.254 scope global tap1797d9b1-e1</div>
<div> valid_lft forever preferred_lft forever</div>
<div> inet 192.168.0.2/24 brd 192.168.0.255 scope global tap1797d9b1-e1</div>
<div> valid_lft forever preferred_lft forever</div>
<div> inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative </div>
<div> valid_lft forever preferred_lft forever</div>
<div> inet6 fe80::f816:3eff:fe77:640d/64 scope link </div>
<span> valid_lft forever preferred_lft forever</span><span></span></span></li><li style="display:block"><span><span><br>
</span></span></li><li><span>This blocked dhcp agent to finish sync_state function, and NetworkCache was not updated with subnets of such neutron network</span></li><li><span>During creation of VM assigned to such network, agent does not detect any subnets (see point 2), so he thinks (reload_allocations()) there is no dhcp needed and deletes qdhcp-xxxx namespace, so no DHCP and no Metadata are working on such network since
that moment, and after 24h we see connectivity issues.</span></li><li><span>Restart of DHCP agent recreates missing qdhcp-xxxx namespaces, but <span>
NetworkCache</span> in dhcp agent is again empty, so creation of VM deletes the qdhcp-xxxx namespace again
<span id="๐" contenteditable="false">๐</span><br>
</span></li></ol>
<div><span>Workaround is to remove dhcp agent from that network and add it again. Interestingly, sometimes I need to do it multiple times, because in few cases tap interface in new qdhcp finishes again in
<span>dadfailed tentative</span> state. After year in production we have 20 networks out of 60 in such state.</span></div>
<div><span><br>
</span></div>
<div><span>We are using kolla-ansible deployment on Ubuntu 20.04, kernel 5.4.0-65-generic. Openstack version Victoria and neutron is in version 17.2.2.dev70.</span></div>
<div><span><br>
</span></div>
<div><span><br>
</span></div>
<div><span>Is that bug in neutron, or is it misconfiguration of OS on our side?</span></div>
<div><span><br>
</span></div>
<div><span>I'm locally testing patch which disables ipv6 dad in qdhcp-xxxx namespace (net.ipv6.conf.default.accept_dad = 1), but I'm not sure it is good solution when it comes to other neutron features?</span></div>
<div><span><br>
</span></div>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Kamil Madรกฤ</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:8pt"><i>Slovensko IT a.s.</i></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
</div>
</div>
</div>
</body>
</html>