<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I'm trying to resolve an issue that happens about 30% of the time. DVR is being used in this environment. The environment is deployed with Kolla Ansible on CentOS 7 using Stein (the latest Kolla Ansible installer). Physical servers have the latest CentOS 7 patches through November 30th, 2019.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This seems to happen on new and existing projects, including newly created subnets, and, so far, it seems to be only related to VM launches. Delaying the time between a subnet creation and the VM launch makes no difference in the behavior.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When the problem occurs, after a VM is launched, it takes a longer-than-normal time to boot. This would usually indicate a DNS issue or possible a DHCP issue.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>After the OS boots (after 60 seconds or so), I can login, but with a delayed login (another 20 seconds or so) due to the DNS issue I'm diagnosing.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>DHCP works fine - no issues with IP assignment, and the /etc/resolv.conf entries are set to the routers' two private IPs. No DHCP errors in the dmesg output.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I can connect to both respective qdhcp network namespaces where dnsmasq is running for the two distributed routers, immediately after the subnet is created (in our test case, it is 192.168.99.0/24), and can resolve names using "nslookup <name> 192.168.99.1" and "nslookup <name> 192.168.99.2" instantly. So, dnsmasq is working properly.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>A floating IP is assigned to this VM, which is used for the following SSH sessions.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>After SSH'ing to the VM, I can ping the 192.168.99.1 and 192.168.99.2 addresses, so it does not appear to be a network issue (ICMP works), unless there is a firewall rule either in iptables or OVS that is blocking UDP/TCP port 53 for DNS requests, which would be odd, since the egress security group assigned to this VM is unrestricted.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I can also ping external IPs from the VM (Internet-accessible IPs), so the routers are working.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I can also perform nslookups against external DNS servers without issue, such as "nslookup <name> 1.1.1.1". If I direct nslookup to the internal DNS servers using "nslookup <name> 192.168.99.1", the lookup times out.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We normally use CentOS images, but I tried an Ubuntu 19.04 image and the same problem occurs often.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Now, what is ODD - the problem goes away after about 3 minutes! DNS lookups against the internal DNS servers (the router IPs) start working perfectly. So something is taking a while to get configured, but only about 30% of the time.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>If, during the creation of the subnet, we specify --dns-nameserver options, such as:<o:p></o:p></p><p class=MsoNormal> --dns-nameserver 1.1.1.1<o:p></o:p></p><p class=MsoNormal>the issue NEVER occurs. So it seems to be limited to DNS traffic between the VM and the dnsmasq service in the qdhcp namespace.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>All traffic in this environment is switched (we are not routing VXLAN traffic, for example), so there can be no physical network interference at Layers 3 or higher.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The problem has also occurred on multiple hosts in the environment (the VM has been launched on various hosts with the same issue).<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any ideas how to diagnose this? I'm assuming it has something to do with the iptables or OVS configuration, but if someone knows specifically what could cause this, such as a specific rule that is supposed to be created for DNS traffic to the internal routers, I could dive into the these configurations.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Eric<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>