<div dir="ltr">Hi All,<div><br></div><div>I posted below question on launchpad -quantum, but i didn't get any response from the team, may be its not that active as openstack mailing list.</div><div><br></div><div>I am facing an issue detailed in this question [<a href="https://answers.launchpad.net/quantum/+question/221283" target="_blank">https://answers.launchpad.net/quantum/+question/221283</a>] and did some analysis and shared it on the same question. You can find my analysis in the below mail as well. </div>
<div><br></div><div>I am looking for suggestion from the openstack networking expert on how to further debug this issue. My deployment is stuck because of this issue. I really appreciate your help. </div><div><br></div><div>
Thanks</div><div>Anil<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Anil Vishnoi</b> <span dir="ltr"><<a href="mailto:question221283@answers.launchpad.net" target="_blank">question221283@answers.launchpad.net</a>></span><br>
Date: Fri, Feb 8, 2013 at 3:41 AM<br>Subject: Re: [Question #221283]: VM instance is not able to get IP address<br>To: <a href="mailto:vishnoianil@gmail.com" target="_blank">vishnoianil@gmail.com</a><br><br><br><div>Your question #221283 on quantum changed:<br>
<a href="https://answers.launchpad.net/quantum/+question/221283" target="_blank">https://answers.launchpad.net/quantum/+question/221283</a><br>
<br>
</div>You gave more information on the question:<br>
Hi Team,<br>
<br>
I further debugged this issue, and figure out one workaround. I really<br>
don't want to say it a "workaround" but moreover its a hack.<br>
<br>
As i mentioned in the above description that because of the action=drop,<br>
DHCP packets were getting dropped and not reaching to the DHCP agent,<br>
and hence it was not able to respond with the DHCPOFFER response.<br>
<br>
First i resolve this error [Feb 07 17:32:40|00001|netdev_linux|WARN|/sys/class/net/tap9fdb5c15-26/carrier: open failed: ] with the following steps :<br>
1. disable the network namespace for l3_agent and dhcp agent by modifying the use_namespace=false in the respective configuration file.<br>
2. Delete the port (tap9fdb5c15-26) from the br-int bridge.<br>
[Quick instructions :<br>
root@management:~# ovs-vsctl del-port tap9fdb5c15-26<br>
root@management:~# ovs-vsctl add-port br-int tap9fdb5c15-26<br>
root@management:~# ovs-vsctl set port tap9fdb5c15-26 tag=1<br>
root@management:~# ovs-vsctl set Interface tap9fdb5c15-26 type=internal<br>
]<br>
3. Restart both the services and it will create tap devices outside the network name space.<br>
<br>
If network namespace is enabled, ifconfig will not show this tap device<br>
in its output, but if you fire command 'ip netns exec dhcpnsXXXX ip -d<br>
link' it will show you the device.<br>
<br>
In my setup i followed the above step, but even if you don't want to<br>
disable namespace, you can stop dhcp agent, delete the port from br-int<br>
and restart the service. It possibly will resolve this error ( it did<br>
worked in my setup).<br>
<br>
So in my setup, namespace is disabled. And following is the output of<br>
ovs-dpctl<br>
<br>
root@management:~# ovs-dpctl show<br>
system@br-eth1:<br>
lookups: hit:151651 missed:37759 lost:0<br>
<div> flows: 3<br>
port 0: br-eth1 (internal)<br>
port 1: eth1<br>
</div> port 3: phy-br-eth1<br>
system@br-int:<br>
lookups: hit:1183 missed:23283 lost:0<br>
flows: 1<br>
port 0: br-int (internal)<br>
port 6: tap9fdb5c15-26 (internal)<br>
port 7: int-br-eth1<br>
system@br-ex:<br>
lookups: hit:96895 missed:67156 lost:0<br>
flows: 16<br>
<div> port 0: br-ex (internal)<br>
port 1: eth0<br>
<br>
</div>DHCP request packet is broadcast packet and it takes following path to<br>
reach the br-int port 1: eth1 (br-eth1) --> port 7: int-br-eth1(br-<br>
int) and this packet gets drop here because of the following rule<br>
installed on br-int bridge<br>
<br>
cookie=0x0, duration=11422.615s, table=0, n_packets=16711,<br>
n_bytes=1178562, priority=2,in_port=7 actions=drop<br>
<br>
Ideally it should be forwarded to port 6: tap9fdb5c15-26 (internal) (br-<br>
int) and that way it can reach DHCP agent. So i modified above flow to<br>
following flow<br>
<br>
cookie=0x0, duration=3169.501s, table=0, n_packets=2562, n_bytes=228241,<br>
priority=2,in_port=7 actions=output:6<br>
<br>
and also installed following rule to route back the DHCPOFFER packet<br>
<br>
cookie=0x0, duration=4536.551s, table=0, n_packets=233, n_bytes=28896,<br>
priority=2,in_port=6 actions=output:7<br>
<br>
So after installing these two flow rules, DHCP agent got the request and<br>
responded with the DHCPOFFER response.<br>
<br>
root@management:~# tail -f /var/log/syslog<br>
Feb 8 03:26:16 management dnsmasq-dhcp[25811]: DHCPREQUEST(tap9fdb5c15-26) 192.168.0.3 fa:16:3e:93:74:73<br>
Feb 8 03:26:16 management dnsmasq-dhcp[25811]: DHCPACK(tap9fdb5c15-26) 192.168.0.3 fa:16:3e:93:74:73 192-168-0-3<br>
<br>
DHCP response packet will take following path port 6: tap9fdb5c15-26<br>
(internal)(br-int) ---> port 7: int-br-eth1(br-int) ---> port 3: phy-br-<br>
eth1 (br-eth1) ---> port 1: eth1 (br-eth1) and that way this packet<br>
will go out of controller node. But on br-eth1 bridge another rule was<br>
installed which was dropping the response<br>
<br>
cookie=0x0, duration=2669.22s, table=0, n_packets=173, n_bytes=18144,<br>
priority=2,in_port=3 actions=drop<br>
<br>
and i changed this flow to<br>
<br>
cookie=0x0, duration=2669.22s, table=0, n_packets=173, n_bytes=18144,<br>
priority=2,in_port=3 actions=output:1<br>
<br>
so now packet can escape from the controller machine. Now follows the<br>
story of compute node side.<br>
<br>
Following is ovs-dpctl output of my compute node :<br>
<br>
system@br-eth1:<br>
lookups: hit:404442 missed:110048 lost:0<br>
<div> flows: 1<br>
port 0: br-eth1 (internal)<br>
port 1: eth1<br>
</div> port 3: phy-br-eth1<br>
system@br-int:<br>
lookups: hit:1884 missed:71022 lost:0<br>
<div> flows: 0<br>
port 0: br-int (internal)<br>
</div> port 3: int-br-eth1<br>
port 4: qvo819abf08-ca<br>
port 6: tap718d359b-d1 <<VM Connected to this tap device<br>
<br>
Response packet should take following path: port 1: eth1(br-eth1) ---><br>
port 3: int-br-eth1 (br-int) --->port 6: tap718d359b-d1 (br-int), but<br>
on br-int bridge following flow rule was installed which was dropping<br>
the response packet<br>
<br>
cookie=0x0, duration=1671.356s, table=0, n_packets=1068, n_bytes=99127,<br>
priority=2,in_port=3 actions=drop<br>
<br>
so i modified this flow to<br>
<br>
cookie=0x0, duration=1671.356s, table=0, n_packets=1068, n_bytes=99127,<br>
priority=2,in_port=3 actions=output:6<br>
<br>
and that way it was forwarding the packet to my VM, and i can now see that IP address 192.168.0.3 is now assigned to my machine. Ideally this is the job of quantum plug-in, but not sure why its dropping all the packets from both the sides.<br>
Above exercise establishes the fact that dhcp agent is working fine here, its the network routing which is causing the issue, and that too openvswitch plug-in as per my understanding.<br>
<br>
Seeking suggestion from the networking experts on the list, what<br>
possibly can cause this issue, do openvswitch plug-in has any dependency<br>
on linux bridge or brcompat module to work properly ? because on<br>
controller node neither bridge module nor brcompat module is loaded.<br>
Obviously this hack won't work for all other cases, so we need to<br>
resolve the issue at the plugin level. Please suggest!<br>
<br>
Thanks<br>
<span><font color="#888888">Anil<br>
</font></span><div><div><br>
--<br>
You received this question notification because you asked the question.<br>
</div></div></div><br><br clear="all"><div><br></div>-- <br>Thanks & Regards<br>--Anil Kumar Vishnoi
</div></div>