<div dir="ltr">Anil,<div><br></div><div style>Thanks a lot for your help. Although I had the trunking configured on the ports properly, I forgot to actually add the VLAN to the vlan database ("set vlan 1024", and I was done). As soon as that was done, the VM got its IP address.</div>
<div style><br></div><div style>Can't ping or ssh though! Argh. I still think the instructions I've been following are leaving something out on the way the physical and bridged interfaces needs to be configured on startup (these instructions: <a href="https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst">https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst</a>).</div>
<div style><br></div><div style>For example, even after rebooting the network node after installation, the br-int, br-ex, and br-eth1 interfaces were all down. And I wasn't able to troubleshoot the VLAN tagging until added those interfaces to /etc/network/interfaces and restarted networking.</div>
<div style><br></div><div style>The symptoms now are:</div><div style><br></div><div style>* Whereas before I was able to ping the tenant's router and dhcp IPs from the controller, I can't now.</div><div style>* The VNC console inside Horizon fails to connect (it could before).</div>
<div style>* External VNC connections work, however. Once inside, I can see that the VM interface is now configured with <a href="http://192.168.1.3/24">192.168.1.3/24</a>. It can ping the DHCP server (192.168.1.2) but not the default router (192.168.1.1).</div>
<div style><br></div><div style>I have all the salient configs paste-binned here: <a href="http://pastebin.com/ZZAuaH4u">http://pastebin.com/ZZAuaH4u</a></div><div style><br></div><div style>Suggestions? I'm tempted to re-install at this point, I've mucked around with manual network changes so much.</div>
<div style><br></div><div class="gmail_extra">Thanks again!<br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 1:44 PM, Anil Vishnoi <span dir="ltr"><<a href="mailto:vishnoianil@gmail.com" target="_blank">vishnoianil@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">so if the packet is going out from compute node -eth1 interface and not reaching network node eth1 interface, then the only element in between these two interface is physical switch. Both the switch ports should be trunked, so that they allow all the vlan traffic on both the ports where network node and compute node are connected.<div>
<br></div><div>Process for VLAN trunking depends on the switch vendor. AFAIK in cisco switch all the ports are by default trunk-ed, but thats not the case with all vendors. You might want to re-check the switch configuration and test whether its really passing the tagged traffic or not. </div>
</div><div class=""><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 11:54 PM, Greg Chavez <span dir="ltr"><<a href="mailto:greg.chavez@gmail.com" target="_blank">greg.chavez@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div>I'm seeing three BOOTP/DHCP packets on the eth1 interface of the compute node, but it doesn't make it to the network node. So I may have a VLAN tagging issue.</div>
<div><br></div>
<div>The segmentation id for the VM network is 1024 (same as what's in the github instructions). The segmentation id for the public network is 3001. </div><div><br></div><div><div>root@kcon-cs-gen-01i:/etc/init.d# quantum net-show 654a49a3-f042-45eb-a937-0dcd6fcaa84c | grep seg</div>
<div>| provider:segmentation_id | 3001 |</div><div><br></div><div>root@kcon-cs-gen-01i:/etc/init.d# quantum net-show c9c6a895-8bc1-4319-a207-30422d0d1a27 | grep seg</div><div>| provider:segmentation_id | 1024 </div>
</div><div><br></div><div>The physical switch ports for the eth1 interfaces of the compute and network node are set to trunk and are allowing both 3001 and 1024.</div><div><br></div><div>Here are the interfaces on the compute node:</div>
<div><br></div><div><div>root@kvm-cs-sn-10i:/etc/network/if-up.d# ip add show | egrep ^[0-9]+</div><div><div>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN </div><div>2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000</div>
</div><div>3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000</div><div><div>4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000</div><div>5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000</div>
</div><div>6: br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN </div><div>12: qvo5334a0cb-64: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</div>
<div>13: qvb5334a0cb-64: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</div><div>29: br-int: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN </div>
<div>37: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</div><div>38: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</div>
<div>39: qbr9cf85869-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP </div><div>40: qvo9cf85869-88: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</div>
<div>41: qvb9cf85869-88: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr9cf85869-88 state UP qlen 1000</div><div>47: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr9cf85869-88 state UNKNOWN qlen 500</div>
<div><br></div><div>I set the interfaces file up like this (excluding eth0):</div><div><br></div><div><div>auto br-eth1</div><div>iface br-eth1 inet static</div><div>address 192.168.239.110</div><div>netmask 255.255.255.0</div>
<div>bridge_ports eth1</div><div><br></div><div>auto eth1</div><div> iface eth1 inet manual</div><div> up ifconfig $IFACE 0.0.0.0 up</div><div> up ip link set $IFACE promisc on</div><div> down ip link set $IFACE promisc off</div>
<div> down ifconfig $IFACE down</div><div><br></div><div>auto br-int</div><div> iface br-int inet manual</div><div> up ifconfig $IFACE 0.0.0.0 up</div><div> up ip link set $IFACE promisc on</div><div> down ip link set $IFACE promisc off</div>
<div> down ifconfig $IFACE down</div><div><br></div><div>But check this out. On the network node:</div><div><br></div><div><div><div>root@knet-cs-gen-01i:~# ip addr show | egrep ^[0-9]+</div><div><div>
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN </div><div>2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000</div></div><div>3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000</div>
<div>4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master br-ex state UP qlen 1000</div><div><div>5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000</div>
</div><div>6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP </div><div>
<div>7: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN </div></div><div>8: br-int: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN </div><div>9: tapca2d1ff4-7b: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN </div>
<div>12: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</div><div>13: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000</div>
</div><div><br></div><div>What??? I have br-eth1, br-ex, and br-int configured the same way in /etc/network/interfaces. So i run "ifconfig eth1 up", restart the agents... and I still don't see any DHCP packets hitting the network node...</div>
<div>. </div><div><br></div></div></div></div></div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 12:42 PM, Anil Vishnoi <span dir="ltr"><<a href="mailto:vishnoianil@gmail.com" target="_blank">vishnoianil@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">now can you dump packet at the eth1 and see if DHCP traffic is leaving your host or not.<div>
<br></div>
<div>tcpdump -nnei eth1 |grep DHCP</div><div><br></div><div>And also dump the packets at the controller node, and see if you see the same DHCP packet there.</div>
<div><br></div><div>Just start these tcpdump on compute and controller node and restart your VM instance.</div></div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 11:07 PM, Greg Chavez <span dir="ltr"><<a href="mailto:greg.chavez@gmail.com" target="_blank">greg.chavez@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Yeah, I was wondering about that! I was expecting to see it too. I've terminated that VM and launched another one. Here's the output of those commands again, from the compute node:<div>
<br></div><div>
<div>root@kvm-cs-sn-10i:~# ovs-vsctl show</div><div>9d9f7949-2b80-40c8-a9e0-6a116200ed96</div><div> Bridge br-int</div><div> Port "qvo9cf85869-88"</div><div> tag: 1</div><div> Interface "qvo9cf85869-88"</div>
<div>
<div> Port br-int</div><div> Interface br-int</div><div> type: internal</div><div> Port "int-br-eth1"</div><div> Interface "int-br-eth1"</div><div> Bridge "br-eth1"</div>
</div><div><div> Port "br-eth1"</div><div> Interface "br-eth1"</div><div> type: internal</div></div><div> Port "phy-br-eth1"</div><div> Interface "phy-br-eth1"</div>
<div>
<div> Port "eth1"</div><div> Interface "eth1"</div><div> ovs_version: "1.4.3"</div><div><br></div></div><div>root@kvm-cs-sn-10i:~# ovs-dpctl show</div><div>system@br-eth1:</div>
<div><span style="white-space:pre-wrap"> </span>lookups: hit:5497 missed:25267 lost:0</div><div><div><span style="white-space:pre-wrap"> </span>flows: 1</div><div><span style="white-space:pre-wrap"> </span>port 0: br-eth1 (internal)</div>
<div><span style="white-space:pre-wrap"> </span>port 1: eth1</div></div><div><span style="white-space:pre-wrap"> </span>port 7: phy-br-eth1</div><div>system@br-int:</div><div><span style="white-space:pre-wrap"> </span>lookups: hit:3270 missed:14993 lost:0</div>
<div>
<div><span style="white-space:pre-wrap"> </span>flows: 1</div><div><span style="white-space:pre-wrap"> </span>port 0: br-int (internal)</div></div><div><span style="white-space:pre-wrap"> </span>port 3: int-br-eth1</div>
<div><span style="white-space:pre-wrap"> </span>port 4: qvo9cf85869-88</div><div><br></div><div>root@kvm-cs-sn-10i:~# brctl show</div><div><div>bridge name<span style="white-space:pre-wrap"> </span>bridge id<span style="white-space:pre-wrap"> </span>STP enabled<span style="white-space:pre-wrap"> </span>interfaces</div>
<div>br-eth1<span style="white-space:pre-wrap"> </span>0000.bc305befedd1<span style="white-space:pre-wrap"> </span>no<span style="white-space:pre-wrap"> </span>eth1</div><div><span style="white-space:pre-wrap"> </span>phy-br-eth1</div>
<div>br-int<span style="white-space:pre-wrap"> </span>0000.8ae31e5f7941<span style="white-space:pre-wrap"> </span>no<span style="white-space:pre-wrap"> </span>int-br-eth1</div></div><div><span style="white-space:pre-wrap"> </span>qvo9cf85869-88</div>
<div>qbr9cf85869-88<span style="white-space:pre-wrap"> </span>8000.aeae57c4b763<span style="white-space:pre-wrap"> </span>no<span style="white-space:pre-wrap"> </span>qvb9cf85869-88</div><div><span style="white-space:pre-wrap"> </span>vnet0</div>
<div><br></div><div>root@kvm-cs-sn-10i:~# ovs-ofctl dump-flows br-int</div><div>NXST_FLOW reply (xid=0x4):</div><div> cookie=0x0, duration=2876.663s, table=0, n_packets=641, n_bytes=122784, priority=2,in_port=3 actions=drop</div>
<div> cookie=0x0, duration=1097.004s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=3,dl_vlan=1024 actions=mod_vlan_vid:1,NORMAL</div><div> cookie=0x0, duration=2877.036s, table=0, n_packets=16, n_bytes=1980, priority=1 actions=NORMAL</div>
<div><br></div><div>root@kvm-cs-sn-10i:~# ovs-ofctl dump-flows br-eth1</div><div>NXST_FLOW reply (xid=0x4):</div><div> cookie=0x0, duration=2878.446s, table=0, n_packets=11, n_bytes=854, priority=2,in_port=7 actions=drop</div>
<div> cookie=0x0, duration=1098.842s, table=0, n_packets=11, n_bytes=1622, priority=4,in_port=7,dl_vlan=1 actions=mod_vlan_vid:1024,NORMAL</div><div> cookie=0x0, duration=2878.788s, table=0, n_packets=635, n_bytes=122320, priority=1 actions=NORMAL</div>
</div><div><br></div><div>And for good measure here's some info from the network node:</div><div><br></div><div><div>root@knet-cs-gen-01i:~# ip netns exec qdhcp-c9c6a895-8bc1-4319-a207-30422d0d1a27 netstat -rn</div>
<div>Kernel IP routing table</div><div>Destination Gateway Genmask Flags MSS Window irtt Iface</div><div>192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tapca2d1ff4-7b</div>
<div><br></div><div>root@knet-cs-gen-01i:~# ip netns exec qrouter-ddd535ad-debb-4810-bc10-f419f105c959 netstat -rn</div><div>Kernel IP routing table</div><div>Destination Gateway Genmask Flags MSS Window irtt Iface</div>
<div>0.0.0.0 10.21.164.1 0.0.0.0 UG 0 0 0 qg-81523176-e1</div><div>10.21.164.0 0.0.0.0 255.255.252.0 U 0 0 0 qg-81523176-e1</div><div>192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-973ae179-06</div>
</div><div><br></div><div><br></div><div>And yes, I have nets set up. From the controller node:</div><div><br></div><div><div>root@kcon-cs-gen-01i:/etc/init.d# nova --os-username=user_one --os-password=user_one --os-tenant-name=project_one list | grep ACT</div>
<div>| 12c37be6-14e3-471b-aae3-750af8cfca32 | server-02 | ACTIVE | net_proj_one=192.168.1.3 |</div><div><br></div><div>root@kcon-cs-gen-01i:/etc/init.d# quantum net-list | grep _</div><div>| 654a49a3-f042-45eb-a937-0dcd6fcaa84c | ext_net | 842a2baa-829c-4daa-8d3d-77dace5fba86 |</div>
<div>| c9c6a895-8bc1-4319-a207-30422d0d1a27 | net_proj_one | 4ecef580-27ad-44f1-851d-c2c4cd64d323 |</div></div><div><br></div><div>Any light bulbs appearing over your head? :)</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 11:51 AM, Anil Vishnoi <span dir="ltr"><<a href="mailto:vishnoianil@gmail.com" target="_blank">vishnoianil@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">I am assuming this output is from compute node, but i don't see any tapX device attached to your "br-int" bridge. <div><br></div><div>If you are running VM instance on this compute node then it should be connected to br-int through tapX device. Quantum automatically does that if you spawn VM and specify network to which you want to connect this VM.</div>
<div><br></div><div>If you fire following command </div><div><br></div><div>~# quantum net-list<br></div><div><br></div><div>do you see any network entry in the output ?</div><div><br></div><div><br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Wed, Feb 20, 2013 at 10:05 PM, Greg Chavez <span dir="ltr"><<a href="mailto:greg.chavez@gmail.com" target="_blank">greg.chavez@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div dir="ltr"><div><br></div>Here's the last command output you asked for:<div>
<br>
</div><div><div>root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-eth1</div>
<div>NXST_FLOW reply (xid=0x4):</div><div>
cookie=0x0, duration=78793.694s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=6 actions=drop</div><div> cookie=0x0, duration=78794.033s, table=0, n_packets=17355, n_bytes=3335788, priority=1 actions=NORMAL</div>
<div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 10:57 AM, Greg Chavez <span dir="ltr"><<a href="mailto:greg.chavez@gmail.com" target="_blank">greg.chavez@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div dir="ltr"><br><div class="gmail_quote">
<div dir="ltr">Hey Anil, thanks for responding. Here's the output:<div><br></div><div><div>root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-vsctl show</div><div>9d9f7949-2b80-40c8-a9e0-6a116200ed96</div>
<div> Bridge br-int</div>
<div> Port br-int</div><div> Interface br-int</div><div> type: internal</div><div> Port "int-br-eth1"</div><div> Interface "int-br-eth1"</div><div> Bridge "br-eth1"</div>
<div> Port "phy-br-eth1"</div><div> Interface "phy-br-eth1"</div><div> Port "br-eth1"</div><div> Interface "br-eth1"</div><div> type: internal</div>
<div> Port "eth1"</div><div> Interface "eth1"</div><div> ovs_version: "1.4.3"</div><div><br></div><div>root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-dpctl show</div><div>
system@br-eth1:</div><div><span style="white-space:pre-wrap"> </span>lookups: hit:5227 missed:24022 lost:0</div><div><span style="white-space:pre-wrap"> </span>flows: 1</div><div><span style="white-space:pre-wrap"> </span>port 0: br-eth1 (internal)</div>
<div><span style="white-space:pre-wrap"> </span>port 1: eth1</div><div><span style="white-space:pre-wrap"> </span>port 6: phy-br-eth1</div><div>system@br-int:</div><div><span style="white-space:pre-wrap"> </span>lookups: hit:2994 missed:13754 lost:0</div>
<div><span style="white-space:pre-wrap"> </span>flows: 1</div><div><span style="white-space:pre-wrap"> </span>port 0: br-int (internal)</div><div><span style="white-space:pre-wrap"> </span>port 2: int-br-eth1</div>
<div><br></div><div>root@kvm-cs-sn-10i:/var/lib/nova/instances# brctl show</div><div>bridge name<span style="white-space:pre-wrap"> </span>bridge id<span style="white-space:pre-wrap"> </span>STP enabled<span style="white-space:pre-wrap"> </span>interfaces</div>
<div>br-eth1<span style="white-space:pre-wrap"> </span>0000.bc305befedd1<span style="white-space:pre-wrap"> </span>no<span style="white-space:pre-wrap"> </span>eth1</div><div><span style="white-space:pre-wrap"> </span>phy-br-eth1</div>
<div>br-int<span style="white-space:pre-wrap"> </span>0000.8ae31e5f7941<span style="white-space:pre-wrap"> </span>no<span style="white-space:pre-wrap"> </span>int-br-eth1</div><div>qbr5334a0cb-64<span style="white-space:pre-wrap"> </span>8000.76fb293fe9cf<span style="white-space:pre-wrap"> </span>no<span style="white-space:pre-wrap"> </span>qvb5334a0cb-64</div>
<div><span style="white-space:pre-wrap"> </span>vnet0</div><div><br></div><div>root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-int<br></div><div>NXST_FLOW reply (xid=0x4):</div><div> cookie=0x0, duration=75251.156s, table=0, n_packets=16581, n_bytes=3186436, priority=2,in_port=2 actions=drop</div>
<div> cookie=0x0, duration=75251.527s, table=0, n_packets=0, n_bytes=0, priority=1 actions=NORMAL</div></div><div><br></div><div>Thanks! I have until tomorrow to get this working, then my boss has mandating that I try Cloudstack. Argh, but I'm so close!</div>
</div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 10:29 AM, Anil Vishnoi <span dir="ltr"><<a href="mailto:vishnoianil@gmail.com" target="_blank">vishnoianil@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi Greg,<div><br></div><div>Can you paste the output of following command from your compute node. </div>
<div><br></div><div>ovs-vsctl show</div><div>ovs-dpctl show</div><div>brctl show</div>
<div><br></div><div>ovs-ofctl dump-flows br-int</div><div>ovs-ofctl dump-flows br-eth1</div><div><br></div><div>Because i think first issue we need to resolve here is why DHCP packet is not leaving your compute host.</div>
<div><br></div><div>Anil</div></div><div class="gmail_extra"><div><br><br><div class="gmail_quote">On Wed, Feb 20, 2013 at 6:48 AM, Greg Chavez <span dir="ltr"><<a href="mailto:greg.chavez@gmail.com" target="_blank">greg.chavez@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">
<div>From my perspective, it seems that the OVS bridges are not being brought up correctly. As you can see in my earlier post, the integration bridge (int) and the physical interface bridges (br-ex and br-eth1) are downed. I've tried to bring them up in promiscuous mode in the case of br-int, and with the physical interfaces ported to the bridge in the case of br-ex and br-eth1. I've had no luck unfortunately.</div>
<div><br></div><div>It seems that nothing is getting past br-int. I can see BOOTP packets on the VM side of br-int, and I can see VTP packets on the physical side of br-int. But that's where it ends.</div>
<div><br></div><div>For example, when I reboot my VM, I see this:</div><div><br></div><div><div>root@kvm-cs-sn-10i:/var/lib/nova/instances# tcpdump -i qvo5334a0cb-64</div><div><br></div><div>tcpdump: WARNING: qvo5334a0cb-64: no IPv4 address assigned</div>
<div>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div><div>listening on qvo5334a0cb-64, link-type EN10MB (Ethernet), capture size 65535 bytes</div><div><br></div><div>13:42:08.099061 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280</div>
<div>13:42:08.101675 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28</div><div>13:42:08.161728 IP6 :: > ff02::1:ff06:4809: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe06:4809, length 24</div>
<div>13:42:08.373745 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28</div><div>13:42:11.102528 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280</div>
<div>13:42:14.105850 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280</div><div><br></div><div>But that's as far as it goes. The dhcp agent never get this.</div>
<div><br></div><div>I"ve tried deleting and recreating the bridges, rebooting the systems, but nothing seems to work. Maybe it's just the right combination of things. I don't know. </div><div>
<br></div><div>Help!</div></div></div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Feb 19, 2013 at 5:23 AM, Sylvain Bauza <span dir="ltr"><<a href="mailto:sylvain.bauza@digimind.com" target="_blank">sylvain.bauza@digimind.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi Greg,<br>
<br>
I did have trouble with DHCP assignation (see my previous post in
this list), which was being fixed by deleting ovs bridges on
network node, recreating them and restarting OVS plugin and
L3/DHCP agents (which were all on the same physical node).<br>
Maybe it helps.<br>
<br>
Anyway, when DHCP'ing from your VM (asking for an IP), could you
please tcpdump : <br>
1. your virtual network interface on compute node<br>
2. your physical network interface on compute node<br>
3. your physical network interface on network node<br>
<br>
and see BOOTP/DHCP packets ?<br>
On the physical layer, you should see GRE packets (provided you
correctly followed the mentioned guide) encapsulating your
BOOTP/DHCP packets.<br>
<br>
If that's OK, could you please issue the below commands (on the
network node) :<br>
- brctl show<br>
- ip a<br>
- ovs-vsctl show<br>
- route -n<br>
<br>
Thanks,<br>
-Sylvain<br>
<br>
Le 19/02/2013 00:55, Greg Chavez a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Third time I'm replying to my own message. It
seems like the initial network state is a problem for many first
time openstackers. Surely somewhere would be well to assist me.
I'm running out of time to make this work. Thanks.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Sun, Feb 17, 2013 at 3:08 AM, Greg
Chavez <span dir="ltr"><<a href="mailto:greg.chavez@gmail.com" target="_blank">greg.chavez@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">I'm replying to my own message because I'm
desperate. My network situation is a mess. I need to add
this as well: my bridge interfaces are all down. On my
compute node:
<div>
<br>
</div>
<div>
<div>root@kvm-cs-sn-10i:/var/lib/nova/instances/instance-00000005#
ip addr show | grep ^[0-9]</div>
<div>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc
noqueue state UNKNOWN </div>
<div>2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc mq state UP qlen 1000</div>
<div>3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc mq state UP qlen 1000</div>
<div>4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc
noop state DOWN qlen 1000</div>
<div>5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc
noop state DOWN qlen 1000</div>
<div>9: br-int: <BROADCAST,MULTICAST> mtu 1500
qdisc noop state DOWN </div>
<div>10: br-eth1: <BROADCAST,MULTICAST> mtu 1500
qdisc noop state DOWN </div>
<div>13: phy-br-eth1:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP qlen 1000</div>
<div>14: int-br-eth1:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP qlen 1000</div>
<div>15: qbre56c5d9e-b6:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP </div>
<div>16: qvoe56c5d9e-b6:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast state UP qlen 1000</div>
<div>17: qvbe56c5d9e-b6:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast master qbre56c5d9e-b6 state UP
qlen 1000</div>
<div>19: qbrb805a9c9-11:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP </div>
<div>20: qvob805a9c9-11:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast state UP qlen 1000</div>
<div>21: qvbb805a9c9-11:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast master qbrb805a9c9-11 state UP
qlen 1000</div>
<div>34: qbr2b23c51f-02:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP </div>
<div>35: qvo2b23c51f-02:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast state UP qlen 1000</div>
<div>36: qvb2b23c51f-02:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu
1500 qdisc pfifo_fast master qbr2b23c51f-02 state UP
qlen 1000</div>
<div>37: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc pfifo_fast master qbr2b23c51f-02 state
UNKNOWN qlen 500</div>
</div>
<div><br>
</div>
<div>And on my network node:</div>
<div><br>
</div>
<div>
<div>root@knet-cs-gen-01i:~# ip addr show | grep ^[0-9]</div>
<div>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc
noqueue state UNKNOWN </div>
<div>2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc mq state UP qlen 1000</div>
<div>3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc mq state UP qlen 1000</div>
<div>4: eth2:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu
1500 qdisc mq state UP qlen 1000</div>
<div>5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc
noop state DOWN qlen 1000</div>
<div>6: br-int: <BROADCAST,MULTICAST> mtu 1500
qdisc noop state DOWN </div>
<div>7: br-eth1: <BROADCAST,MULTICAST> mtu 1500
qdisc noop state DOWN </div>
<div>8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc noqueue state UNKNOWN </div>
<div>22: phy-br-eth1:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP qlen 1000</div>
<div>23: int-br-eth1:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP qlen 1000</div>
<div><br>
</div>
<div>I gave br-ex an IP and UP'ed it manually. I assume
this is correct. By I honestly don't know.</div>
<div><br>
</div>
<div>Thanks.</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Feb 15, 2013 at 6:54
PM, Greg Chavez <span dir="ltr"><<a href="mailto:greg.chavez@gmail.com" target="_blank">greg.chavez@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
Sigh. So I abandoned RHEL 6.3, rekicked my
systems and set up the scale-ready installation
described in these instructions:
<div>
<br>
</div>
<div><a href="https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst" target="_blank">https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst</a></div>
<div><br>
</div>
<div>Basically:</div>
<div><br>
</div>
<div>(o) controller node on a mgmt and public
net</div>
<div>(o) network node (quantum and openvs) on a
mgmt, net-config, and public net</div>
<div>
(o) compute node is on a mgmt and net-config
net</div>
<div><br>
</div>
<div>Took me just over an hour and ran into only
a few easily-fixed speed bumps. But the VM
networks are totally non-functioning. VMs
launch but no network traffic can go in or
out.</div>
<div><br>
</div>
<div>I'm particularly befuddled by these
problems:</div>
<div><br>
</div>
<div>( 1 ) This error in nova-compute:</div>
<div><br>
</div>
<div>ERROR nova.network.quantumv2 [-]
_get_auth_token() failed<br>
</div>
<div><br>
</div>
<div>( 2 ) No NAT rules on the compute node,
which probably explains why the VMs complain
about not finding a network or being able to
get metadata from 169.254.169.254.</div>
<div><br>
</div>
<div>
<div>root@kvm-cs-sn-10i:~# iptables -t nat -S</div>
<div>-P PREROUTING ACCEPT</div>
<div>-P INPUT ACCEPT</div>
<div>-P OUTPUT ACCEPT</div>
<div>-P POSTROUTING ACCEPT</div>
<div>-N nova-api-metadat-OUTPUT</div>
<div>-N nova-api-metadat-POSTROUTING</div>
<div>-N nova-api-metadat-PREROUTING</div>
<div>-N nova-api-metadat-float-snat</div>
<div>-N nova-api-metadat-snat</div>
<div>-N nova-compute-OUTPUT</div>
<div>-N nova-compute-POSTROUTING</div>
<div>-N nova-compute-PREROUTING</div>
<div>-N nova-compute-float-snat</div>
<div>-N nova-compute-snat</div>
<div>-N nova-postrouting-bottom</div>
<div>-A PREROUTING -j
nova-api-metadat-PREROUTING</div>
<div>-A PREROUTING -j nova-compute-PREROUTING</div>
<div>-A OUTPUT -j nova-api-metadat-OUTPUT</div>
<div>-A OUTPUT -j nova-compute-OUTPUT</div>
<div>-A POSTROUTING -j
nova-api-metadat-POSTROUTING</div>
<div>-A POSTROUTING -j
nova-compute-POSTROUTING</div>
<div>-A POSTROUTING -j nova-postrouting-bottom</div>
<div>-A nova-api-metadat-snat -j
nova-api-metadat-float-snat</div>
<div>-A nova-compute-snat -j
nova-compute-float-snat</div>
<div>-A nova-postrouting-bottom -j
nova-api-metadat-snat</div>
<div>-A nova-postrouting-bottom -j
nova-compute-snat</div>
</div>
<div><br>
</div>
<div>(3) A lastly, no default secgroup rules,
whose function governs... what exactly?
Connections to the VM's public or private
IPs? I guess I'm just not sure if this is
relevant to my overall problem of ZERO VM
network connectivity.</div>
<div><br>
</div>
<div>I seek guidance please. Thanks.</div>
<span><font color="#888888">
<div><br></div></font></span></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></div></div></div>
</blockquote></div></div></div></blockquote></div></div></div></div></div></div></div></blockquote></div><br></div></div></div></div></div>
<br></div></div><div>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></div></blockquote></div><div><br><br clear="all"><div><br></div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div>
</div></div>