<div dir="ltr"><br><div class="gmail_extra"> <br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im">
> For example, even after rebooting the network node after installation, the<br>
> br-int, br-ex, and br-eth1 interfaces were all down. And I wasn't able to<br>
> troubleshoot the VLAN tagging until added those interfaces to<br>
> /etc/network/interfaces and restarted networking.<br>
<br>
</div>Yes, Quantum does not persist the configuration for you. Actually,<br>
Quantum is polite and does not want to mess with your host's network<br>
configuration.<br>
Indeed the typical workflow is that the admin configures those<br>
bridges, and then tells quantum to use them.<br>
<div class="im"><br>
><br>
> The symptoms now are:<br>
><br>
> * Whereas before I was able to ping the tenant's router and dhcp IPs from<br>
> the controller, I can't now.<br>
<br>
</div>The controller usually does not have a route to the tenant network.<br>
For instance it might be running in some management network (say<br>
<a href="http://10.127.1.0/24" target="_blank">10.127.1.0/24</a>) whereas your tenant network is <a href="http://10.0.0.0/24" target="_blank">10.0.0.0/24</a>.<br>
<div class="im"><br></div></blockquote><div><br></div><div style>The instructions I followed had me set a route for the tenant network on the controller</div><div style><br></div><div style><div>root@kcon-cs-gen-01i:~# quantum port-list -c fixed_ips -c device_owner | grep router_gateway | awk -F '"' '{ print $8; }'</div>
<div>10.21.166.1</div><div><br></div><div>root@kcon-cs-gen-01i:~# netstat -rn | grep 166</div><div>192.168.1.0 10.21.166.1 255.255.255.0 UG 0 0 0 eth0</div></div><div style><br></div><div style><div>
root@kcon-cs-gen-01i:~# ping 192.168.1.3</div><div>PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.</div><div>From 10.21.164.75 icmp_seq=1 Destination Host Unreachable</div><div>From 10.21.164.75 icmp_seq=2 Destination Host Unreachable</div>
<div>From 10.21.164.75 icmp_seq=3 Destination Host Unreachable</div></div><div style><br></div><div style>Don't even see the icmp packets on eth0. Same result with 192.168.1.1 and .2. Those *did* work before I reconfigured the physical interfaces as ports of the OVS bridges.</div>
<div style><br></div><div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
> * The VNC console inside Horizon fails to connect (it could before).<br>
<br>
</div>Unfortunately I do not have an explanation for that. It might be<br>
related to the fact that you're using a provider network, and hence<br>
there might be an underlying bug.<br>
I am unfortunately unable to setup a repro environment now.</blockquote><div><br></div><div style>I think I can set this one aside for now. Just thought it might be a symptom of the main problem: can't ping/ssh to or from VM.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
> * External VNC connections work, however. Once inside, I can see that the<br>
> VM interface is now configured with <a href="http://192.168.1.3/24" target="_blank">192.168.1.3/24</a>. It can ping the DHCP<br>
> server (192.168.1.2) but not the default router (192.168.1.1).<br>
<br>
</div>Can you check the interface is up and running on the l3-agent. You<br>
should see a qr-<partofrouteruuid> iface configured with 1ip<br>
92.168.1.1<br>
<div class=""><div class="h5"><br></div></div></blockquote><div><br></div><div style>Ah-ha! I see this in the l3 logs after I restart the l3-agent:</div><div style><br></div><div style><div>Stderr: 'Cannot find device "qg-81523176-e1"\n'</div>
<div>2013-02-20 17:15:57 ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop</div><div><br></div><div style>But it's right here:</div><div style><br></div><div style><div>root@knet-cs-gen-01i:/var/log/quantum# ovs-vsctl show</div>
<div>7dc4a669-a330-4c5f-a6f1-ab9e35b82685</div><div> Bridge br-int</div><div> Port br-int</div><div> Interface br-int</div><div> type: internal</div><div> Port "qr-973ae179-06"</div>
<div> Interface "qr-973ae179-06"</div><div> type: internal</div><div> Port "int-br-eth1"</div><div> Interface "int-br-eth1"</div><div> Port "tapca2d1ff4-7b"</div>
<div> tag: 1</div><div> Interface "tapca2d1ff4-7b"</div><div> type: internal</div><div> Bridge br-ex</div><div> Port br-ex</div><div> Interface br-ex</div>
<div> type: internal</div><div> Port "qg-81523176-e1"</div><div> Interface "qg-81523176-e1"</div><div> type: internal</div><div> Port "eth2"</div>
<div> Interface "eth2"</div><div> Bridge "br-eth1"</div><div> Port "br-eth1"</div><div> Interface "br-eth1"</div><div> type: internal</div>
<div> Port "eth1"</div><div> Interface "eth1"</div><div> Port "phy-br-eth1"</div><div> Interface "phy-br-eth1"</div></div></div><div style><div><br>
</div></div><div style>Hrm......</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="">
<div class="h5">
><br>
> I have all the salient configs paste-binned here:<br>
> <a href="http://pastebin.com/ZZAuaH4u" target="_blank">http://pastebin.com/ZZAuaH4u</a><br>
><br>
> Suggestions? I'm tempted to re-install at this point, I've mucked around<br>
> with manual network changes so much.<br>
><br>
> Thanks again!<br>
><br>
> On Wed, Feb 20, 2013 at 1:44 PM, Anil Vishnoi <<a href="mailto:vishnoianil@gmail.com">vishnoianil@gmail.com</a>> wrote:<br>
>><br>
>> so if the packet is going out from compute node -eth1 interface and not<br>
>> reaching network node eth1 interface, then the only element in between these<br>
>> two interface is physical switch. Both the switch ports should be trunked,<br>
>> so that they allow all the vlan traffic on both the ports where network node<br>
>> and compute node are connected.<br>
>><br>
>> Process for VLAN trunking depends on the switch vendor. AFAIK in cisco<br>
>> switch all the ports are by default trunk-ed, but thats not the case with<br>
>> all vendors. You might want to re-check the switch configuration and test<br>
>> whether its really passing the tagged traffic or not.<br>
>><br>
>><br>
>> On Wed, Feb 20, 2013 at 11:54 PM, Greg Chavez <<a href="mailto:greg.chavez@gmail.com">greg.chavez@gmail.com</a>><br>
>> wrote:<br>
>>><br>
>>><br>
>>> I'm seeing three BOOTP/DHCP packets on the eth1 interface of the compute<br>
>>> node, but it doesn't make it to the network node. So I may have a VLAN<br>
>>> tagging issue.<br>
>>><br>
>>> The segmentation id for the VM network is 1024 (same as what's in the<br>
>>> github instructions). The segmentation id for the public network is 3001.<br>
>>><br>
>>> root@kcon-cs-gen-01i:/etc/init.d# quantum net-show<br>
>>> 654a49a3-f042-45eb-a937-0dcd6fcaa84c | grep seg<br>
>>> | provider:segmentation_id | 3001 |<br>
>>><br>
>>> root@kcon-cs-gen-01i:/etc/init.d# quantum net-show<br>
>>> c9c6a895-8bc1-4319-a207-30422d0d1a27 | grep seg<br>
>>> | provider:segmentation_id | 1024<br>
>>><br>
>>> The physical switch ports for the eth1 interfaces of the compute and<br>
>>> network node are set to trunk and are allowing both 3001 and 1024.<br>
>>><br>
>>> Here are the interfaces on the compute node:<br>
>>><br>
>>> root@kvm-cs-sn-10i:/etc/network/if-up.d# ip add show | egrep ^[0-9]+<br>
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN<br>
>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP<br>
>>> qlen 1000<br>
>>> 3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq<br>
>>> state UP qlen 1000<br>
>>> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000<br>
>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000<br>
>>> 6: br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue<br>
>>> state UNKNOWN<br>
>>> 12: qvo5334a0cb-64: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500<br>
>>> qdisc pfifo_fast state UP qlen 1000<br>
>>> 13: qvb5334a0cb-64: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500<br>
>>> qdisc pfifo_fast state UP qlen 1000<br>
>>> 29: br-int: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc<br>
>>> noqueue state UNKNOWN<br>
>>> 37: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
>>> pfifo_fast state UP qlen 1000<br>
>>> 38: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
>>> pfifo_fast state UP qlen 1000<br>
>>> 39: qbr9cf85869-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
>>> noqueue state UP<br>
>>> 40: qvo9cf85869-88: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500<br>
>>> qdisc pfifo_fast state UP qlen 1000<br>
>>> 41: qvb9cf85869-88: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500<br>
>>> qdisc pfifo_fast master qbr9cf85869-88 state UP qlen 1000<br>
>>> 47: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast<br>
>>> master qbr9cf85869-88 state UNKNOWN qlen 500<br>
>>><br>
>>> I set the interfaces file up like this (excluding eth0):<br>
>>><br>
>>> auto br-eth1<br>
>>> iface br-eth1 inet static<br>
>>> address 192.168.239.110<br>
>>> netmask 255.255.255.0<br>
>>> bridge_ports eth1<br>
>>><br>
>>> auto eth1<br>
>>> iface eth1 inet manual<br>
>>> up ifconfig $IFACE 0.0.0.0 up<br>
>>> up ip link set $IFACE promisc on<br>
>>> down ip link set $IFACE promisc off<br>
>>> down ifconfig $IFACE down<br>
>>><br>
>>> auto br-int<br>
>>> iface br-int inet manual<br>
>>> up ifconfig $IFACE 0.0.0.0 up<br>
>>> up ip link set $IFACE promisc on<br>
>>> down ip link set $IFACE promisc off<br>
>>> down ifconfig $IFACE down<br>
>>><br>
>>> But check this out. On the network node:<br>
>>><br>
>>> root@knet-cs-gen-01i:~# ip addr show | egrep ^[0-9]+<br>
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN<br>
>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP<br>
>>> qlen 1000<br>
>>> 3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq<br>
>>> state UP qlen 1000<br>
>>> 4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq<br>
>>> master br-ex state UP qlen 1000<br>
>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000<br>
>>> 6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state<br>
>>> UP<br>
>>> 7: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>> 8: br-int: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc<br>
>>> noqueue state UNKNOWN<br>
>>> 9: tapca2d1ff4-7b: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>> 12: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
>>> pfifo_fast state UP qlen 1000<br>
>>> 13: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
>>> pfifo_fast state UP qlen 1000<br>
>>><br>
>>> What??? I have br-eth1, br-ex, and br-int configured the same way in<br>
>>> /etc/network/interfaces. So i run "ifconfig eth1 up", restart the agents...<br>
>>> and I still don't see any DHCP packets hitting the network node...<br>
>>> .<br>
>>><br>
>>><br>
>>><br>
>>> On Wed, Feb 20, 2013 at 12:42 PM, Anil Vishnoi <<a href="mailto:vishnoianil@gmail.com">vishnoianil@gmail.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> now can you dump packet at the eth1 and see if DHCP traffic is leaving<br>
>>>> your host or not.<br>
>>>><br>
>>>> tcpdump -nnei eth1 |grep DHCP<br>
>>>><br>
>>>> And also dump the packets at the controller node, and see if you see the<br>
>>>> same DHCP packet there.<br>
>>>><br>
>>>> Just start these tcpdump on compute and controller node and restart your<br>
>>>> VM instance.<br>
>>>><br>
>>>><br>
>>>> On Wed, Feb 20, 2013 at 11:07 PM, Greg Chavez <<a href="mailto:greg.chavez@gmail.com">greg.chavez@gmail.com</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>> Yeah, I was wondering about that! I was expecting to see it too. I've<br>
>>>>> terminated that VM and launched another one. Here's the output of those<br>
>>>>> commands again, from the compute node:<br>
>>>>><br>
>>>>> root@kvm-cs-sn-10i:~# ovs-vsctl show<br>
>>>>> 9d9f7949-2b80-40c8-a9e0-6a116200ed96<br>
>>>>> Bridge br-int<br>
>>>>> Port "qvo9cf85869-88"<br>
>>>>> tag: 1<br>
>>>>> Interface "qvo9cf85869-88"<br>
>>>>> Port br-int<br>
>>>>> Interface br-int<br>
>>>>> type: internal<br>
>>>>> Port "int-br-eth1"<br>
>>>>> Interface "int-br-eth1"<br>
>>>>> Bridge "br-eth1"<br>
>>>>> Port "br-eth1"<br>
>>>>> Interface "br-eth1"<br>
>>>>> type: internal<br>
>>>>> Port "phy-br-eth1"<br>
>>>>> Interface "phy-br-eth1"<br>
>>>>> Port "eth1"<br>
>>>>> Interface "eth1"<br>
>>>>> ovs_version: "1.4.3"<br>
>>>>><br>
>>>>> root@kvm-cs-sn-10i:~# ovs-dpctl show<br>
>>>>> system@br-eth1:<br>
>>>>> lookups: hit:5497 missed:25267 lost:0<br>
>>>>> flows: 1<br>
>>>>> port 0: br-eth1 (internal)<br>
>>>>> port 1: eth1<br>
>>>>> port 7: phy-br-eth1<br>
>>>>> system@br-int:<br>
>>>>> lookups: hit:3270 missed:14993 lost:0<br>
>>>>> flows: 1<br>
>>>>> port 0: br-int (internal)<br>
>>>>> port 3: int-br-eth1<br>
>>>>> port 4: qvo9cf85869-88<br>
>>>>><br>
>>>>> root@kvm-cs-sn-10i:~# brctl show<br>
>>>>> bridge name bridge id STP enabled interfaces<br>
>>>>> br-eth1 0000.bc305befedd1 no eth1<br>
>>>>> phy-br-eth1<br>
>>>>> br-int 0000.8ae31e5f7941 no int-br-eth1<br>
>>>>> qvo9cf85869-88<br>
>>>>> qbr9cf85869-88 8000.aeae57c4b763 no qvb9cf85869-88<br>
>>>>> vnet0<br>
>>>>><br>
>>>>> root@kvm-cs-sn-10i:~# ovs-ofctl dump-flows br-int<br>
>>>>> NXST_FLOW reply (xid=0x4):<br>
>>>>> cookie=0x0, duration=2876.663s, table=0, n_packets=641,<br>
>>>>> n_bytes=122784, priority=2,in_port=3 actions=drop<br>
>>>>> cookie=0x0, duration=1097.004s, table=0, n_packets=0, n_bytes=0,<br>
>>>>> priority=3,in_port=3,dl_vlan=1024 actions=mod_vlan_vid:1,NORMAL<br>
>>>>> cookie=0x0, duration=2877.036s, table=0, n_packets=16, n_bytes=1980,<br>
>>>>> priority=1 actions=NORMAL<br>
>>>>><br>
>>>>> root@kvm-cs-sn-10i:~# ovs-ofctl dump-flows br-eth1<br>
>>>>> NXST_FLOW reply (xid=0x4):<br>
>>>>> cookie=0x0, duration=2878.446s, table=0, n_packets=11, n_bytes=854,<br>
>>>>> priority=2,in_port=7 actions=drop<br>
>>>>> cookie=0x0, duration=1098.842s, table=0, n_packets=11, n_bytes=1622,<br>
>>>>> priority=4,in_port=7,dl_vlan=1 actions=mod_vlan_vid:1024,NORMAL<br>
>>>>> cookie=0x0, duration=2878.788s, table=0, n_packets=635,<br>
>>>>> n_bytes=122320, priority=1 actions=NORMAL<br>
>>>>><br>
>>>>> And for good measure here's some info from the network node:<br>
>>>>><br>
>>>>> root@knet-cs-gen-01i:~# ip netns exec<br>
>>>>> qdhcp-c9c6a895-8bc1-4319-a207-30422d0d1a27 netstat -rn<br>
>>>>> Kernel IP routing table<br>
>>>>> Destination Gateway Genmask Flags MSS Window<br>
>>>>> irtt Iface<br>
>>>>> 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0<br>
>>>>> 0 tapca2d1ff4-7b<br>
>>>>><br>
>>>>> root@knet-cs-gen-01i:~# ip netns exec<br>
>>>>> qrouter-ddd535ad-debb-4810-bc10-f419f105c959 netstat -rn<br>
>>>>> Kernel IP routing table<br>
>>>>> Destination Gateway Genmask Flags MSS Window<br>
>>>>> irtt Iface<br>
>>>>> 0.0.0.0 10.21.164.1 0.0.0.0 UG 0 0<br>
>>>>> 0 qg-81523176-e1<br>
>>>>> 10.21.164.0 0.0.0.0 255.255.252.0 U 0 0<br>
>>>>> 0 qg-81523176-e1<br>
>>>>> 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0<br>
>>>>> 0 qr-973ae179-06<br>
>>>>><br>
>>>>><br>
>>>>> And yes, I have nets set up. From the controller node:<br>
>>>>><br>
>>>>> root@kcon-cs-gen-01i:/etc/init.d# nova --os-username=user_one<br>
>>>>> --os-password=user_one --os-tenant-name=project_one list | grep ACT<br>
>>>>> | 12c37be6-14e3-471b-aae3-750af8cfca32 | server-02 | ACTIVE |<br>
>>>>> net_proj_one=192.168.1.3 |<br>
>>>>><br>
>>>>> root@kcon-cs-gen-01i:/etc/init.d# quantum net-list | grep _<br>
>>>>> | 654a49a3-f042-45eb-a937-0dcd6fcaa84c | ext_net |<br>
>>>>> 842a2baa-829c-4daa-8d3d-77dace5fba86 |<br>
>>>>> | c9c6a895-8bc1-4319-a207-30422d0d1a27 | net_proj_one |<br>
>>>>> 4ecef580-27ad-44f1-851d-c2c4cd64d323 |<br>
>>>>><br>
>>>>> Any light bulbs appearing over your head? :)<br>
>>>>><br>
>>>>><br>
>>>>> On Wed, Feb 20, 2013 at 11:51 AM, Anil Vishnoi <<a href="mailto:vishnoianil@gmail.com">vishnoianil@gmail.com</a>><br>
>>>>> wrote:<br>
>>>>>><br>
>>>>>> I am assuming this output is from compute node, but i don't see any<br>
>>>>>> tapX device attached to your "br-int" bridge.<br>
>>>>>><br>
>>>>>> If you are running VM instance on this compute node then it should be<br>
>>>>>> connected to br-int through tapX device. Quantum automatically does that if<br>
>>>>>> you spawn VM and specify network to which you want to connect this VM.<br>
>>>>>><br>
>>>>>> If you fire following command<br>
>>>>>><br>
>>>>>> ~# quantum net-list<br>
>>>>>><br>
>>>>>> do you see any network entry in the output ?<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On Wed, Feb 20, 2013 at 10:05 PM, Greg Chavez <<a href="mailto:greg.chavez@gmail.com">greg.chavez@gmail.com</a>><br>
>>>>>> wrote:<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> Here's the last command output you asked for:<br>
>>>>>>><br>
>>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows<br>
>>>>>>> br-eth1<br>
>>>>>>> NXST_FLOW reply (xid=0x4):<br>
>>>>>>> cookie=0x0, duration=78793.694s, table=0, n_packets=6, n_bytes=468,<br>
>>>>>>> priority=2,in_port=6 actions=drop<br>
>>>>>>> cookie=0x0, duration=78794.033s, table=0, n_packets=17355,<br>
>>>>>>> n_bytes=3335788, priority=1 actions=NORMAL<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Wed, Feb 20, 2013 at 10:57 AM, Greg Chavez <<a href="mailto:greg.chavez@gmail.com">greg.chavez@gmail.com</a>><br>
>>>>>>> wrote:<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> Hey Anil, thanks for responding. Here's the output:<br>
>>>>>>>><br>
>>>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-vsctl show<br>
>>>>>>>> 9d9f7949-2b80-40c8-a9e0-6a116200ed96<br>
>>>>>>>> Bridge br-int<br>
>>>>>>>> Port br-int<br>
>>>>>>>> Interface br-int<br>
>>>>>>>> type: internal<br>
>>>>>>>> Port "int-br-eth1"<br>
>>>>>>>> Interface "int-br-eth1"<br>
>>>>>>>> Bridge "br-eth1"<br>
>>>>>>>> Port "phy-br-eth1"<br>
>>>>>>>> Interface "phy-br-eth1"<br>
>>>>>>>> Port "br-eth1"<br>
>>>>>>>> Interface "br-eth1"<br>
>>>>>>>> type: internal<br>
>>>>>>>> Port "eth1"<br>
>>>>>>>> Interface "eth1"<br>
>>>>>>>> ovs_version: "1.4.3"<br>
>>>>>>>><br>
>>>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-dpctl show<br>
>>>>>>>> system@br-eth1:<br>
>>>>>>>> lookups: hit:5227 missed:24022 lost:0<br>
>>>>>>>> flows: 1<br>
>>>>>>>> port 0: br-eth1 (internal)<br>
>>>>>>>> port 1: eth1<br>
>>>>>>>> port 6: phy-br-eth1<br>
>>>>>>>> system@br-int:<br>
>>>>>>>> lookups: hit:2994 missed:13754 lost:0<br>
>>>>>>>> flows: 1<br>
>>>>>>>> port 0: br-int (internal)<br>
>>>>>>>> port 2: int-br-eth1<br>
>>>>>>>><br>
>>>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# brctl show<br>
>>>>>>>> bridge name bridge id STP enabled interfaces<br>
>>>>>>>> br-eth1 0000.bc305befedd1 no eth1<br>
>>>>>>>> phy-br-eth1<br>
>>>>>>>> br-int 0000.8ae31e5f7941 no int-br-eth1<br>
>>>>>>>> qbr5334a0cb-64 8000.76fb293fe9cf no qvb5334a0cb-64<br>
>>>>>>>> vnet0<br>
>>>>>>>><br>
>>>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows<br>
>>>>>>>> br-int<br>
>>>>>>>> NXST_FLOW reply (xid=0x4):<br>
>>>>>>>> cookie=0x0, duration=75251.156s, table=0, n_packets=16581,<br>
>>>>>>>> n_bytes=3186436, priority=2,in_port=2 actions=drop<br>
>>>>>>>> cookie=0x0, duration=75251.527s, table=0, n_packets=0, n_bytes=0,<br>
>>>>>>>> priority=1 actions=NORMAL<br>
>>>>>>>><br>
>>>>>>>> Thanks! I have until tomorrow to get this working, then my boss has<br>
>>>>>>>> mandating that I try Cloudstack. Argh, but I'm so close!<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> On Wed, Feb 20, 2013 at 10:29 AM, Anil Vishnoi<br>
>>>>>>>> <<a href="mailto:vishnoianil@gmail.com">vishnoianil@gmail.com</a>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> Hi Greg,<br>
>>>>>>>>><br>
>>>>>>>>> Can you paste the output of following command from your compute<br>
>>>>>>>>> node.<br>
>>>>>>>>><br>
>>>>>>>>> ovs-vsctl show<br>
>>>>>>>>> ovs-dpctl show<br>
>>>>>>>>> brctl show<br>
>>>>>>>>><br>
>>>>>>>>> ovs-ofctl dump-flows br-int<br>
>>>>>>>>> ovs-ofctl dump-flows br-eth1<br>
>>>>>>>>><br>
>>>>>>>>> Because i think first issue we need to resolve here is why DHCP<br>
>>>>>>>>> packet is not leaving your compute host.<br>
>>>>>>>>><br>
>>>>>>>>> Anil<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> On Wed, Feb 20, 2013 at 6:48 AM, Greg Chavez<br>
>>>>>>>>> <<a href="mailto:greg.chavez@gmail.com">greg.chavez@gmail.com</a>> wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> From my perspective, it seems that the OVS bridges are not being<br>
>>>>>>>>>> brought up correctly. As you can see in my earlier post, the integration<br>
>>>>>>>>>> bridge (int) and the physical interface bridges (br-ex and br-eth1) are<br>
>>>>>>>>>> downed. I've tried to bring them up in promiscuous mode in the case of<br>
>>>>>>>>>> br-int, and with the physical interfaces ported to the bridge in the case of<br>
>>>>>>>>>> br-ex and br-eth1. I've had no luck unfortunately.<br>
>>>>>>>>>><br>
>>>>>>>>>> It seems that nothing is getting past br-int. I can see BOOTP<br>
>>>>>>>>>> packets on the VM side of br-int, and I can see VTP packets on the physical<br>
>>>>>>>>>> side of br-int. But that's where it ends.<br>
>>>>>>>>>><br>
>>>>>>>>>> For example, when I reboot my VM, I see this:<br>
>>>>>>>>>><br>
>>>>>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# tcpdump -i<br>
>>>>>>>>>> qvo5334a0cb-64<br>
>>>>>>>>>><br>
>>>>>>>>>> tcpdump: WARNING: qvo5334a0cb-64: no IPv4 address assigned<br>
>>>>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full<br>
>>>>>>>>>> protocol decode<br>
>>>>>>>>>> listening on qvo5334a0cb-64, link-type EN10MB (Ethernet), capture<br>
>>>>>>>>>> size 65535 bytes<br>
>>>>>>>>>><br>
>>>>>>>>>> 13:42:08.099061 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:<br>
>>>>>>>>>> BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280<br>
>>>>>>>>>> 13:42:08.101675 IP6 :: > ff02::16: HBH ICMP6, multicast listener<br>
>>>>>>>>>> report v2, 1 group record(s), length 28<br>
>>>>>>>>>> 13:42:08.161728 IP6 :: > ff02::1:ff06:4809: ICMP6, neighbor<br>
>>>>>>>>>> solicitation, who has fe80::f816:3eff:fe06:4809, length 24<br>
>>>>>>>>>> 13:42:08.373745 IP6 :: > ff02::16: HBH ICMP6, multicast listener<br>
>>>>>>>>>> report v2, 1 group record(s), length 28<br>
>>>>>>>>>> 13:42:11.102528 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:<br>
>>>>>>>>>> BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280<br>
>>>>>>>>>> 13:42:14.105850 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:<br>
>>>>>>>>>> BOOTP/DHCP, Request from fa:16:3e:06:48:09 (oui Unknown), length 280<br>
>>>>>>>>>><br>
>>>>>>>>>> But that's as far as it goes. The dhcp agent never get this.<br>
>>>>>>>>>><br>
>>>>>>>>>> I"ve tried deleting and recreating the bridges, rebooting the<br>
>>>>>>>>>> systems, but nothing seems to work. Maybe it's just the right combination<br>
>>>>>>>>>> of things. I don't know.<br>
>>>>>>>>>><br>
>>>>>>>>>> Help!<br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> On Tue, Feb 19, 2013 at 5:23 AM, Sylvain Bauza<br>
>>>>>>>>>> <<a href="mailto:sylvain.bauza@digimind.com">sylvain.bauza@digimind.com</a>> wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hi Greg,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> I did have trouble with DHCP assignation (see my previous post in<br>
>>>>>>>>>>> this list), which was being fixed by deleting ovs bridges on network node,<br>
>>>>>>>>>>> recreating them and restarting OVS plugin and L3/DHCP agents (which were all<br>
>>>>>>>>>>> on the same physical node).<br>
>>>>>>>>>>> Maybe it helps.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Anyway, when DHCP'ing from your VM (asking for an IP), could you<br>
>>>>>>>>>>> please tcpdump :<br>
>>>>>>>>>>> 1. your virtual network interface on compute node<br>
>>>>>>>>>>> 2. your physical network interface on compute node<br>
>>>>>>>>>>> 3. your physical network interface on network node<br>
>>>>>>>>>>><br>
>>>>>>>>>>> and see BOOTP/DHCP packets ?<br>
>>>>>>>>>>> On the physical layer, you should see GRE packets (provided you<br>
>>>>>>>>>>> correctly followed the mentioned guide) encapsulating your BOOTP/DHCP<br>
>>>>>>>>>>> packets.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> If that's OK, could you please issue the below commands (on the<br>
>>>>>>>>>>> network node) :<br>
>>>>>>>>>>> - brctl show<br>
>>>>>>>>>>> - ip a<br>
>>>>>>>>>>> - ovs-vsctl show<br>
>>>>>>>>>>> - route -n<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Thanks,<br>
>>>>>>>>>>> -Sylvain<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Le 19/02/2013 00:55, Greg Chavez a écrit :<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Third time I'm replying to my own message. It seems like the<br>
>>>>>>>>>>> initial network state is a problem for many first time openstackers. Surely<br>
>>>>>>>>>>> somewhere would be well to assist me. I'm running out of time to make this<br>
>>>>>>>>>>> work. Thanks.<br>
>>>>>>>>>>><br>
>>>>>>>>>>><br>
>>>>>>>>>>> On Sun, Feb 17, 2013 at 3:08 AM, Greg Chavez<br>
>>>>>>>>>>> <<a href="mailto:greg.chavez@gmail.com">greg.chavez@gmail.com</a>> wrote:<br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> I'm replying to my own message because I'm desperate. My<br>
>>>>>>>>>>>> network situation is a mess. I need to add this as well: my bridge<br>
>>>>>>>>>>>> interfaces are all down. On my compute node:<br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances/instance-00000005# ip<br>
>>>>>>>>>>>> addr show | grep ^[0-9]<br>
>>>>>>>>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state<br>
>>>>>>>>>>>> UNKNOWN<br>
>>>>>>>>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq<br>
>>>>>>>>>>>> state UP qlen 1000<br>
>>>>>>>>>>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq<br>
>>>>>>>>>>>> state UP qlen 1000<br>
>>>>>>>>>>>> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>>>>>>>>>>> qlen 1000<br>
>>>>>>>>>>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>>>>>>>>>>> qlen 1000<br>
>>>>>>>>>>>> 9: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>>>>>>>>>>> 10: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state<br>
>>>>>>>>>>>> DOWN<br>
>>>>>>>>>>>> 13: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc pfifo_fast state UP qlen 1000<br>
>>>>>>>>>>>> 14: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc pfifo_fast state UP qlen 1000<br>
>>>>>>>>>>>> 15: qbre56c5d9e-b6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc noqueue state UP<br>
>>>>>>>>>>>> 16: qvoe56c5d9e-b6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP><br>
>>>>>>>>>>>> mtu 1500 qdisc pfifo_fast state UP qlen 1000<br>
>>>>>>>>>>>> 17: qvbe56c5d9e-b6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP><br>
>>>>>>>>>>>> mtu 1500 qdisc pfifo_fast master qbre56c5d9e-b6 state UP qlen 1000<br>
>>>>>>>>>>>> 19: qbrb805a9c9-11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc noqueue state UP<br>
>>>>>>>>>>>> 20: qvob805a9c9-11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP><br>
>>>>>>>>>>>> mtu 1500 qdisc pfifo_fast state UP qlen 1000<br>
>>>>>>>>>>>> 21: qvbb805a9c9-11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP><br>
>>>>>>>>>>>> mtu 1500 qdisc pfifo_fast master qbrb805a9c9-11 state UP qlen 1000<br>
>>>>>>>>>>>> 34: qbr2b23c51f-02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc noqueue state UP<br>
>>>>>>>>>>>> 35: qvo2b23c51f-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP><br>
>>>>>>>>>>>> mtu 1500 qdisc pfifo_fast state UP qlen 1000<br>
>>>>>>>>>>>> 36: qvb2b23c51f-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP><br>
>>>>>>>>>>>> mtu 1500 qdisc pfifo_fast master qbr2b23c51f-02 state UP qlen 1000<br>
>>>>>>>>>>>> 37: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
>>>>>>>>>>>> pfifo_fast master qbr2b23c51f-02 state UNKNOWN qlen 500<br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> And on my network node:<br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> root@knet-cs-gen-01i:~# ip addr show | grep ^[0-9]<br>
>>>>>>>>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state<br>
>>>>>>>>>>>> UNKNOWN<br>
>>>>>>>>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq<br>
>>>>>>>>>>>> state UP qlen 1000<br>
>>>>>>>>>>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq<br>
>>>>>>>>>>>> state UP qlen 1000<br>
>>>>>>>>>>>> 4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc mq state UP qlen 1000<br>
>>>>>>>>>>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>>>>>>>>>>> qlen 1000<br>
>>>>>>>>>>>> 6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>>>>>>>>>>> 7: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN<br>
>>>>>>>>>>>> 8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
>>>>>>>>>>>> noqueue state UNKNOWN<br>
>>>>>>>>>>>> 22: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc pfifo_fast state UP qlen 1000<br>
>>>>>>>>>>>> 23: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<br>
>>>>>>>>>>>> qdisc pfifo_fast state UP qlen 1000<br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> I gave br-ex an IP and UP'ed it manually. I assume this is<br>
>>>>>>>>>>>> correct. By I honestly don't know.<br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> Thanks.<br>
>>>>>>>>>>>><br>
>>>>>>>>>>>><br>
>>>>>>>>>>>><br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> On Fri, Feb 15, 2013 at 6:54 PM, Greg Chavez<br>
>>>>>>>>>>>> <<a href="mailto:greg.chavez@gmail.com">greg.chavez@gmail.com</a>> wrote:<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> Sigh. So I abandoned RHEL 6.3, rekicked my systems and set up<br>
>>>>>>>>>>>>> the scale-ready installation described in these instructions:<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> <a href="https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst" target="_blank">https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst</a><br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> Basically:<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> (o) controller node on a mgmt and public net<br>
>>>>>>>>>>>>> (o) network node (quantum and openvs) on a mgmt, net-config,<br>
>>>>>>>>>>>>> and public net<br>
>>>>>>>>>>>>> (o) compute node is on a mgmt and net-config net<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> Took me just over an hour and ran into only a few easily-fixed<br>
>>>>>>>>>>>>> speed bumps. But the VM networks are totally non-functioning. VMs launch<br>
>>>>>>>>>>>>> but no network traffic can go in or out.<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> I'm particularly befuddled by these problems:<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> ( 1 ) This error in nova-compute:<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> ERROR nova.network.quantumv2 [-] _get_auth_token() failed<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> ( 2 ) No NAT rules on the compute node, which probably explains<br>
>>>>>>>>>>>>> why the VMs complain about not finding a network or being able to get<br>
>>>>>>>>>>>>> metadata from 169.254.169.254.<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> root@kvm-cs-sn-10i:~# iptables -t nat -S<br>
>>>>>>>>>>>>> -P PREROUTING ACCEPT<br>
>>>>>>>>>>>>> -P INPUT ACCEPT<br>
>>>>>>>>>>>>> -P OUTPUT ACCEPT<br>
>>>>>>>>>>>>> -P POSTROUTING ACCEPT<br>
>>>>>>>>>>>>> -N nova-api-metadat-OUTPUT<br>
>>>>>>>>>>>>> -N nova-api-metadat-POSTROUTING<br>
>>>>>>>>>>>>> -N nova-api-metadat-PREROUTING<br>
>>>>>>>>>>>>> -N nova-api-metadat-float-snat<br>
>>>>>>>>>>>>> -N nova-api-metadat-snat<br>
>>>>>>>>>>>>> -N nova-compute-OUTPUT<br>
>>>>>>>>>>>>> -N nova-compute-POSTROUTING<br>
>>>>>>>>>>>>> -N nova-compute-PREROUTING<br>
>>>>>>>>>>>>> -N nova-compute-float-snat<br>
>>>>>>>>>>>>> -N nova-compute-snat<br>
>>>>>>>>>>>>> -N nova-postrouting-bottom<br>
>>>>>>>>>>>>> -A PREROUTING -j nova-api-metadat-PREROUTING<br>
>>>>>>>>>>>>> -A PREROUTING -j nova-compute-PREROUTING<br>
>>>>>>>>>>>>> -A OUTPUT -j nova-api-metadat-OUTPUT<br>
>>>>>>>>>>>>> -A OUTPUT -j nova-compute-OUTPUT<br>
>>>>>>>>>>>>> -A POSTROUTING -j nova-api-metadat-POSTROUTING<br>
>>>>>>>>>>>>> -A POSTROUTING -j nova-compute-POSTROUTING<br>
>>>>>>>>>>>>> -A POSTROUTING -j nova-postrouting-bottom<br>
>>>>>>>>>>>>> -A nova-api-metadat-snat -j nova-api-metadat-float-snat<br>
>>>>>>>>>>>>> -A nova-compute-snat -j nova-compute-float-snat<br>
>>>>>>>>>>>>> -A nova-postrouting-bottom -j nova-api-metadat-snat<br>
>>>>>>>>>>>>> -A nova-postrouting-bottom -j nova-compute-snat<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> (3) A lastly, no default secgroup rules, whose function<br>
>>>>>>>>>>>>> governs... what exactly? Connections to the VM's public or private IPs? I<br>
>>>>>>>>>>>>> guess I'm just not sure if this is relevant to my overall problem of ZERO VM<br>
>>>>>>>>>>>>> network connectivity.<br>
>>>>>>>>>>>>><br>
>>>>>>>>>>>>> I seek guidance please. Thanks.<br>
>>>>>>>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> _______________________________________________<br>
>>>>>>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
>>>>>>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>\*..+.-<br>--Greg Chavez<br>+//..;};
</div></div>