[Openstack] Neutron Issues...
Qin, Xiaohong
Xiaohong.Qin at emc.com
Wed Apr 16 17:12:18 UTC 2014
Hi Erich,
That DOWN status output from ovs-vsctl show is confusing, perhaps it meant to say that the interface is sort of "on demand" consumption that if there is a traffic it will be on. What you can do is to session into the dhcp namespace and run tcpdump on that interface there. Here are the steps,
1. sudo ip netns exec <your qdhcp namespace> bash
2. sudo tcpdump -n -e -vv -i <your dhcp server tap interface>
Then on your controller node, create a new VM, you should see some packets dump on your dhcp namespace, similar to this,
09:43:22.710102 fa:16:3e:a9:ba:20 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ffa9:ba20 to_ex { }]
09:43:22.842593 fa:16:3e:a9:ba:20 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from fa:16:3e:a9:ba:20, length 300, xid 0x3d5e463e, Flags [none] (0x0000)
Client-Ethernet-Address fa:16:3e:a9:ba:20
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Requested-IP Option 50, length 4: 192.168.160.154
Hostname Option 12, length 13: "openstacktest"
Parameter-Request Option 55, length 13:
Subnet-Mask, BR, Time-Zone, Default-Gateway
Domain-Name, Domain-Name-Server, Option 119, Hostname
Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
NTP
09:43:22.843152 fa:16:3e:e6:a6:7f > fa:16:3e:a9:ba:20, ethertype IPv4 (0x0800), length 350: (tos 0xc0, ttl 64, id 35983, offset 0, flags [none], proto UDP (17), length 336)
10.0.0.3.67 > 10.0.0.9.68: [bad udp cksum 0x1559 -> 0xab30!] BOOTP/DHCP, Reply, length 308, xid 0x3d5e463e, Flags [none] (0x0000)
Your-IP 10.0.0.9
Server-IP 10.0.0.3
Client-Ethernet-Address fa:16:3e:a9:ba:20
Here is my dnsmasq server process,
nobody 12782 1 0 Apr15 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7e1faebd-1c --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/90a8a84a-d4dd-4c98-bf87-492342c55b94/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/90a8a84a-d4dd-4c98-bf87-492342c55b94/host --dhcp-optsfile=/opt/stack/data/neutron/dhcp/90a8a84a-d4dd-4c98-bf87-492342c55b94/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
Hope that helps.
Dennis Qin (aka Xiaohong Qin)
-----Original Message-----
From: Erich Weiler [mailto:weiler at soe.ucsc.edu]
Sent: Tuesday, April 15, 2014 4:56 PM
To: openstack
Subject: Re: [Openstack] Neutron Issues...
I think I've identified a problem - when I create a new network and subnet via [neutron net-create and neutron subnet-create], it creates the networks fine, and then I start the dhcp service on the network node:
/etc/init.d/neutron-dhcp-agent start
It seems to add a tap interface to br-int:
# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c2864f59274e n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
1(tapbd844c9f-90): addr:4a:01:00:00:00:00 <------ ADDS THIS
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
4(int-br-eth2): addr:52:76:9c:cb:4a:47
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
5(int-br-ex): addr:2e:d8:0d:0b:b9:e8
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-int): addr:c2:86:4f:59:27:4e
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
But as you note, the link is down. I would expect a corresponding interface called "tapbd844c9f-90" be created when I do a simple "ifconfig" on the network node, but it's not there:
# ifconfig
br-eth2 Link encap:Ethernet HWaddr 00:14:4F:CB:26:F0
inet6 addr: fe80::68a6:5ff:fe23:18e8/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:354 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:27568 (26.9 KiB) TX bytes:468 (468.0 b)
br-ex Link encap:Ethernet HWaddr 00:14:4F:CB:26:EF
inet6 addr: fe80::214:4fff:fecb:26ef/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:589 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:40950 (39.9 KiB) TX bytes:936 (936.0 b)
br-int Link encap:Ethernet HWaddr C2:86:4F:59:27:4E
inet6 addr: fe80::58ad:bff:feb6:2a38/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:369 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:28334 (27.6 KiB) TX bytes:468 (468.0 b)
eth0 Link encap:Ethernet HWaddr 00:14:4F:CB:26:EE
inet addr:10.1.1.145 Bcast:10.1.255.255 Mask:255.255.0.0
inet6 addr: fe80::214:4fff:fecb:26ee/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12610 errors:0 dropped:0 overruns:0 frame:0
TX packets:9659 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1431159 (1.3 MiB) TX bytes:2084840 (1.9 MiB)
Memory:fab60000-fab80000
eth1 Link encap:Ethernet HWaddr 00:14:4F:CB:26:EF
inet6 addr: fe80::214:4fff:fecb:26ef/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:3422 errors:0 dropped:0 overruns:0 frame:0
TX packets:59 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:205320 (200.5 KiB) TX bytes:9464 (9.2 KiB)
Memory:fabe0000-fac00000
eth2 Link encap:Ethernet HWaddr 00:14:4F:CB:26:F0
inet6 addr: fe80::214:4fff:fecb:26f0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:25018 errors:0 dropped:0 overruns:0 frame:0
TX packets:344 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1801516 (1.7 MiB) TX bytes:19639 (19.1 KiB)
Memory:fac60000-fac80000
int-br-eth2 Link encap:Ethernet HWaddr 52:76:9C:CB:4A:47
inet6 addr: fe80::5076:9cff:fecb:4a47/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:468 (468.0 b) TX bytes:468 (468.0 b)
int-br-ex Link encap:Ethernet HWaddr 2E:D8:0D:0B:B9:E8
inet6 addr: fe80::2cd8:dff:fe0b:b9e8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:21 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1368 (1.3 KiB) TX bytes:468 (468.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
phy-br-eth2 Link encap:Ethernet HWaddr CA:37:ED:2E:1A:57
inet6 addr: fe80::c837:edff:fe2e:1a57/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:468 (468.0 b) TX bytes:468 (468.0 b)
phy-br-ex Link encap:Ethernet HWaddr 22:F0:60:46:26:FD
inet6 addr: fe80::20f0:60ff:fe46:26fd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:468 (468.0 b) TX bytes:1368 (1.3 KiB)
I think becasue it's not there, that's why "ovs-ofctl show br-int" shows the link is down. Does anyone know why it doesn't create that interface? Or if I can create it manually...?
On 04/14/14 14:09, Erich Weiler wrote:
> I think I'm getting a little closer.. I see that on my neutron
> network node my DHCP server (dnsmasq) is on 10.200.0.3:
>
> # ip netns
> qrouter-eefaa5c8-95fc-4b3d-a8d2-27eebc449337
> qdhcp-e4448083-ec61-4293-ad0e-62239986965f
>
> # ip netns exec qdhcp-e4448083-ec61-4293-ad0e-62239986965f ifconfig
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:7 errors:0 dropped:0 overruns:0 frame:0
> TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:1158 (1.1 KiB) TX bytes:1158 (1.1 KiB)
>
> tapc5195287-8f Link encap:Ethernet HWaddr FA:16:3E:C2:EC:B5
> inet addr:10.200.0.3 Bcast:10.200.255.255 Mask:255.255.0.0
> inet6 addr: fe80::f816:3eff:fec2:ecb5/64 Scope:Link
> UP BROADCAST RUNNING MTU:1500 Metric:1
> RX packets:10769 errors:0 dropped:0 overruns:0 frame:0
> TX packets:10799 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:647044 (631.8 KiB) TX bytes:454965 (444.3 KiB)
>
> But when I look at my OVS state on the same neutron node:
>
> # ovs-ofctl show br-int
>
> OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c2864f59274e n_tables:254,
> n_buffers:256
> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS
> ARP_MATCH_IP
> actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC
> SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
> 1(int-br-eth2): addr:0a:0d:73:7f:91:8f
> config: 0
> state: 0
> current: 10GB-FD COPPER
> speed: 10000 Mbps now, 0 Mbps max
> 2(int-br-ex): addr:d6:10:30:bf:67:15
> config: 0
> state: 0
> current: 10GB-FD COPPER
> speed: 10000 Mbps now, 0 Mbps max
> 3(qr-b87526f1-3d): addr:c6:25:03:f0:51:08
> config: 0
> state: 0
> speed: 0 Mbps now, 0 Mbps max
> 4(tap5a3a1ef2-9c): addr:3e:63:c2:19:54:6a
> config: 0
> state: 0
> speed: 0 Mbps now, 0 Mbps max
> 5(tapc5195287-8f): addr:af:00:00:00:00:00 <--- HUH??
> config: PORT_DOWN
> state: LINK_DOWN
> speed: 0 Mbps now, 0 Mbps max
> LOCAL(br-int): addr:c2:86:4f:59:27:4e
> config: 0
> state: 0
> speed: 0 Mbps now, 0 Mbps max
> OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
>
> It shows the tapc5195287-8f port, the one dnsmasq is on, is PORT_DOWN,
> LINK_DOWN. I wonder why that is? The earlier "ip netns ifconfig" on
> the dnsmasq port showed status "UP". Anyone see anything I'm missing?
>
> Thanks,
> erich
>
> On 04/14/14 11:30, Erich Weiler wrote:
>> And additionally here's my "iptables -L" on the compute node with the
>> VM running on it, if anyone can see anything wrong with it that might
>> block ARP responses... I didn't modify this manually, looks like
>> Open vSwitch did all the tweaking. Remember 10.200.0.3 is the
>> dnsmasq server and
>> 10.200.0.6 is the VM:
>>
>> # iptables -L
>> Chain INPUT (policy ACCEPT)
>> target prot opt source destination
>> neutron-openvswi-INPUT all -- anywhere anywhere
>> ACCEPT udp -- anywhere anywhere udp
>> dpt:domain
>> ACCEPT tcp -- anywhere anywhere tcp
>> dpt:domain
>> ACCEPT udp -- anywhere anywhere udp
>> dpt:bootps
>> ACCEPT tcp -- anywhere anywhere tcp
>> dpt:bootps
>>
>> Chain FORWARD (policy ACCEPT)
>> target prot opt source destination
>> neutron-filter-top all -- anywhere anywhere
>> neutron-openvswi-FORWARD all -- anywhere anywhere
>> ACCEPT all -- anywhere 192.168.122.0/24 state
>> RELATED,ESTABLISHED
>> ACCEPT all -- 192.168.122.0/24 anywhere
>> ACCEPT all -- anywhere anywhere
>> REJECT all -- anywhere anywhere reject-with
>> icmp-port-unreachable
>> REJECT all -- anywhere anywhere reject-with
>> icmp-port-unreachable
>>
>> Chain OUTPUT (policy ACCEPT)
>> target prot opt source destination
>> neutron-filter-top all -- anywhere anywhere
>> neutron-openvswi-OUTPUT all -- anywhere anywhere
>>
>> Chain neutron-filter-top (2 references)
>> target prot opt source destination
>> neutron-openvswi-local all -- anywhere anywhere
>>
>> Chain neutron-openvswi-FORWARD (1 references)
>> target prot opt source destination
>> neutron-openvswi-sg-chain all -- anywhere anywhere
>> PHYSDEV match --physdev-out tapb2c5d867-34 --physdev-is-bridged
>> neutron-openvswi-sg-chain all -- anywhere anywhere
>> PHYSDEV match --physdev-in tapb2c5d867-34 --physdev-is-bridged
>>
>> Chain neutron-openvswi-INPUT (1 references)
>> target prot opt source destination
>> neutron-openvswi-ob2c5d867-3 all -- anywhere anywhere
>> PHYSDEV match --physdev-in tapb2c5d867-34
>> --physdev-is-bridged
>>
>> Chain neutron-openvswi-OUTPUT (1 references)
>> target prot opt source destination
>>
>> Chain neutron-openvswi-ib2c5d867-3 (1 references)
>> target prot opt source destination
>> DROP all -- anywhere anywhere state
>> INVALID
>> RETURN all -- anywhere anywhere state
>> RELATED,ESTABLISHED
>> RETURN udp -- 10.200.0.3 anywhere udp
>> spt:bootps dpt:bootpc
>> neutron-openvswi-sg-fallback all -- anywhere anywhere
>>
>> Chain neutron-openvswi-local (1 references)
>> target prot opt source destination
>>
>> Chain neutron-openvswi-ob2c5d867-3 (2 references)
>> target prot opt source destination
>> RETURN udp -- anywhere anywhere udp
>> spt:bootpc dpt:bootps
>> neutron-openvswi-sb2c5d867-3 all -- anywhere anywhere
>> DROP udp -- anywhere anywhere udp
>> spt:bootps dpt:bootpc
>> DROP all -- anywhere anywhere state
>> INVALID
>> RETURN all -- anywhere anywhere state
>> RELATED,ESTABLISHED
>> RETURN all -- anywhere anywhere
>> neutron-openvswi-sg-fallback all -- anywhere anywhere
>>
>> Chain neutron-openvswi-sb2c5d867-3 (1 references)
>> target prot opt source destination
>> RETURN all -- 10.200.0.6 anywhere MAC
>> FA:16:3E:5D:11:5B
>> DROP all -- anywhere anywhere
>>
>> Chain neutron-openvswi-sg-chain (2 references)
>> target prot opt source destination
>> neutron-openvswi-ib2c5d867-3 all -- anywhere anywhere
>> PHYSDEV match --physdev-out tapb2c5d867-34 --physdev-is-bridged
>> neutron-openvswi-ob2c5d867-3 all -- anywhere anywhere
>> PHYSDEV match --physdev-in tapb2c5d867-34 --physdev-is-bridged
>> ACCEPT all -- anywhere anywhere
>>
>> Chain neutron-openvswi-sg-fallback (2 references)
>> target prot opt source destination
>> DROP all -- anywhere anywhere
>>
>>
>> On 04/14/14 11:06, Erich Weiler wrote:
>>> No one can see anything wrong?
>>>
>>> Here's "ovs-vsctl show" on the neutron network node, if it helps:
>>>
>>> # ovs-vsctl show
>>> 52702cef-6433-4627-ade8-51561b4e8126
>>> Bridge "br-eth2"
>>> Port "phy-br-eth2"
>>> Interface "phy-br-eth2"
>>> Port "eth2"
>>> Interface "eth2"
>>> Port "br-eth2"
>>> Interface "br-eth2"
>>> type: internal
>>> Bridge br-int
>>> Port int-br-ex
>>> Interface int-br-ex
>>> Port br-int
>>> Interface br-int
>>> type: internal
>>> Port "qr-b87526f1-3d"
>>> tag: 4095
>>> Interface "qr-b87526f1-3d"
>>> type: internal
>>> Port "int-br-eth2"
>>> Interface "int-br-eth2"
>>> Port "tapc5195287-8f"
>>> tag: 1
>>> Interface "tapc5195287-8f"
>>> type: internal
>>> Port "tap5a3a1ef2-9c"
>>> tag: 4095
>>> Interface "tap5a3a1ef2-9c"
>>> type: internal
>>> Bridge br-ex
>>> Port phy-br-ex
>>> Interface phy-br-ex
>>> Port "qg-0b1ba3a7-16"
>>> Interface "qg-0b1ba3a7-16"
>>> type: internal
>>> Port "eth1"
>>> Interface "eth1"
>>> Port br-ex
>>> Interface br-ex
>>> type: internal
>>> ovs_version: "1.11.0"
>>>
>>> On 04/12/14 15:26, Erich Weiler wrote:
>>>> And one more, I think I found something. On the network node, I
>>>> see on
>>>> phy-br-eth2:
>>>>
>>>> # tcpdump -n -e -vv -i phy-br-eth2
>>>>
>>>> tcpdump: WARNING: phy-br-eth2: no IPv4 address assigned
>>>> tcpdump: listening on phy-br-eth2, link-type EN10MB (Ethernet),
>>>> capture size 65535 bytes
>>>> 15:23:47.394584 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len
>>>> 6),
>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>> 15:23:47.394605 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>> 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet
>>>> (len 6), IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5,
>>>> length 28
>>>> 15:23:48.394677 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len
>>>> 6),
>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>> 15:23:48.394693 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>> 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet
>>>> (len 6), IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5,
>>>> length 28
>>>> 15:23:49.395364 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len
>>>> 6),
>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>> 15:23:49.395379 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>> 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet
>>>> (len 6), IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5,
>>>> length 28
>>>>
>>>> It sees the ARP request on VLAN 200, but it looks like it replies
>>>> on VLAN 1. I'm not sure why it would reply on VLAN 1... I'm not
>>>> entirely sure how phy-br-eth2 works though...
>>>>
>>>> On 4/12/14, 2:07 PM, Erich Weiler wrote:
>>>>> Sorry, one more... ;)
>>>>>
>>>>> Wile trying to ping the dnsmasq host (10.200.0.3) from the VM
>>>>> (statically configured to 10.200.0.6), if I tcpdump specifically
>>>>> on vlan
>>>>> 200 on the network node:
>>>>>
>>>>> # tcpdump -ni eth2 vlan 200
>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>> tcpdump: verbose output suppressed, use -v or -vv for full
>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),
>>>>> capture size 65535 bytes
>>>>> 13:55:18.826384 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:18.956941 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:55:19.825947 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:20.826056 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:20.960779 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:55:21.826718 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:22.825870 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:22.976173 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:55:23.825964 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:24.826730 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:24.981795 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>
>>>>> I see the ARP requests coming in. But I don't see any replies, at
>>>>> least on vlan 200. If I tcpdump with a specific VLAN, I do see
>>>>> the reply on the network node:
>>>>>
>>>>> # tcpdump -i eth2
>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>> tcpdump: verbose output suppressed, use -v or -vv for full
>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),
>>>>> capture size 65535 bytes
>>>>> 13:58:11.830842 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:58:11.830878 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>> Unknown), length 28
>>>>> 13:58:12.833434 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:58:12.833457 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>> Unknown), length 28
>>>>> 13:58:13.494885 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.495217 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.495577 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.496017 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.496355 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>
>>>>> So maybe the ARP reply is being sent out, but maybe just not over
>>>>> VLAN
>>>>> 200 where it needs to go, very strange...
>>>>>
>>>>> Also more verbose tcpdump on the network node:
>>>>>
>>>>> # tcpdump -n -e -vv -i eth2
>>>>>
>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture
>>>>> size
>>>>> 65535 bytes
>>>>> 14:04:10.848783 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len
>>>>> 6),
>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length
>>>>> 46
>>>>> 14:04:10.848815 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet
>>>>> (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>> 14:04:10.918013 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.918374 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.918705 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 204, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80cc.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.919171 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 202, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80ca.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.919525 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 203, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80cb.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.919889 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 206, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80ce.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.920346 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 208, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80d0.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.920682 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 209, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80d1.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.921010 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 200, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80c8.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.921295 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 207, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80cf.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:11.287103 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cc, 802.3,
>>>>> length
>>>>> 188: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command,
>>>>> ctrl
>>>>> 0x03: oui Cisco (0x00000c), pid VTP (0x2003): VTPv1, Message Join
>>>>> message (0x04), length 166
>>>>> Domain name: cbse, Rsvd: 0
>>>>> 14:04:11.848874 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len
>>>>> 6),
>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length
>>>>> 46
>>>>> 14:04:11.848896 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet
>>>>> (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>> 14:04:12.851406 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len
>>>>> 6),
>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length
>>>>> 46
>>>>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet
>>>>> (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>> 14:04:12.923686 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:12.924039 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa)
>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco
>>>>> (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>> root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>>
>>>>> That one seems to indicate the ARP reply is being sent on vlan
>>>>> 200..??? :
>>>>>
>>>>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet
>>>>> (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>>
>>>>> I'm totally confused... ARP should be working according to that....
>>>>>
>>>>> On 4/12/14, 1:44 PM, Erich Weiler wrote:
>>>>>> I tried additionally checking the dnsmasq instance on the network
>>>>>> node while trying to ping it from the VM (I set the VMs IP
>>>>>> statically to
>>>>>> 10.200.0.6 and tried to ping dnsmasq which is 10.200.0.3). From
>>>>>> the network node, I did:
>>>>>>
>>>>>> # ip netns
>>>>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f
>>>>>>
>>>>>> [root at cloudnet-1 network-scripts]# ip netns exec
>>>>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f ifconfig
>>>>>> lo Link encap:Local Loopback
>>>>>> inet addr:127.0.0.1 Mask:255.0.0.0
>>>>>> inet6 addr: ::1/128 Scope:Host
>>>>>> UP LOOPBACK RUNNING MTU:16436 Metric:1
>>>>>> RX packets:17 errors:0 dropped:0 overruns:0 frame:0
>>>>>> TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
>>>>>> collisions:0 txqueuelen:0
>>>>>> RX bytes:3184 (3.1 KiB) TX bytes:3184 (3.1 KiB)
>>>>>>
>>>>>> tapc5195287-8f Link encap:Ethernet HWaddr FA:16:3E:C2:EC:B5
>>>>>> inet addr:10.200.0.3 Bcast:10.200.255.255
>>>>>> Mask:255.255.0.0
>>>>>> inet6 addr: fe80::f816:3eff:fec2:ecb5/64 Scope:Link
>>>>>> UP BROADCAST RUNNING MTU:1500 Metric:1
>>>>>> RX packets:2369 errors:0 dropped:0 overruns:0 frame:0
>>>>>> TX packets:2220 errors:0 dropped:0 overruns:0 carrier:0
>>>>>> collisions:0 txqueuelen:0
>>>>>> RX bytes:158390 (154.6 KiB) TX bytes:94804 (92.5 KiB)
>>>>>>
>>>>>> # ip netns exec qdhcp-e4448083-ec61-4293-ad0e-62239986965f
>>>>>> tcpdump -i tapc5195287-8f
>>>>>>
>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full
>>>>>> protocol decode listening on tapc5195287-8f, link-type EN10MB
>>>>>> (Ethernet), capture size
>>>>>> 65535 bytes
>>>>>> 13:38:03.997934 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:03.997948 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:03.999280 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:04.998033 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:04.998044 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:04.999187 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:05.999197 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:06.001111 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:06.001121 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:06.998171 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:06.998181 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:07.998279 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:07.998289 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:09.001444 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:09.001454 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:09.004258 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:09.998436 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:09.998446 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:10.004161 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:10.998488 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:10.998498 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:11.004181 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:12.001742 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:12.001752 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:12.998647 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:12.998657 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:13.998762 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:13.998772 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:14.009308 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:15.000801 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:15.000812 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:24.019260 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:24.998348 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:34.999130 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:34.999143 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:36.003134 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:36.003147 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:36.999278 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:36.999291 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:37.999362 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:37.999375 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:39.003456 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:39.003469 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:40.003526 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:40.003539 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:41.003609 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:41.003622 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:42.003783 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:42.003796 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>> 13:38:43.003781 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:43.003794 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>> (oui Unknown), length 28
>>>>>>
>>>>>> I assume because of the fact I'm seeing the ARP requests and
>>>>>> replies on the dnsmasq interface that VLANs are working, as they
>>>>>> are supposed to be talking on VLAN 200, but ARP isn't working for
>>>>>> some reason, or at least the VM isn't getting the ARP reply.
>>>>>>
>>>>>> I tried setting my data interfaces on my network node and my
>>>>>> compute node to promiscuous mode (they weren't before), but it
>>>>>> didn't seem to help. Are those interfaces supposed to be in promiscuous mode?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/12/14, 1:02 PM, Erich Weiler wrote:
>>>>>>> Hi Y'all,
>>>>>>>
>>>>>>> I'm making some progress on my neutron VLAN deployment issues,
>>>>>>> but it's still not working as expected. I have my compute nodes
>>>>>>> data port connected to a switchport that is a trunk, allowing
>>>>>>> VLANs 200-209 to flow over the trunk. The neutron node also has
>>>>>>> its internal data port on the same switch, on a trunk port, also
>>>>>>> allowing VLANs 200-209 to flow over it.
>>>>>>>
>>>>>>> I *think* I have OVS configured correctly on the neutron network
>>>>>>> node and the compute nodes.
>>>>>>>
>>>>>>> To test I just created an internal network (on physnet1) callled
>>>>>>> test-net on segmentation-id 200 (network 10.200.0.0/16). So I
>>>>>>> assume it will use VLAN 200. I then created a VM and attached
>>>>>>> it to that network.
>>>>>>> But when the VM boots, it does not get an IP address from dnsmasq.
>>>>>>>
>>>>>>> Back on the network node, I see dnsmasq running on that VLAN, I
>>>>>>> think:
>>>>>>>
>>>>>>> # ps -afe | grep masq
>>>>>>> nobody 16674 1 0 Apr10 ? 00:00:00 dnsmasq --no-hosts
>>>>>>> --no-resolv --strict-order --bind-interfaces
>>>>>>> --interface=tapc5195287-8f --except-interface=lo
>>>>>>> --pid-file=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-6223998
>>>>>>> 6965f/pid
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --dhcp-hostsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-6
>>>>>>> 2239986965f/host
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --addn-hosts=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239
>>>>>>> 986965f/addn_hosts
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --dhcp-optsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62
>>>>>>> 239986965f/opts
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --leasefile-ro --dhcp-range=tag0,10.200.0.0,static,86400s
>>>>>>> --dhcp-lease-max=65536 --conf-file= --domain=openstacklocal
>>>>>>>
>>>>>>> And I tried doing a tcpdump on eth2 (which is the data interface
>>>>>>> with the VLANs) while doing a DHCPDISCOVER from the VM:
>>>>>>>
>>>>>>> # tcpdump -i eth2
>>>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full
>>>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),
>>>>>>> capture size 65535 bytes
>>>>>>> 12:43:46.440221 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.440563 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.440896 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.441414 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.441672 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.442001 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.442502 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.442830 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.443171 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.443470 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:47.718633 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>>>>>> BOOTP/DHCP,
>>>>>>> Request from fa:16:3e:5d:11:5b (oui Unknown), length 280
>>>>>>> 12:43:47.719183 IP 10.200.0.3.bootps > 10.200.0.6.bootpc:
>>>>>>> BOOTP/DHCP,
>>>>>>> Reply, length 325
>>>>>>> 12:43:48.447761 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.448109 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.448593 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.448896 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.449266 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.449672 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.450010 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.450385 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.450678 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.451115 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:49.946348 VTPv1, Message Join message (0x04), length 166
>>>>>>> 12:43:50.459700 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.460184 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.460529 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.460867 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.461195 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.461625 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.461956 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.462335 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.462745 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.463072 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.455118 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.455591 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.455930 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.456258 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.456575 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.457053 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.457414 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.457702 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.458171 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.458503 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.718355 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,
>>>>>>> length 28
>>>>>>> 12:43:53.718209 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,
>>>>>>> length 28
>>>>>>> 12:43:54.460398 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.460871 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.461187 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.461588 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.461984 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.462361 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.462652 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.462981 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.463474 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.463751 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>>
>>>>>>> Where 10.200.0.3 is the IP address of the dnsmasq instance on
>>>>>>> the network node and 10.200.0.6 is the IP (supposedly) of the
>>>>>>> VM, from what nova tells me. It seems the traffic is getting to
>>>>>>> the network node, but the VM never gets the IP. Even if I
>>>>>>> configure the VM statically to be 10.200.0.6, I still can't ping
>>>>>>> the DHCP server on 10.200.0.3.
>>>>>>>
>>>>>>> It looks like ARP requests are getting sent out for the DHCP
>>>>>>> server hen I ping (as seen fro the network node):
>>>>>>>
>>>>>>> # tcpdump -i eth2
>>>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full
>>>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),
>>>>>>> capture size 65535 bytes
>>>>>>> 12:57:08.871232 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.871717 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.872063 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.872421 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.872819 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.873212 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.873507 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.873839 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.874329 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.874621 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:09.041413 VTPv1, Message Join message (0x04), length 166
>>>>>>> 12:57:09.289413 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:09.289435 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>>> (oui Unknown), length 28
>>>>>>> 12:57:10.289706 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:10.289728 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>>> (oui Unknown), length 28
>>>>>>> 12:57:10.876689 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.877031 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.877439 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.877843 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.878190 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.878542 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.878824 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.879282 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.879612 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.879933 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:11.289582 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:11.289603 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>>> (oui Unknown), length 28
>>>>>>> 12:57:12.289668 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:12.289689 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5
>>>>>>> (oui Unknown), length 28
>>>>>>>
>>>>>>> But the VM doesn't get any ping replies... Maybe ARP is having
>>>>>>> issues?
>>>>>>>
>>>>>>> Can anyone see anything wrong with this picture?
>>>>>>>
>>>>>>> Thanks!!
>>>>>>> -erich
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140416/94702fb3/attachment.html>
More information about the Openstack
mailing list