[Openstack] Neutron Issues...
Erich Weiler
weiler at soe.ucsc.edu
Mon Apr 14 18:06:35 UTC 2014
No one can see anything wrong?
Here's "ovs-vsctl show" on the neutron network node, if it helps:
# ovs-vsctl show
52702cef-6433-4627-ade8-51561b4e8126
Bridge "br-eth2"
Port "phy-br-eth2"
Interface "phy-br-eth2"
Port "eth2"
Interface "eth2"
Port "br-eth2"
Interface "br-eth2"
type: internal
Bridge br-int
Port int-br-ex
Interface int-br-ex
Port br-int
Interface br-int
type: internal
Port "qr-b87526f1-3d"
tag: 4095
Interface "qr-b87526f1-3d"
type: internal
Port "int-br-eth2"
Interface "int-br-eth2"
Port "tapc5195287-8f"
tag: 1
Interface "tapc5195287-8f"
type: internal
Port "tap5a3a1ef2-9c"
tag: 4095
Interface "tap5a3a1ef2-9c"
type: internal
Bridge br-ex
Port phy-br-ex
Interface phy-br-ex
Port "qg-0b1ba3a7-16"
Interface "qg-0b1ba3a7-16"
type: internal
Port "eth1"
Interface "eth1"
Port br-ex
Interface br-ex
type: internal
ovs_version: "1.11.0"
On 04/12/14 15:26, Erich Weiler wrote:
> And one more, I think I found something. On the network node, I see on
> phy-br-eth2:
>
> # tcpdump -n -e -vv -i phy-br-eth2
>
> tcpdump: WARNING: phy-br-eth2: no IPv4 address assigned
> tcpdump: listening on phy-br-eth2, link-type EN10MB (Ethernet), capture
> size 65535 bytes
> 15:23:47.394584 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
> 15:23:47.394605 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
> (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4
> (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
> 15:23:48.394677 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
> 15:23:48.394693 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
> (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4
> (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
> 15:23:49.395364 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
> 15:23:49.395379 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
> (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4
> (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>
> It sees the ARP request on VLAN 200, but it looks like it replies on
> VLAN 1. I'm not sure why it would reply on VLAN 1... I'm not entirely
> sure how phy-br-eth2 works though...
>
> On 4/12/14, 2:07 PM, Erich Weiler wrote:
>> Sorry, one more... ;)
>>
>> Wile trying to ping the dnsmasq host (10.200.0.3) from the VM
>> (statically configured to 10.200.0.6), if I tcpdump specifically on vlan
>> 200 on the network node:
>>
>> # tcpdump -ni eth2 vlan 200
>> tcpdump: WARNING: eth2: no IPv4 address assigned
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>> decode
>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 13:55:18.826384 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:55:18.956941 STP 802.1d, Config, Flags [none], bridge-id
>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>> 13:55:19.825947 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:55:20.826056 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:55:20.960779 STP 802.1d, Config, Flags [none], bridge-id
>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>> 13:55:21.826718 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:55:22.825870 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:55:22.976173 STP 802.1d, Config, Flags [none], bridge-id
>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>> 13:55:23.825964 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:55:24.826730 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:55:24.981795 STP 802.1d, Config, Flags [none], bridge-id
>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>
>> I see the ARP requests coming in. But I don't see any replies, at least
>> on vlan 200. If I tcpdump with a specific VLAN, I do see the reply on
>> the network node:
>>
>> # tcpdump -i eth2
>> tcpdump: WARNING: eth2: no IPv4 address assigned
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>> decode
>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 13:58:11.830842 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:58:11.830878 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>> Unknown), length 28
>> 13:58:12.833434 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>> length 46
>> 13:58:12.833457 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>> Unknown), length 28
>> 13:58:13.494885 STP 802.1d, Config, Flags [none], bridge-id
>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>> 13:58:13.495217 STP 802.1d, Config, Flags [none], bridge-id
>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>> 13:58:13.495577 STP 802.1d, Config, Flags [none], bridge-id
>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>> 13:58:13.496017 STP 802.1d, Config, Flags [none], bridge-id
>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>> 13:58:13.496355 STP 802.1d, Config, Flags [none], bridge-id
>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>
>> So maybe the ARP reply is being sent out, but maybe just not over VLAN
>> 200 where it needs to go, very strange...
>>
>> Also more verbose tcpdump on the network node:
>>
>> # tcpdump -n -e -vv -i eth2
>>
>> tcpdump: WARNING: eth2: no IPv4 address assigned
>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>> 65535 bytes
>> 14:04:10.848783 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>> 14:04:10.848815 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>> 14:04:10.918013 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.918374 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.918705 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 204, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80cc.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.919171 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 202, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80ca.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.919525 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 203, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80cb.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.919889 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 206, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80ce.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.920346 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 208, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80d0.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.920682 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 209, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80d1.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.921010 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 200, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80c8.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:10.921295 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 207, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80cf.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:11.287103 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cc, 802.3, length
>> 188: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl
>> 0x03: oui Cisco (0x00000c), pid VTP (0x2003): VTPv1, Message Join
>> message (0x04), length 166
>> Domain name: cbse, Rsvd: 0
>> 14:04:11.848874 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>> 14:04:11.848896 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>> 14:04:12.851406 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>> 14:04:12.923686 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008
>> 14:04:12.924039 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype 802.1Q
>> (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa) Individual,
>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>> message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>> forwarding-delay 15.00s
>> root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008
>>
>> That one seems to indicate the ARP reply is being sent on vlan
>> 200..??? :
>>
>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>
>> I'm totally confused... ARP should be working according to that....
>>
>> On 4/12/14, 1:44 PM, Erich Weiler wrote:
>>> I tried additionally checking the dnsmasq instance on the network node
>>> while trying to ping it from the VM (I set the VMs IP statically to
>>> 10.200.0.6 and tried to ping dnsmasq which is 10.200.0.3). From the
>>> network node, I did:
>>>
>>> # ip netns
>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f
>>>
>>> [root at cloudnet-1 network-scripts]# ip netns exec
>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f ifconfig
>>> lo Link encap:Local Loopback
>>> inet addr:127.0.0.1 Mask:255.0.0.0
>>> inet6 addr: ::1/128 Scope:Host
>>> UP LOOPBACK RUNNING MTU:16436 Metric:1
>>> RX packets:17 errors:0 dropped:0 overruns:0 frame:0
>>> TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
>>> collisions:0 txqueuelen:0
>>> RX bytes:3184 (3.1 KiB) TX bytes:3184 (3.1 KiB)
>>>
>>> tapc5195287-8f Link encap:Ethernet HWaddr FA:16:3E:C2:EC:B5
>>> inet addr:10.200.0.3 Bcast:10.200.255.255 Mask:255.255.0.0
>>> inet6 addr: fe80::f816:3eff:fec2:ecb5/64 Scope:Link
>>> UP BROADCAST RUNNING MTU:1500 Metric:1
>>> RX packets:2369 errors:0 dropped:0 overruns:0 frame:0
>>> TX packets:2220 errors:0 dropped:0 overruns:0 carrier:0
>>> collisions:0 txqueuelen:0
>>> RX bytes:158390 (154.6 KiB) TX bytes:94804 (92.5 KiB)
>>>
>>> # ip netns exec qdhcp-e4448083-ec61-4293-ad0e-62239986965f tcpdump -i
>>> tapc5195287-8f
>>>
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on tapc5195287-8f, link-type EN10MB (Ethernet), capture size
>>> 65535 bytes
>>> 13:38:03.997934 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:03.997948 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:03.999280 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:04.998033 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:04.998044 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:04.999187 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:05.999197 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:06.001111 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:06.001121 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:06.998171 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:06.998181 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:07.998279 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:07.998289 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:09.001444 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:09.001454 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:09.004258 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:09.998436 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:09.998446 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:10.004161 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:10.998488 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:10.998498 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:11.004181 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:12.001742 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:12.001752 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:12.998647 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:12.998657 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:13.998762 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:13.998772 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:14.009308 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:15.000801 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:15.000812 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:24.019260 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>> length 28
>>> 13:38:24.998348 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:34.999130 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:34.999143 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:36.003134 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:36.003147 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:36.999278 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:36.999291 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:37.999362 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:37.999375 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:39.003456 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:39.003469 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:40.003526 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:40.003539 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:41.003609 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:41.003622 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:42.003783 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:42.003796 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>> 13:38:43.003781 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>> length 46
>>> 13:38:43.003794 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>> Unknown), length 28
>>>
>>> I assume because of the fact I'm seeing the ARP requests and replies on
>>> the dnsmasq interface that VLANs are working, as they are supposed to be
>>> talking on VLAN 200, but ARP isn't working for some reason, or at least
>>> the VM isn't getting the ARP reply.
>>>
>>> I tried setting my data interfaces on my network node and my compute
>>> node to promiscuous mode (they weren't before), but it didn't seem to
>>> help. Are those interfaces supposed to be in promiscuous mode?
>>>
>>>
>>>
>>> On 4/12/14, 1:02 PM, Erich Weiler wrote:
>>>> Hi Y'all,
>>>>
>>>> I'm making some progress on my neutron VLAN deployment issues, but it's
>>>> still not working as expected. I have my compute nodes data port
>>>> connected to a switchport that is a trunk, allowing VLANs 200-209 to
>>>> flow over the trunk. The neutron node also has its internal data port
>>>> on the same switch, on a trunk port, also allowing VLANs 200-209 to
>>>> flow
>>>> over it.
>>>>
>>>> I *think* I have OVS configured correctly on the neutron network node
>>>> and the compute nodes.
>>>>
>>>> To test I just created an internal network (on physnet1) callled
>>>> test-net on segmentation-id 200 (network 10.200.0.0/16). So I
>>>> assume it
>>>> will use VLAN 200. I then created a VM and attached it to that
>>>> network.
>>>> But when the VM boots, it does not get an IP address from dnsmasq.
>>>>
>>>> Back on the network node, I see dnsmasq running on that VLAN, I think:
>>>>
>>>> # ps -afe | grep masq
>>>> nobody 16674 1 0 Apr10 ? 00:00:00 dnsmasq --no-hosts
>>>> --no-resolv --strict-order --bind-interfaces --interface=tapc5195287-8f
>>>> --except-interface=lo
>>>> --pid-file=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/pid
>>>>
>>>>
>>>> --dhcp-hostsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/host
>>>>
>>>>
>>>>
>>>> --addn-hosts=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/addn_hosts
>>>>
>>>>
>>>>
>>>> --dhcp-optsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/opts
>>>>
>>>>
>>>>
>>>> --leasefile-ro --dhcp-range=tag0,10.200.0.0,static,86400s
>>>> --dhcp-lease-max=65536 --conf-file= --domain=openstacklocal
>>>>
>>>> And I tried doing a tcpdump on eth2 (which is the data interface with
>>>> the VLANs) while doing a DHCPDISCOVER from the VM:
>>>>
>>>> # tcpdump -i eth2
>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>> decode
>>>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535
>>>> bytes
>>>> 12:43:46.440221 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.440563 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.440896 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.441414 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.441672 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.442001 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.442502 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.442830 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.443171 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:46.443470 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:47.718633 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>>> Request from fa:16:3e:5d:11:5b (oui Unknown), length 280
>>>> 12:43:47.719183 IP 10.200.0.3.bootps > 10.200.0.6.bootpc: BOOTP/DHCP,
>>>> Reply, length 325
>>>> 12:43:48.447761 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.448109 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.448593 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.448896 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.449266 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.449672 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.450010 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.450385 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.450678 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:48.451115 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:49.946348 VTPv1, Message Join message (0x04), length 166
>>>> 12:43:50.459700 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.460184 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.460529 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.460867 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.461195 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.461625 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.461956 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.462335 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.462745 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:50.463072 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.455118 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.455591 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.455930 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.456258 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.456575 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.457053 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.457414 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.457702 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.458171 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.458503 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:52.718355 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,
>>>> length 28
>>>> 12:43:53.718209 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,
>>>> length 28
>>>> 12:43:54.460398 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.460871 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.461187 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.461588 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.461984 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.462361 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.462652 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.462981 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.463474 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:43:54.463751 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>
>>>> Where 10.200.0.3 is the IP address of the dnsmasq instance on the
>>>> network node and 10.200.0.6 is the IP (supposedly) of the VM, from what
>>>> nova tells me. It seems the traffic is getting to the network node,
>>>> but
>>>> the VM never gets the IP. Even if I configure the VM statically to be
>>>> 10.200.0.6, I still can't ping the DHCP server on 10.200.0.3.
>>>>
>>>> It looks like ARP requests are getting sent out for the DHCP server hen
>>>> I ping (as seen fro the network node):
>>>>
>>>> # tcpdump -i eth2
>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>> decode
>>>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535
>>>> bytes
>>>> 12:57:08.871232 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.871717 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.872063 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.872421 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.872819 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.873212 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.873507 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.873839 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.874329 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:08.874621 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:09.041413 VTPv1, Message Join message (0x04), length 166
>>>> 12:57:09.289413 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>> length 46
>>>> 12:57:09.289435 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>> Unknown), length 28
>>>> 12:57:10.289706 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>> length 46
>>>> 12:57:10.289728 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>> Unknown), length 28
>>>> 12:57:10.876689 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.877031 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.877439 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.877843 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.878190 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.878542 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.878824 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.879282 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.879612 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:10.879933 STP 802.1d, Config, Flags [none], bridge-id
>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>> 12:57:11.289582 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>> length 46
>>>> 12:57:11.289603 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>> Unknown), length 28
>>>> 12:57:12.289668 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>> length 46
>>>> 12:57:12.289689 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>> Unknown), length 28
>>>>
>>>> But the VM doesn't get any ping replies... Maybe ARP is having issues?
>>>>
>>>> Can anyone see anything wrong with this picture?
>>>>
>>>> Thanks!!
>>>> -erich
More information about the Openstack
mailing list