[Openstack] Neutron Issues...

Ageeleshwar Kandavelu Ageeleshwar.Kandavelu at csscorp.com
Wed Apr 16 01:13:25 UTC 2014


It will be inside the dhcp namespace for that network. Use ip netns.

Sent using CloudMagic<https://cloudmagic.com/k/d/mailapp?ct=pa&cv=1.0.10.8&pv=4.2.2> <https://cloudmagic.com/k/d/mailapp?ct=pa&cv=1.0.10.8&pv=4.2.2>


On Wed, Apr 16, 2014 at 5:34 AM, Erich Weiler <weiler at soe.ucsc.edu<mailto:weiler at soe.ucsc.edu>> wrote:


I think I've identified a problem - when I create a new network and
subnet via [neutron net-create and neutron subnet-create], it creates
the networks fine, and then I start the dhcp service on the network node:

/etc/init.d/neutron-dhcp-agent start

It seems to add a tap interface to br-int:

# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c2864f59274e
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC
SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
  1(tapbd844c9f-90): addr:4a:01:00:00:00:00  <------  ADDS THIS
      config:     PORT_DOWN
      state:      LINK_DOWN
      speed: 0 Mbps now, 0 Mbps max
  4(int-br-eth2): addr:52:76:9c:cb:4a:47
      config:     0
      state:      0
      current:    10GB-FD COPPER
      speed: 10000 Mbps now, 0 Mbps max
  5(int-br-ex): addr:2e:d8:0d:0b:b9:e8
      config:     0
      state:      0
      current:    10GB-FD COPPER
      speed: 10000 Mbps now, 0 Mbps max
  LOCAL(br-int): addr:c2:86:4f:59:27:4e
      config:     0
      state:      0
      speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

But as you note, the link is down.  I would expect a corresponding
interface called "tapbd844c9f-90" be created when I do a simple
"ifconfig" on the network node, but it's not there:

# ifconfig
br-eth2   Link encap:Ethernet  HWaddr 00:14:4F:CB:26:F0
           inet6 addr: fe80::68a6:5ff:fe23:18e8/64 Scope:Link
           UP BROADCAST RUNNING  MTU:1500  Metric:1
           RX packets:354 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:27568 (26.9 KiB)  TX bytes:468 (468.0 b)

br-ex     Link encap:Ethernet  HWaddr 00:14:4F:CB:26:EF
           inet6 addr: fe80::214:4fff:fecb:26ef/64 Scope:Link
           UP BROADCAST RUNNING  MTU:1500  Metric:1
           RX packets:589 errors:0 dropped:0 overruns:0 frame:0
           TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:40950 (39.9 KiB)  TX bytes:936 (936.0 b)

br-int    Link encap:Ethernet  HWaddr C2:86:4F:59:27:4E
           inet6 addr: fe80::58ad:bff:feb6:2a38/64 Scope:Link
           UP BROADCAST RUNNING  MTU:1500  Metric:1
           RX packets:369 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:28334 (27.6 KiB)  TX bytes:468 (468.0 b)

eth0      Link encap:Ethernet  HWaddr 00:14:4F:CB:26:EE
           inet addr:10.1.1.145  Bcast:10.1.255.255  Mask:255.255.0.0
           inet6 addr: fe80::214:4fff:fecb:26ee/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:12610 errors:0 dropped:0 overruns:0 frame:0
           TX packets:9659 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:1431159 (1.3 MiB)  TX bytes:2084840 (1.9 MiB)
           Memory:fab60000-fab80000

eth1      Link encap:Ethernet  HWaddr 00:14:4F:CB:26:EF
           inet6 addr: fe80::214:4fff:fecb:26ef/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
           RX packets:3422 errors:0 dropped:0 overruns:0 frame:0
           TX packets:59 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:205320 (200.5 KiB)  TX bytes:9464 (9.2 KiB)
           Memory:fabe0000-fac00000

eth2      Link encap:Ethernet  HWaddr 00:14:4F:CB:26:F0
           inet6 addr: fe80::214:4fff:fecb:26f0/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:25018 errors:0 dropped:0 overruns:0 frame:0
           TX packets:344 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:1801516 (1.7 MiB)  TX bytes:19639 (19.1 KiB)
           Memory:fac60000-fac80000

int-br-eth2 Link encap:Ethernet  HWaddr 52:76:9C:CB:4A:47
           inet6 addr: fe80::5076:9cff:fecb:4a47/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:6 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:468 (468.0 b)  TX bytes:468 (468.0 b)

int-br-ex Link encap:Ethernet  HWaddr 2E:D8:0D:0B:B9:E8
           inet6 addr: fe80::2cd8:dff:fe0b:b9e8/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:21 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:1368 (1.3 KiB)  TX bytes:468 (468.0 b)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

phy-br-eth2 Link encap:Ethernet  HWaddr CA:37:ED:2E:1A:57
           inet6 addr: fe80::c837:edff:fe2e:1a57/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:6 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:468 (468.0 b)  TX bytes:468 (468.0 b)

phy-br-ex Link encap:Ethernet  HWaddr 22:F0:60:46:26:FD
           inet6 addr: fe80::20f0:60ff:fe46:26fd/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:6 errors:0 dropped:0 overruns:0 frame:0
           TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:468 (468.0 b)  TX bytes:1368 (1.3 KiB)

I think becasue it's not there, that's why "ovs-ofctl show br-int" shows
the link is down.  Does anyone know why it doesn't create that
interface?  Or if I can create it manually...?



On 04/14/14 14:09, Erich Weiler wrote:
> I think I'm getting a little closer..  I see that on my neutron network
> node my DHCP server (dnsmasq) is on 10.200.0.3:
>
> # ip netns
> qrouter-eefaa5c8-95fc-4b3d-a8d2-27eebc449337
> qdhcp-e4448083-ec61-4293-ad0e-62239986965f
>
> # ip netns exec qdhcp-e4448083-ec61-4293-ad0e-62239986965f ifconfig
> lo        Link encap:Local Loopback
>            inet addr:127.0.0.1  Mask:255.0.0.0
>            inet6 addr: ::1/128 Scope:Host
>            UP LOOPBACK RUNNING  MTU:16436  Metric:1
>            RX packets:7 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:0
>            RX bytes:1158 (1.1 KiB)  TX bytes:1158 (1.1 KiB)
>
> tapc5195287-8f Link encap:Ethernet  HWaddr FA:16:3E:C2:EC:B5
>            inet addr:10.200.0.3  Bcast:10.200.255.255  Mask:255.255.0.0
>            inet6 addr: fe80::f816:3eff:fec2:ecb5/64 Scope:Link
>            UP BROADCAST RUNNING  MTU:1500  Metric:1
>            RX packets:10769 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:10799 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:0
>            RX bytes:647044 (631.8 KiB)  TX bytes:454965 (444.3 KiB)
>
> But when I look at my OVS state on the same neutron node:
>
> # ovs-ofctl show br-int
>
> OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c2864f59274e
> n_tables:254, n_buffers:256
> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
> actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC
> SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
>   1(int-br-eth2): addr:0a:0d:73:7f:91:8f
>       config:     0
>       state:      0
>       current:    10GB-FD COPPER
>       speed: 10000 Mbps now, 0 Mbps max
>   2(int-br-ex): addr:d6:10:30:bf:67:15
>       config:     0
>       state:      0
>       current:    10GB-FD COPPER
>       speed: 10000 Mbps now, 0 Mbps max
>   3(qr-b87526f1-3d): addr:c6:25:03:f0:51:08
>       config:     0
>       state:      0
>       speed: 0 Mbps now, 0 Mbps max
>   4(tap5a3a1ef2-9c): addr:3e:63:c2:19:54:6a
>       config:     0
>       state:      0
>       speed: 0 Mbps now, 0 Mbps max
>   5(tapc5195287-8f): addr:af:00:00:00:00:00  <--- HUH??
>       config:     PORT_DOWN
>       state:      LINK_DOWN
>       speed: 0 Mbps now, 0 Mbps max
>   LOCAL(br-int): addr:c2:86:4f:59:27:4e
>       config:     0
>       state:      0
>       speed: 0 Mbps now, 0 Mbps max
> OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
>
> It shows the tapc5195287-8f port, the one dnsmasq is on, is PORT_DOWN,
> LINK_DOWN.  I wonder why that is?  The earlier "ip netns ifconfig" on
> the dnsmasq port showed status "UP".  Anyone see anything I'm missing?
>
> Thanks,
> erich
>
> On 04/14/14 11:30, Erich Weiler wrote:
>> And additionally here's my "iptables -L" on the compute node with the VM
>> running on it, if anyone can see anything wrong with it that might block
>> ARP responses...  I didn't modify this manually, looks like Open vSwitch
>> did all the tweaking.  Remember 10.200.0.3 is the dnsmasq server and
>> 10.200.0.6 is the VM:
>>
>> # iptables -L
>> Chain INPUT (policy ACCEPT)
>> target     prot opt source               destination
>> neutron-openvswi-INPUT  all  --  anywhere             anywhere
>> ACCEPT     udp  --  anywhere             anywhere            udp
>> dpt:domain
>> ACCEPT     tcp  --  anywhere             anywhere            tcp
>> dpt:domain
>> ACCEPT     udp  --  anywhere             anywhere            udp
>> dpt:bootps
>> ACCEPT     tcp  --  anywhere             anywhere            tcp
>> dpt:bootps
>>
>> Chain FORWARD (policy ACCEPT)
>> target     prot opt source               destination
>> neutron-filter-top  all  --  anywhere             anywhere
>> neutron-openvswi-FORWARD  all  --  anywhere             anywhere
>> ACCEPT     all  --  anywhere             192.168.122.0/24    state
>> RELATED,ESTABLISHED
>> ACCEPT     all  --  192.168.122.0/24     anywhere
>> ACCEPT     all  --  anywhere             anywhere
>> REJECT     all  --  anywhere             anywhere            reject-with
>> icmp-port-unreachable
>> REJECT     all  --  anywhere             anywhere            reject-with
>> icmp-port-unreachable
>>
>> Chain OUTPUT (policy ACCEPT)
>> target     prot opt source               destination
>> neutron-filter-top  all  --  anywhere             anywhere
>> neutron-openvswi-OUTPUT  all  --  anywhere             anywhere
>>
>> Chain neutron-filter-top (2 references)
>> target     prot opt source               destination
>> neutron-openvswi-local  all  --  anywhere             anywhere
>>
>> Chain neutron-openvswi-FORWARD (1 references)
>> target     prot opt source               destination
>> neutron-openvswi-sg-chain  all  --  anywhere             anywhere
>> PHYSDEV match --physdev-out tapb2c5d867-34 --physdev-is-bridged
>> neutron-openvswi-sg-chain  all  --  anywhere             anywhere
>> PHYSDEV match --physdev-in tapb2c5d867-34 --physdev-is-bridged
>>
>> Chain neutron-openvswi-INPUT (1 references)
>> target     prot opt source               destination
>> neutron-openvswi-ob2c5d867-3  all  --  anywhere             anywhere
>>          PHYSDEV match --physdev-in tapb2c5d867-34 --physdev-is-bridged
>>
>> Chain neutron-openvswi-OUTPUT (1 references)
>> target     prot opt source               destination
>>
>> Chain neutron-openvswi-ib2c5d867-3 (1 references)
>> target     prot opt source               destination
>> DROP       all  --  anywhere             anywhere            state
>> INVALID
>> RETURN     all  --  anywhere             anywhere            state
>> RELATED,ESTABLISHED
>> RETURN     udp  --  10.200.0.3           anywhere            udp
>> spt:bootps dpt:bootpc
>> neutron-openvswi-sg-fallback  all  --  anywhere             anywhere
>>
>> Chain neutron-openvswi-local (1 references)
>> target     prot opt source               destination
>>
>> Chain neutron-openvswi-ob2c5d867-3 (2 references)
>> target     prot opt source               destination
>> RETURN     udp  --  anywhere             anywhere            udp
>> spt:bootpc dpt:bootps
>> neutron-openvswi-sb2c5d867-3  all  --  anywhere             anywhere
>> DROP       udp  --  anywhere             anywhere            udp
>> spt:bootps dpt:bootpc
>> DROP       all  --  anywhere             anywhere            state
>> INVALID
>> RETURN     all  --  anywhere             anywhere            state
>> RELATED,ESTABLISHED
>> RETURN     all  --  anywhere             anywhere
>> neutron-openvswi-sg-fallback  all  --  anywhere             anywhere
>>
>> Chain neutron-openvswi-sb2c5d867-3 (1 references)
>> target     prot opt source               destination
>> RETURN     all  --  10.200.0.6           anywhere            MAC
>> FA:16:3E:5D:11:5B
>> DROP       all  --  anywhere             anywhere
>>
>> Chain neutron-openvswi-sg-chain (2 references)
>> target     prot opt source               destination
>> neutron-openvswi-ib2c5d867-3  all  --  anywhere             anywhere
>>          PHYSDEV match --physdev-out tapb2c5d867-34 --physdev-is-bridged
>> neutron-openvswi-ob2c5d867-3  all  --  anywhere             anywhere
>>          PHYSDEV match --physdev-in tapb2c5d867-34 --physdev-is-bridged
>> ACCEPT     all  --  anywhere             anywhere
>>
>> Chain neutron-openvswi-sg-fallback (2 references)
>> target     prot opt source               destination
>> DROP       all  --  anywhere             anywhere
>>
>>
>> On 04/14/14 11:06, Erich Weiler wrote:
>>> No one can see anything wrong?
>>>
>>> Here's "ovs-vsctl show" on the neutron network node, if it helps:
>>>
>>> # ovs-vsctl show
>>> 52702cef-6433-4627-ade8-51561b4e8126
>>>      Bridge "br-eth2"
>>>          Port "phy-br-eth2"
>>>              Interface "phy-br-eth2"
>>>          Port "eth2"
>>>              Interface "eth2"
>>>          Port "br-eth2"
>>>              Interface "br-eth2"
>>>                  type: internal
>>>      Bridge br-int
>>>          Port int-br-ex
>>>              Interface int-br-ex
>>>          Port br-int
>>>              Interface br-int
>>>                  type: internal
>>>          Port "qr-b87526f1-3d"
>>>              tag: 4095
>>>              Interface "qr-b87526f1-3d"
>>>                  type: internal
>>>          Port "int-br-eth2"
>>>              Interface "int-br-eth2"
>>>          Port "tapc5195287-8f"
>>>              tag: 1
>>>              Interface "tapc5195287-8f"
>>>                  type: internal
>>>          Port "tap5a3a1ef2-9c"
>>>              tag: 4095
>>>              Interface "tap5a3a1ef2-9c"
>>>                  type: internal
>>>      Bridge br-ex
>>>          Port phy-br-ex
>>>              Interface phy-br-ex
>>>          Port "qg-0b1ba3a7-16"
>>>              Interface "qg-0b1ba3a7-16"
>>>                  type: internal
>>>          Port "eth1"
>>>              Interface "eth1"
>>>          Port br-ex
>>>              Interface br-ex
>>>                  type: internal
>>>      ovs_version: "1.11.0"
>>>
>>> On 04/12/14 15:26, Erich Weiler wrote:
>>>> And one more, I think I found something.  On the network node, I see on
>>>> phy-br-eth2:
>>>>
>>>> # tcpdump -n -e -vv -i phy-br-eth2
>>>>
>>>> tcpdump: WARNING: phy-br-eth2: no IPv4 address assigned
>>>> tcpdump: listening on phy-br-eth2, link-type EN10MB (Ethernet), capture
>>>> size 65535 bytes
>>>> 15:23:47.394584 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>> 15:23:47.394605 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
>>>> (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4
>>>> (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>> 15:23:48.394677 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>> 15:23:48.394693 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
>>>> (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4
>>>> (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>> 15:23:49.395364 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>> 15:23:49.395379 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype 802.1Q
>>>> (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4
>>>> (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>
>>>> It sees the ARP request on VLAN 200, but it looks like it replies on
>>>> VLAN 1.  I'm not sure why it would reply on VLAN 1...  I'm not entirely
>>>> sure how phy-br-eth2 works though...
>>>>
>>>> On 4/12/14, 2:07 PM, Erich Weiler wrote:
>>>>> Sorry, one more...  ;)
>>>>>
>>>>> Wile trying to ping the dnsmasq host (10.200.0.3) from the VM
>>>>> (statically configured to 10.200.0.6), if I tcpdump specifically on
>>>>> vlan
>>>>> 200 on the network node:
>>>>>
>>>>> # tcpdump -ni eth2 vlan 200
>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>> decode
>>>>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535
>>>>> bytes
>>>>> 13:55:18.826384 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:18.956941 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:55:19.825947 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:20.826056 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:20.960779 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:55:21.826718 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:22.825870 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:22.976173 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:55:23.825964 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:24.826730 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:55:24.981795 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>
>>>>> I see the ARP requests coming in.  But I don't see any replies, at
>>>>> least
>>>>> on vlan 200.  If I tcpdump with a specific VLAN, I do see the reply on
>>>>> the network node:
>>>>>
>>>>> # tcpdump -i eth2
>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>> decode
>>>>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535
>>>>> bytes
>>>>> 13:58:11.830842 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:58:11.830878 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>> Unknown), length 28
>>>>> 13:58:12.833434 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>> length 46
>>>>> 13:58:12.833457 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>> Unknown), length 28
>>>>> 13:58:13.494885 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.495217 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.495577 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.496017 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>> 13:58:13.496355 STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>
>>>>> So maybe the ARP reply is being sent out, but maybe just not over VLAN
>>>>> 200 where it needs to go, very strange...
>>>>>
>>>>> Also more verbose tcpdump on the network node:
>>>>>
>>>>> # tcpdump -n -e -vv -i eth2
>>>>>
>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>>>> 65535 bytes
>>>>> 14:04:10.848783 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>>> 14:04:10.848815 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>> 14:04:10.918013 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.918374 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.918705 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 204, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80cc.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.919171 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 202, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80ca.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.919525 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 203, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80cb.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.919889 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 206, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80ce.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.920346 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 208, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80d0.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.920682 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 209, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80d1.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.921010 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 200, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80c8.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:10.921295 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 207, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80cf.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:11.287103 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cc, 802.3, length
>>>>> 188: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl
>>>>> 0x03: oui Cisco (0x00000c), pid VTP (0x2003): VTPv1, Message Join
>>>>> message (0x04), length 166
>>>>>      Domain name: cbse, Rsvd: 0
>>>>> 14:04:11.848874 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>>> 14:04:11.848896 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>> 14:04:12.851406 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q
>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46
>>>>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>> 14:04:12.923686 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>> 14:04:12.924039 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa) Individual,
>>>>> ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST
>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id
>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,
>>>>> forwarding-delay 15.00s
>>>>>      root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008
>>>>>
>>>>> That one seems to indicate the ARP reply is being sent on vlan
>>>>> 200..???  :
>>>>>
>>>>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype
>>>>> 802.1Q
>>>>> (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6),
>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28
>>>>>
>>>>> I'm totally confused...  ARP should be working according to that....
>>>>>
>>>>> On 4/12/14, 1:44 PM, Erich Weiler wrote:
>>>>>> I tried additionally checking the dnsmasq instance on the network
>>>>>> node
>>>>>> while trying to ping it from the VM (I set the VMs IP statically to
>>>>>> 10.200.0.6 and tried to ping dnsmasq which is 10.200.0.3).  From the
>>>>>> network node, I did:
>>>>>>
>>>>>> # ip netns
>>>>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f
>>>>>>
>>>>>> [root at cloudnet-1 network-scripts]# ip netns exec
>>>>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f ifconfig
>>>>>> lo        Link encap:Local Loopback
>>>>>>            inet addr:127.0.0.1  Mask:255.0.0.0
>>>>>>            inet6 addr: ::1/128 Scope:Host
>>>>>>            UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>>>>>            RX packets:17 errors:0 dropped:0 overruns:0 frame:0
>>>>>>            TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
>>>>>>            collisions:0 txqueuelen:0
>>>>>>            RX bytes:3184 (3.1 KiB)  TX bytes:3184 (3.1 KiB)
>>>>>>
>>>>>> tapc5195287-8f Link encap:Ethernet  HWaddr FA:16:3E:C2:EC:B5
>>>>>>            inet addr:10.200.0.3  Bcast:10.200.255.255
>>>>>> Mask:255.255.0.0
>>>>>>            inet6 addr: fe80::f816:3eff:fec2:ecb5/64 Scope:Link
>>>>>>            UP BROADCAST RUNNING  MTU:1500  Metric:1
>>>>>>            RX packets:2369 errors:0 dropped:0 overruns:0 frame:0
>>>>>>            TX packets:2220 errors:0 dropped:0 overruns:0 carrier:0
>>>>>>            collisions:0 txqueuelen:0
>>>>>>            RX bytes:158390 (154.6 KiB)  TX bytes:94804 (92.5 KiB)
>>>>>>
>>>>>> # ip netns exec qdhcp-e4448083-ec61-4293-ad0e-62239986965f tcpdump -i
>>>>>> tapc5195287-8f
>>>>>>
>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>> decode
>>>>>> listening on tapc5195287-8f, link-type EN10MB (Ethernet), capture
>>>>>> size
>>>>>> 65535 bytes
>>>>>> 13:38:03.997934 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:03.997948 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:03.999280 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:04.998033 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:04.998044 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:04.999187 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:05.999197 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:06.001111 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:06.001121 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:06.998171 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:06.998181 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:07.998279 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:07.998289 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:09.001444 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:09.001454 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:09.004258 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:09.998436 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:09.998446 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:10.004161 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:10.998488 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:10.998498 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:11.004181 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:12.001742 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:12.001752 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:12.998647 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:12.998657 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:13.998762 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:13.998772 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:14.009308 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:15.000801 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:15.000812 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:24.019260 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,
>>>>>> length 28
>>>>>> 13:38:24.998348 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:34.999130 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:34.999143 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:36.003134 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:36.003147 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:36.999278 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:36.999291 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:37.999362 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:37.999375 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:39.003456 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:39.003469 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:40.003526 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:40.003539 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:41.003609 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:41.003622 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:42.003783 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:42.003796 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>> 13:38:43.003781 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>> length 46
>>>>>> 13:38:43.003794 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>> Unknown), length 28
>>>>>>
>>>>>> I assume because of the fact I'm seeing the ARP requests and
>>>>>> replies on
>>>>>> the dnsmasq interface that VLANs are working, as they are supposed
>>>>>> to be
>>>>>> talking on VLAN 200, but ARP isn't working for some reason, or at
>>>>>> least
>>>>>> the VM isn't getting the ARP reply.
>>>>>>
>>>>>> I tried setting my data interfaces on my network node and my compute
>>>>>> node to promiscuous mode (they weren't before), but it didn't seem to
>>>>>> help.  Are those interfaces supposed to be in promiscuous mode?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/12/14, 1:02 PM, Erich Weiler wrote:
>>>>>>> Hi Y'all,
>>>>>>>
>>>>>>> I'm making some progress on my neutron VLAN deployment issues, but
>>>>>>> it's
>>>>>>> still not working as expected.  I have my compute nodes data port
>>>>>>> connected to a switchport that is a trunk, allowing VLANs 200-209 to
>>>>>>> flow over the trunk.  The neutron node also has its internal data
>>>>>>> port
>>>>>>> on the same switch, on a trunk port, also allowing VLANs 200-209 to
>>>>>>> flow
>>>>>>> over it.
>>>>>>>
>>>>>>> I *think* I have OVS configured correctly on the neutron network
>>>>>>> node
>>>>>>> and the compute nodes.
>>>>>>>
>>>>>>> To test I just created an internal network (on physnet1) callled
>>>>>>> test-net on segmentation-id 200 (network 10.200.0.0/16).  So I
>>>>>>> assume it
>>>>>>> will use VLAN 200.  I then created a VM and attached it to that
>>>>>>> network.
>>>>>>>   But when the VM boots, it does not get an IP address from dnsmasq.
>>>>>>>
>>>>>>> Back on the network node, I see dnsmasq running on that VLAN, I
>>>>>>> think:
>>>>>>>
>>>>>>> # ps -afe | grep masq
>>>>>>> nobody   16674     1  0 Apr10 ?        00:00:00 dnsmasq --no-hosts
>>>>>>> --no-resolv --strict-order --bind-interfaces
>>>>>>> --interface=tapc5195287-8f
>>>>>>> --except-interface=lo
>>>>>>> --pid-file=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/pid
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --dhcp-hostsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/host
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --addn-hosts=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/addn_hosts
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --dhcp-optsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239986965f/opts
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --leasefile-ro --dhcp-range=tag0,10.200.0.0,static,86400s
>>>>>>> --dhcp-lease-max=65536 --conf-file= --domain=openstacklocal
>>>>>>>
>>>>>>> And I tried doing a tcpdump on eth2 (which is the data interface
>>>>>>> with
>>>>>>> the VLANs) while doing a DHCPDISCOVER from the VM:
>>>>>>>
>>>>>>> # tcpdump -i eth2
>>>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>>> decode
>>>>>>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535
>>>>>>> bytes
>>>>>>> 12:43:46.440221 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.440563 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.440896 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.441414 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.441672 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.442001 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.442502 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.442830 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.443171 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:46.443470 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:47.718633 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>>>>>> BOOTP/DHCP,
>>>>>>> Request from fa:16:3e:5d:11:5b (oui Unknown), length 280
>>>>>>> 12:43:47.719183 IP 10.200.0.3.bootps > 10.200.0.6.bootpc:
>>>>>>> BOOTP/DHCP,
>>>>>>> Reply, length 325
>>>>>>> 12:43:48.447761 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.448109 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.448593 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.448896 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.449266 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.449672 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.450010 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.450385 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.450678 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:48.451115 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:49.946348 VTPv1, Message Join message (0x04), length 166
>>>>>>> 12:43:50.459700 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.460184 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.460529 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.460867 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.461195 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.461625 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.461956 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.462335 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.462745 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:50.463072 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.455118 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.455591 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.455930 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.456258 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.456575 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.457053 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.457414 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.457702 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.458171 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.458503 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:52.718355 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,
>>>>>>> length 28
>>>>>>> 12:43:53.718209 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,
>>>>>>> length 28
>>>>>>> 12:43:54.460398 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.460871 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.461187 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.461588 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.461984 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.462361 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.462652 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.462981 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.463474 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:43:54.463751 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>>
>>>>>>> Where 10.200.0.3 is the IP address of the dnsmasq instance on the
>>>>>>> network node and 10.200.0.6 is the IP (supposedly) of the VM, from
>>>>>>> what
>>>>>>> nova tells me.  It seems the traffic is getting to the network node,
>>>>>>> but
>>>>>>> the VM never gets the IP.  Even if I configure the VM statically
>>>>>>> to be
>>>>>>> 10.200.0.6, I still can't ping the DHCP server on 10.200.0.3.
>>>>>>>
>>>>>>> It looks like ARP requests are getting sent out for the DHCP server
>>>>>>> hen
>>>>>>> I ping (as seen fro the network node):
>>>>>>>
>>>>>>> # tcpdump -i eth2
>>>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned
>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>>> decode
>>>>>>> listening on eth2, link-type EN10MB (Ethernet), capture size 65535
>>>>>>> bytes
>>>>>>> 12:57:08.871232 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.871717 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.872063 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.872421 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.872819 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.873212 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.873507 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.873839 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.874329 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:08.874621 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:09.041413 VTPv1, Message Join message (0x04), length 166
>>>>>>> 12:57:09.289413 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:09.289435 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>>> Unknown), length 28
>>>>>>> 12:57:10.289706 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:10.289728 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>>> Unknown), length 28
>>>>>>> 12:57:10.876689 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.877031 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.877439 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.877843 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.878190 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.878542 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.878824 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.879282 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.879612 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:10.879933 STP 802.1d, Config, Flags [none], bridge-id
>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42
>>>>>>> 12:57:11.289582 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:11.289603 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>>> Unknown), length 28
>>>>>>> 12:57:12.289668 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,
>>>>>>> length 46
>>>>>>> 12:57:12.289689 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui
>>>>>>> Unknown), length 28
>>>>>>>
>>>>>>> But the VM doesn't get any ping replies...  Maybe ARP is having
>>>>>>> issues?
>>>>>>>
>>>>>>> Can anyone see anything wrong with this picture?
>>>>>>>
>>>>>>> Thanks!!
>>>>>>> -erich

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

http://www.csscorp.com/common/email-disclaimer.php
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140416/2499207c/attachment.html>


More information about the Openstack mailing list