[Openstack] Neutron Issues...

Qin, Xiaohong Xiaohong.Qin at emc.com
Wed Apr 16 17:12:18 UTC 2014


Hi Erich,



That DOWN status output from ovs-vsctl show is confusing, perhaps it meant to say that the interface is sort of "on demand" consumption that if there is a traffic it will be on. What you can do is to session into the dhcp namespace and run tcpdump on that interface there. Here are the steps,



1. sudo ip netns exec <your qdhcp namespace> bash

2. sudo tcpdump -n -e -vv -i <your dhcp server tap interface>



Then on your controller node, create a new VM, you should see some packets dump on your dhcp namespace, similar to this,



09:43:22.710102 fa:16:3e:a9:ba:20 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ffa9:ba20 to_ex { }]

09:43:22.842593 fa:16:3e:a9:ba:20 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)

    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from fa:16:3e:a9:ba:20, length 300, xid 0x3d5e463e, Flags [none] (0x0000)

          Client-Ethernet-Address fa:16:3e:a9:ba:20

          Vendor-rfc1048 Extensions

            Magic Cookie 0x63825363

            DHCP-Message Option 53, length 1: Discover

            Requested-IP Option 50, length 4: 192.168.160.154

            Hostname Option 12, length 13: "openstacktest"

            Parameter-Request Option 55, length 13:

              Subnet-Mask, BR, Time-Zone, Default-Gateway

              Domain-Name, Domain-Name-Server, Option 119, Hostname

              Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route

              NTP

09:43:22.843152 fa:16:3e:e6:a6:7f > fa:16:3e:a9:ba:20, ethertype IPv4 (0x0800), length 350: (tos 0xc0, ttl 64, id 35983, offset 0, flags [none], proto UDP (17), length 336)

    10.0.0.3.67 > 10.0.0.9.68: [bad udp cksum 0x1559 -> 0xab30!] BOOTP/DHCP, Reply, length 308, xid 0x3d5e463e, Flags [none] (0x0000)

          Your-IP 10.0.0.9

          Server-IP 10.0.0.3

          Client-Ethernet-Address fa:16:3e:a9:ba:20



Here is my dnsmasq server process,



nobody   12782     1  0 Apr15 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7e1faebd-1c --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/90a8a84a-d4dd-4c98-bf87-492342c55b94/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/90a8a84a-d4dd-4c98-bf87-492342c55b94/host --dhcp-optsfile=/opt/stack/data/neutron/dhcp/90a8a84a-d4dd-4c98-bf87-492342c55b94/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal



Hope that helps.



Dennis Qin (aka Xiaohong Qin)



-----Original Message-----
From: Erich Weiler [mailto:weiler at soe.ucsc.edu]
Sent: Tuesday, April 15, 2014 4:56 PM
To: openstack
Subject: Re: [Openstack] Neutron Issues...



I think I've identified a problem - when I create a new network and subnet via [neutron net-create and neutron subnet-create], it creates the networks fine, and then I start the dhcp service on the network node:



/etc/init.d/neutron-dhcp-agent start



It seems to add a tap interface to br-int:



# ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c2864f59274e n_tables:254, n_buffers:256

capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP

actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE

  1(tapbd844c9f-90): addr:4a:01:00:00:00:00  <------  ADDS THIS

      config:     PORT_DOWN

      state:      LINK_DOWN

      speed: 0 Mbps now, 0 Mbps max

  4(int-br-eth2): addr:52:76:9c:cb:4a:47

      config:     0

      state:      0

      current:    10GB-FD COPPER

      speed: 10000 Mbps now, 0 Mbps max

  5(int-br-ex): addr:2e:d8:0d:0b:b9:e8

      config:     0

      state:      0

      current:    10GB-FD COPPER

      speed: 10000 Mbps now, 0 Mbps max

  LOCAL(br-int): addr:c2:86:4f:59:27:4e

      config:     0

      state:      0

      speed: 0 Mbps now, 0 Mbps max

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0



But as you note, the link is down.  I would expect a corresponding interface called "tapbd844c9f-90" be created when I do a simple "ifconfig" on the network node, but it's not there:



# ifconfig

br-eth2   Link encap:Ethernet  HWaddr 00:14:4F:CB:26:F0

           inet6 addr: fe80::68a6:5ff:fe23:18e8/64 Scope:Link

           UP BROADCAST RUNNING  MTU:1500  Metric:1

           RX packets:354 errors:0 dropped:0 overruns:0 frame:0

           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:0

           RX bytes:27568 (26.9 KiB)  TX bytes:468 (468.0 b)



br-ex     Link encap:Ethernet  HWaddr 00:14:4F:CB:26:EF

           inet6 addr: fe80::214:4fff:fecb:26ef/64 Scope:Link

           UP BROADCAST RUNNING  MTU:1500  Metric:1

           RX packets:589 errors:0 dropped:0 overruns:0 frame:0

           TX packets:12 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:0

           RX bytes:40950 (39.9 KiB)  TX bytes:936 (936.0 b)



br-int    Link encap:Ethernet  HWaddr C2:86:4F:59:27:4E

           inet6 addr: fe80::58ad:bff:feb6:2a38/64 Scope:Link

           UP BROADCAST RUNNING  MTU:1500  Metric:1

           RX packets:369 errors:0 dropped:0 overruns:0 frame:0

           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:0

           RX bytes:28334 (27.6 KiB)  TX bytes:468 (468.0 b)



eth0      Link encap:Ethernet  HWaddr 00:14:4F:CB:26:EE

           inet addr:10.1.1.145  Bcast:10.1.255.255  Mask:255.255.0.0

           inet6 addr: fe80::214:4fff:fecb:26ee/64 Scope:Link

           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

           RX packets:12610 errors:0 dropped:0 overruns:0 frame:0

           TX packets:9659 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:1000

           RX bytes:1431159 (1.3 MiB)  TX bytes:2084840 (1.9 MiB)

           Memory:fab60000-fab80000



eth1      Link encap:Ethernet  HWaddr 00:14:4F:CB:26:EF

           inet6 addr: fe80::214:4fff:fecb:26ef/64 Scope:Link

           UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

           RX packets:3422 errors:0 dropped:0 overruns:0 frame:0

           TX packets:59 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:1000

           RX bytes:205320 (200.5 KiB)  TX bytes:9464 (9.2 KiB)

           Memory:fabe0000-fac00000



eth2      Link encap:Ethernet  HWaddr 00:14:4F:CB:26:F0

           inet6 addr: fe80::214:4fff:fecb:26f0/64 Scope:Link

           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

           RX packets:25018 errors:0 dropped:0 overruns:0 frame:0

           TX packets:344 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:1000

           RX bytes:1801516 (1.7 MiB)  TX bytes:19639 (19.1 KiB)

           Memory:fac60000-fac80000



int-br-eth2 Link encap:Ethernet  HWaddr 52:76:9C:CB:4A:47

           inet6 addr: fe80::5076:9cff:fecb:4a47/64 Scope:Link

           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

           RX packets:6 errors:0 dropped:0 overruns:0 frame:0

           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:1000

           RX bytes:468 (468.0 b)  TX bytes:468 (468.0 b)



int-br-ex Link encap:Ethernet  HWaddr 2E:D8:0D:0B:B9:E8

           inet6 addr: fe80::2cd8:dff:fe0b:b9e8/64 Scope:Link

           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

           RX packets:21 errors:0 dropped:0 overruns:0 frame:0

           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:1000

           RX bytes:1368 (1.3 KiB)  TX bytes:468 (468.0 b)



lo        Link encap:Local Loopback

           inet addr:127.0.0.1  Mask:255.0.0.0

           inet6 addr: ::1/128 Scope:Host

           UP LOOPBACK RUNNING  MTU:16436  Metric:1

           RX packets:0 errors:0 dropped:0 overruns:0 frame:0

           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:0

           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)



phy-br-eth2 Link encap:Ethernet  HWaddr CA:37:ED:2E:1A:57

           inet6 addr: fe80::c837:edff:fe2e:1a57/64 Scope:Link

           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

           RX packets:6 errors:0 dropped:0 overruns:0 frame:0

           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:1000

           RX bytes:468 (468.0 b)  TX bytes:468 (468.0 b)



phy-br-ex Link encap:Ethernet  HWaddr 22:F0:60:46:26:FD

           inet6 addr: fe80::20f0:60ff:fe46:26fd/64 Scope:Link

           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

           RX packets:6 errors:0 dropped:0 overruns:0 frame:0

           TX packets:21 errors:0 dropped:0 overruns:0 carrier:0

           collisions:0 txqueuelen:1000

           RX bytes:468 (468.0 b)  TX bytes:1368 (1.3 KiB)



I think becasue it's not there, that's why "ovs-ofctl show br-int" shows the link is down.  Does anyone know why it doesn't create that interface?  Or if I can create it manually...?







On 04/14/14 14:09, Erich Weiler wrote:

> I think I'm getting a little closer..  I see that on my neutron

> network node my DHCP server (dnsmasq) is on 10.200.0.3:

>

> # ip netns

> qrouter-eefaa5c8-95fc-4b3d-a8d2-27eebc449337

> qdhcp-e4448083-ec61-4293-ad0e-62239986965f

>

> # ip netns exec qdhcp-e4448083-ec61-4293-ad0e-62239986965f ifconfig

> lo        Link encap:Local Loopback

>            inet addr:127.0.0.1  Mask:255.0.0.0

>            inet6 addr: ::1/128 Scope:Host

>            UP LOOPBACK RUNNING  MTU:16436  Metric:1

>            RX packets:7 errors:0 dropped:0 overruns:0 frame:0

>            TX packets:7 errors:0 dropped:0 overruns:0 carrier:0

>            collisions:0 txqueuelen:0

>            RX bytes:1158 (1.1 KiB)  TX bytes:1158 (1.1 KiB)

>

> tapc5195287-8f Link encap:Ethernet  HWaddr FA:16:3E:C2:EC:B5

>            inet addr:10.200.0.3  Bcast:10.200.255.255  Mask:255.255.0.0

>            inet6 addr: fe80::f816:3eff:fec2:ecb5/64 Scope:Link

>            UP BROADCAST RUNNING  MTU:1500  Metric:1

>            RX packets:10769 errors:0 dropped:0 overruns:0 frame:0

>            TX packets:10799 errors:0 dropped:0 overruns:0 carrier:0

>            collisions:0 txqueuelen:0

>            RX bytes:647044 (631.8 KiB)  TX bytes:454965 (444.3 KiB)

>

> But when I look at my OVS state on the same neutron node:

>

> # ovs-ofctl show br-int

>

> OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c2864f59274e n_tables:254,

> n_buffers:256

> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS

> ARP_MATCH_IP

> actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC

> SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE

>   1(int-br-eth2): addr:0a:0d:73:7f:91:8f

>       config:     0

>       state:      0

>       current:    10GB-FD COPPER

>       speed: 10000 Mbps now, 0 Mbps max

>   2(int-br-ex): addr:d6:10:30:bf:67:15

>       config:     0

>       state:      0

>       current:    10GB-FD COPPER

>       speed: 10000 Mbps now, 0 Mbps max

>   3(qr-b87526f1-3d): addr:c6:25:03:f0:51:08

>       config:     0

>       state:      0

>       speed: 0 Mbps now, 0 Mbps max

>   4(tap5a3a1ef2-9c): addr:3e:63:c2:19:54:6a

>       config:     0

>       state:      0

>       speed: 0 Mbps now, 0 Mbps max

>   5(tapc5195287-8f): addr:af:00:00:00:00:00  <--- HUH??

>       config:     PORT_DOWN

>       state:      LINK_DOWN

>       speed: 0 Mbps now, 0 Mbps max

>   LOCAL(br-int): addr:c2:86:4f:59:27:4e

>       config:     0

>       state:      0

>       speed: 0 Mbps now, 0 Mbps max

> OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

>

> It shows the tapc5195287-8f port, the one dnsmasq is on, is PORT_DOWN,

> LINK_DOWN.  I wonder why that is?  The earlier "ip netns ifconfig" on

> the dnsmasq port showed status "UP".  Anyone see anything I'm missing?

>

> Thanks,

> erich

>

> On 04/14/14 11:30, Erich Weiler wrote:

>> And additionally here's my "iptables -L" on the compute node with the

>> VM running on it, if anyone can see anything wrong with it that might

>> block ARP responses...  I didn't modify this manually, looks like

>> Open vSwitch did all the tweaking.  Remember 10.200.0.3 is the

>> dnsmasq server and

>> 10.200.0.6 is the VM:

>>

>> # iptables -L

>> Chain INPUT (policy ACCEPT)

>> target     prot opt source               destination

>> neutron-openvswi-INPUT  all  --  anywhere             anywhere

>> ACCEPT     udp  --  anywhere             anywhere            udp

>> dpt:domain

>> ACCEPT     tcp  --  anywhere             anywhere            tcp

>> dpt:domain

>> ACCEPT     udp  --  anywhere             anywhere            udp

>> dpt:bootps

>> ACCEPT     tcp  --  anywhere             anywhere            tcp

>> dpt:bootps

>>

>> Chain FORWARD (policy ACCEPT)

>> target     prot opt source               destination

>> neutron-filter-top  all  --  anywhere             anywhere

>> neutron-openvswi-FORWARD  all  --  anywhere             anywhere

>> ACCEPT     all  --  anywhere             192.168.122.0/24    state

>> RELATED,ESTABLISHED

>> ACCEPT     all  --  192.168.122.0/24     anywhere

>> ACCEPT     all  --  anywhere             anywhere

>> REJECT     all  --  anywhere             anywhere            reject-with

>> icmp-port-unreachable

>> REJECT     all  --  anywhere             anywhere            reject-with

>> icmp-port-unreachable

>>

>> Chain OUTPUT (policy ACCEPT)

>> target     prot opt source               destination

>> neutron-filter-top  all  --  anywhere             anywhere

>> neutron-openvswi-OUTPUT  all  --  anywhere             anywhere

>>

>> Chain neutron-filter-top (2 references)

>> target     prot opt source               destination

>> neutron-openvswi-local  all  --  anywhere             anywhere

>>

>> Chain neutron-openvswi-FORWARD (1 references)

>> target     prot opt source               destination

>> neutron-openvswi-sg-chain  all  --  anywhere             anywhere

>> PHYSDEV match --physdev-out tapb2c5d867-34 --physdev-is-bridged

>> neutron-openvswi-sg-chain  all  --  anywhere             anywhere

>> PHYSDEV match --physdev-in tapb2c5d867-34 --physdev-is-bridged

>>

>> Chain neutron-openvswi-INPUT (1 references)

>> target     prot opt source               destination

>> neutron-openvswi-ob2c5d867-3  all  --  anywhere             anywhere

>>          PHYSDEV match --physdev-in tapb2c5d867-34

>> --physdev-is-bridged

>>

>> Chain neutron-openvswi-OUTPUT (1 references)

>> target     prot opt source               destination

>>

>> Chain neutron-openvswi-ib2c5d867-3 (1 references)

>> target     prot opt source               destination

>> DROP       all  --  anywhere             anywhere            state

>> INVALID

>> RETURN     all  --  anywhere             anywhere            state

>> RELATED,ESTABLISHED

>> RETURN     udp  --  10.200.0.3           anywhere            udp

>> spt:bootps dpt:bootpc

>> neutron-openvswi-sg-fallback  all  --  anywhere             anywhere

>>

>> Chain neutron-openvswi-local (1 references)

>> target     prot opt source               destination

>>

>> Chain neutron-openvswi-ob2c5d867-3 (2 references)

>> target     prot opt source               destination

>> RETURN     udp  --  anywhere             anywhere            udp

>> spt:bootpc dpt:bootps

>> neutron-openvswi-sb2c5d867-3  all  --  anywhere             anywhere

>> DROP       udp  --  anywhere             anywhere            udp

>> spt:bootps dpt:bootpc

>> DROP       all  --  anywhere             anywhere            state

>> INVALID

>> RETURN     all  --  anywhere             anywhere            state

>> RELATED,ESTABLISHED

>> RETURN     all  --  anywhere             anywhere

>> neutron-openvswi-sg-fallback  all  --  anywhere             anywhere

>>

>> Chain neutron-openvswi-sb2c5d867-3 (1 references)

>> target     prot opt source               destination

>> RETURN     all  --  10.200.0.6           anywhere            MAC

>> FA:16:3E:5D:11:5B

>> DROP       all  --  anywhere             anywhere

>>

>> Chain neutron-openvswi-sg-chain (2 references)

>> target     prot opt source               destination

>> neutron-openvswi-ib2c5d867-3  all  --  anywhere             anywhere

>>          PHYSDEV match --physdev-out tapb2c5d867-34 --physdev-is-bridged

>> neutron-openvswi-ob2c5d867-3  all  --  anywhere             anywhere

>>          PHYSDEV match --physdev-in tapb2c5d867-34 --physdev-is-bridged

>> ACCEPT     all  --  anywhere             anywhere

>>

>> Chain neutron-openvswi-sg-fallback (2 references)

>> target     prot opt source               destination

>> DROP       all  --  anywhere             anywhere

>>

>>

>> On 04/14/14 11:06, Erich Weiler wrote:

>>> No one can see anything wrong?

>>>

>>> Here's "ovs-vsctl show" on the neutron network node, if it helps:

>>>

>>> # ovs-vsctl show

>>> 52702cef-6433-4627-ade8-51561b4e8126

>>>      Bridge "br-eth2"

>>>          Port "phy-br-eth2"

>>>              Interface "phy-br-eth2"

>>>          Port "eth2"

>>>              Interface "eth2"

>>>          Port "br-eth2"

>>>              Interface "br-eth2"

>>>                  type: internal

>>>      Bridge br-int

>>>          Port int-br-ex

>>>              Interface int-br-ex

>>>          Port br-int

>>>              Interface br-int

>>>                  type: internal

>>>          Port "qr-b87526f1-3d"

>>>              tag: 4095

>>>              Interface "qr-b87526f1-3d"

>>>                  type: internal

>>>          Port "int-br-eth2"

>>>              Interface "int-br-eth2"

>>>          Port "tapc5195287-8f"

>>>              tag: 1

>>>              Interface "tapc5195287-8f"

>>>                  type: internal

>>>          Port "tap5a3a1ef2-9c"

>>>              tag: 4095

>>>              Interface "tap5a3a1ef2-9c"

>>>                  type: internal

>>>      Bridge br-ex

>>>          Port phy-br-ex

>>>              Interface phy-br-ex

>>>          Port "qg-0b1ba3a7-16"

>>>              Interface "qg-0b1ba3a7-16"

>>>                  type: internal

>>>          Port "eth1"

>>>              Interface "eth1"

>>>          Port br-ex

>>>              Interface br-ex

>>>                  type: internal

>>>      ovs_version: "1.11.0"

>>>

>>> On 04/12/14 15:26, Erich Weiler wrote:

>>>> And one more, I think I found something.  On the network node, I

>>>> see on

>>>> phy-br-eth2:

>>>>

>>>> # tcpdump -n -e -vv -i phy-br-eth2

>>>>

>>>> tcpdump: WARNING: phy-br-eth2: no IPv4 address assigned

>>>> tcpdump: listening on phy-br-eth2, link-type EN10MB (Ethernet),

>>>> capture size 65535 bytes

>>>> 15:23:47.394584 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q

>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len

>>>> 6),

>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46

>>>> 15:23:47.394605 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype

>>>> 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet

>>>> (len 6), IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5,

>>>> length 28

>>>> 15:23:48.394677 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q

>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len

>>>> 6),

>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46

>>>> 15:23:48.394693 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype

>>>> 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet

>>>> (len 6), IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5,

>>>> length 28

>>>> 15:23:49.395364 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q

>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len

>>>> 6),

>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length 46

>>>> 15:23:49.395379 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype

>>>> 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Ethernet

>>>> (len 6), IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5,

>>>> length 28

>>>>

>>>> It sees the ARP request on VLAN 200, but it looks like it replies

>>>> on VLAN 1.  I'm not sure why it would reply on VLAN 1...  I'm not

>>>> entirely sure how phy-br-eth2 works though...

>>>>

>>>> On 4/12/14, 2:07 PM, Erich Weiler wrote:

>>>>> Sorry, one more...  ;)

>>>>>

>>>>> Wile trying to ping the dnsmasq host (10.200.0.3) from the VM

>>>>> (statically configured to 10.200.0.6), if I tcpdump specifically

>>>>> on vlan

>>>>> 200 on the network node:

>>>>>

>>>>> # tcpdump -ni eth2 vlan 200

>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned

>>>>> tcpdump: verbose output suppressed, use -v or -vv for full

>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),

>>>>> capture size 65535 bytes

>>>>> 13:55:18.826384 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:55:18.956941 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>> 13:55:19.825947 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:55:20.826056 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:55:20.960779 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>> 13:55:21.826718 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:55:22.825870 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:55:22.976173 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>> 13:55:23.825964 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:55:24.826730 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:55:24.981795 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>

>>>>> I see the ARP requests coming in.  But I don't see any replies, at

>>>>> least on vlan 200.  If I tcpdump with a specific VLAN, I do see

>>>>> the reply on the network node:

>>>>>

>>>>> # tcpdump -i eth2

>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned

>>>>> tcpdump: verbose output suppressed, use -v or -vv for full

>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),

>>>>> capture size 65535 bytes

>>>>> 13:58:11.830842 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:58:11.830878 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui

>>>>> Unknown), length 28

>>>>> 13:58:12.833434 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>> length 46

>>>>> 13:58:12.833457 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5 (oui

>>>>> Unknown), length 28

>>>>> 13:58:13.494885 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>> 13:58:13.495217 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>> 13:58:13.495577 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>> 13:58:13.496017 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>> 13:58:13.496355 STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>

>>>>> So maybe the ARP reply is being sent out, but maybe just not over

>>>>> VLAN

>>>>> 200 where it needs to go, very strange...

>>>>>

>>>>> Also more verbose tcpdump on the network node:

>>>>>

>>>>> # tcpdump -n -e -vv -i eth2

>>>>>

>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned

>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture

>>>>> size

>>>>> 65535 bytes

>>>>> 14:04:10.848783 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q

>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len

>>>>> 6),

>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length

>>>>> 46

>>>>> 14:04:10.848815 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype

>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet

>>>>> (len 6),

>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28

>>>>> 14:04:10.918013 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.918374 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.918705 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 204, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80cc.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.919171 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 202, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80ca.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.919525 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 203, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80cb.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.919889 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 206, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80ce.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.920346 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 208, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80d0.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.920682 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 209, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80d1.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.921010 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 200, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80c8.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:10.921295 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 207, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80cf.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:11.287103 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cc, 802.3,

>>>>> length

>>>>> 188: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command,

>>>>> ctrl

>>>>> 0x03: oui Cisco (0x00000c), pid VTP (0x2003): VTPv1, Message Join

>>>>> message (0x04), length 166

>>>>>      Domain name: cbse, Rsvd: 0

>>>>> 14:04:11.848874 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q

>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len

>>>>> 6),

>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length

>>>>> 46

>>>>> 14:04:11.848896 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype

>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet

>>>>> (len 6),

>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28

>>>>> 14:04:12.851406 fa:16:3e:5d:11:5b > Broadcast, ethertype 802.1Q

>>>>> (0x8100), length 64: vlan 200, p 0, ethertype ARP, Ethernet (len

>>>>> 6),

>>>>> IPv4 (len 4), Request who-has 10.200.0.3 tell 10.200.0.6, length

>>>>> 46

>>>>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype

>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet

>>>>> (len 6),

>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28

>>>>> 14:04:12.923686 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 205, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80cd.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>> 14:04:12.924039 88:43:e1:ff:76:c8 > 01:00:0c:cc:cc:cd, ethertype

>>>>> 802.1Q (0x8100), length 68: vlan 201, p 7, LLC, dsap SNAP (0xaa)

>>>>> Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco

>>>>> (0x00000c), pid PVST

>>>>> (0x010b): STP 802.1d, Config, Flags [none], bridge-id

>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>      message-age 3.00s, max-age 20.00s, hello-time 2.00s,

>>>>> forwarding-delay 15.00s

>>>>>      root-id 80c9.00:13:1a:3d:3e:80, root-pathcost 9008

>>>>>

>>>>> That one seems to indicate the ARP reply is being sent on vlan

>>>>> 200..???  :

>>>>>

>>>>> 14:04:12.851428 fa:16:3e:c2:ec:b5 > fa:16:3e:5d:11:5b, ethertype

>>>>> 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet

>>>>> (len 6),

>>>>> IPv4 (len 4), Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5, length 28

>>>>>

>>>>> I'm totally confused...  ARP should be working according to that....

>>>>>

>>>>> On 4/12/14, 1:44 PM, Erich Weiler wrote:

>>>>>> I tried additionally checking the dnsmasq instance on the network

>>>>>> node while trying to ping it from the VM (I set the VMs IP

>>>>>> statically to

>>>>>> 10.200.0.6 and tried to ping dnsmasq which is 10.200.0.3).  From

>>>>>> the network node, I did:

>>>>>>

>>>>>> # ip netns

>>>>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f

>>>>>>

>>>>>> [root at cloudnet-1 network-scripts]# ip netns exec

>>>>>> qdhcp-e4448083-ec61-4293-ad0e-62239986965f ifconfig

>>>>>> lo        Link encap:Local Loopback

>>>>>>            inet addr:127.0.0.1  Mask:255.0.0.0

>>>>>>            inet6 addr: ::1/128 Scope:Host

>>>>>>            UP LOOPBACK RUNNING  MTU:16436  Metric:1

>>>>>>            RX packets:17 errors:0 dropped:0 overruns:0 frame:0

>>>>>>            TX packets:17 errors:0 dropped:0 overruns:0 carrier:0

>>>>>>            collisions:0 txqueuelen:0

>>>>>>            RX bytes:3184 (3.1 KiB)  TX bytes:3184 (3.1 KiB)

>>>>>>

>>>>>> tapc5195287-8f Link encap:Ethernet  HWaddr FA:16:3E:C2:EC:B5

>>>>>>            inet addr:10.200.0.3  Bcast:10.200.255.255

>>>>>> Mask:255.255.0.0

>>>>>>            inet6 addr: fe80::f816:3eff:fec2:ecb5/64 Scope:Link

>>>>>>            UP BROADCAST RUNNING  MTU:1500  Metric:1

>>>>>>            RX packets:2369 errors:0 dropped:0 overruns:0 frame:0

>>>>>>            TX packets:2220 errors:0 dropped:0 overruns:0 carrier:0

>>>>>>            collisions:0 txqueuelen:0

>>>>>>            RX bytes:158390 (154.6 KiB)  TX bytes:94804 (92.5 KiB)

>>>>>>

>>>>>> # ip netns exec qdhcp-e4448083-ec61-4293-ad0e-62239986965f

>>>>>> tcpdump -i tapc5195287-8f

>>>>>>

>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full

>>>>>> protocol decode listening on tapc5195287-8f, link-type EN10MB

>>>>>> (Ethernet), capture size

>>>>>> 65535 bytes

>>>>>> 13:38:03.997934 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:03.997948 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:03.999280 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:04.998033 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:04.998044 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:04.999187 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:05.999197 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:06.001111 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:06.001121 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:06.998171 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:06.998181 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:07.998279 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:07.998289 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:09.001444 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:09.001454 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:09.004258 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:09.998436 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:09.998446 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:10.004161 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:10.998488 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:10.998498 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:11.004181 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:12.001742 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:12.001752 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:12.998647 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:12.998657 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:13.998762 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:13.998772 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:14.009308 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:15.000801 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:15.000812 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:24.019260 ARP, Request who-has 10.200.0.1 tell 10.200.0.3,

>>>>>> length 28

>>>>>> 13:38:24.998348 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:34.999130 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:34.999143 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:36.003134 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:36.003147 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:36.999278 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:36.999291 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:37.999362 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:37.999375 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:39.003456 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:39.003469 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:40.003526 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:40.003539 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:41.003609 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:41.003622 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:42.003783 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:42.003796 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>> 13:38:43.003781 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>> length 46

>>>>>> 13:38:43.003794 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>> (oui Unknown), length 28

>>>>>>

>>>>>> I assume because of the fact I'm seeing the ARP requests and

>>>>>> replies on the dnsmasq interface that VLANs are working, as they

>>>>>> are supposed to be talking on VLAN 200, but ARP isn't working for

>>>>>> some reason, or at least the VM isn't getting the ARP reply.

>>>>>>

>>>>>> I tried setting my data interfaces on my network node and my

>>>>>> compute node to promiscuous mode (they weren't before), but it

>>>>>> didn't seem to help.  Are those interfaces supposed to be in promiscuous mode?

>>>>>>

>>>>>>

>>>>>>

>>>>>> On 4/12/14, 1:02 PM, Erich Weiler wrote:

>>>>>>> Hi Y'all,

>>>>>>>

>>>>>>> I'm making some progress on my neutron VLAN deployment issues,

>>>>>>> but it's still not working as expected.  I have my compute nodes

>>>>>>> data port connected to a switchport that is a trunk, allowing

>>>>>>> VLANs 200-209 to flow over the trunk.  The neutron node also has

>>>>>>> its internal data port on the same switch, on a trunk port, also

>>>>>>> allowing VLANs 200-209 to flow over it.

>>>>>>>

>>>>>>> I *think* I have OVS configured correctly on the neutron network

>>>>>>> node and the compute nodes.

>>>>>>>

>>>>>>> To test I just created an internal network (on physnet1) callled

>>>>>>> test-net on segmentation-id 200 (network 10.200.0.0/16).  So I

>>>>>>> assume it will use VLAN 200.  I then created a VM and attached

>>>>>>> it to that network.

>>>>>>>   But when the VM boots, it does not get an IP address from dnsmasq.

>>>>>>>

>>>>>>> Back on the network node, I see dnsmasq running on that VLAN, I

>>>>>>> think:

>>>>>>>

>>>>>>> # ps -afe | grep masq

>>>>>>> nobody   16674     1  0 Apr10 ?        00:00:00 dnsmasq --no-hosts

>>>>>>> --no-resolv --strict-order --bind-interfaces

>>>>>>> --interface=tapc5195287-8f --except-interface=lo

>>>>>>> --pid-file=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-6223998

>>>>>>> 6965f/pid

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>> --dhcp-hostsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-6

>>>>>>> 2239986965f/host

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>> --addn-hosts=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62239

>>>>>>> 986965f/addn_hosts

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>> --dhcp-optsfile=/var/lib/neutron/dhcp/e4448083-ec61-4293-ad0e-62

>>>>>>> 239986965f/opts

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>>

>>>>>>> --leasefile-ro --dhcp-range=tag0,10.200.0.0,static,86400s

>>>>>>> --dhcp-lease-max=65536 --conf-file= --domain=openstacklocal

>>>>>>>

>>>>>>> And I tried doing a tcpdump on eth2 (which is the data interface

>>>>>>> with the VLANs) while doing a DHCPDISCOVER from the VM:

>>>>>>>

>>>>>>> # tcpdump -i eth2

>>>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned

>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full

>>>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),

>>>>>>> capture size 65535 bytes

>>>>>>> 12:43:46.440221 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.440563 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.440896 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.441414 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.441672 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.442001 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.442502 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.442830 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.443171 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:46.443470 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:47.718633 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:

>>>>>>> BOOTP/DHCP,

>>>>>>> Request from fa:16:3e:5d:11:5b (oui Unknown), length 280

>>>>>>> 12:43:47.719183 IP 10.200.0.3.bootps > 10.200.0.6.bootpc:

>>>>>>> BOOTP/DHCP,

>>>>>>> Reply, length 325

>>>>>>> 12:43:48.447761 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.448109 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.448593 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.448896 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.449266 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.449672 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.450010 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.450385 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.450678 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:48.451115 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:49.946348 VTPv1, Message Join message (0x04), length 166

>>>>>>> 12:43:50.459700 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.460184 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.460529 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.460867 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.461195 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.461625 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.461956 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.462335 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.462745 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:50.463072 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.455118 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.455591 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.455930 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.456258 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.456575 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.457053 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.457414 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.457702 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.458171 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.458503 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:52.718355 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,

>>>>>>> length 28

>>>>>>> 12:43:53.718209 ARP, Request who-has 10.200.0.6 tell 10.200.0.3,

>>>>>>> length 28

>>>>>>> 12:43:54.460398 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.460871 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.461187 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.461588 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.461984 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.462361 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.462652 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.462981 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.463474 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:43:54.463751 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>>

>>>>>>> Where 10.200.0.3 is the IP address of the dnsmasq instance on

>>>>>>> the network node and 10.200.0.6 is the IP (supposedly) of the

>>>>>>> VM, from what nova tells me.  It seems the traffic is getting to

>>>>>>> the network node, but the VM never gets the IP.  Even if I

>>>>>>> configure the VM statically to be 10.200.0.6, I still can't ping

>>>>>>> the DHCP server on 10.200.0.3.

>>>>>>>

>>>>>>> It looks like ARP requests are getting sent out for the DHCP

>>>>>>> server hen I ping (as seen fro the network node):

>>>>>>>

>>>>>>> # tcpdump -i eth2

>>>>>>> tcpdump: WARNING: eth2: no IPv4 address assigned

>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full

>>>>>>> protocol decode listening on eth2, link-type EN10MB (Ethernet),

>>>>>>> capture size 65535 bytes

>>>>>>> 12:57:08.871232 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.871717 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.872063 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.872421 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.872819 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.873212 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.873507 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.873839 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.874329 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:08.874621 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:09.041413 VTPv1, Message Join message (0x04), length 166

>>>>>>> 12:57:09.289413 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>>> length 46

>>>>>>> 12:57:09.289435 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>>> (oui Unknown), length 28

>>>>>>> 12:57:10.289706 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>>> length 46

>>>>>>> 12:57:10.289728 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>>> (oui Unknown), length 28

>>>>>>> 12:57:10.876689 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cd.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.877031 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c9.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.877439 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cc.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.877843 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ca.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.878190 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cb.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.878542 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0ce.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.878824 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d0.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.879282 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0d1.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.879612 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0c8.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:10.879933 STP 802.1d, Config, Flags [none], bridge-id

>>>>>>> c0cf.c4:7d:4f:ec:d3:00.81a5, length 42

>>>>>>> 12:57:11.289582 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>>> length 46

>>>>>>> 12:57:11.289603 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>>> (oui Unknown), length 28

>>>>>>> 12:57:12.289668 ARP, Request who-has 10.200.0.3 tell 10.200.0.6,

>>>>>>> length 46

>>>>>>> 12:57:12.289689 ARP, Reply 10.200.0.3 is-at fa:16:3e:c2:ec:b5

>>>>>>> (oui Unknown), length 28

>>>>>>>

>>>>>>> But the VM doesn't get any ping replies...  Maybe ARP is having

>>>>>>> issues?

>>>>>>>

>>>>>>> Can anyone see anything wrong with this picture?

>>>>>>>

>>>>>>> Thanks!!

>>>>>>> -erich



_______________________________________________

Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>

Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140416/94702fb3/attachment.html>


More information about the Openstack mailing list