Open Stack

Hi,
I've been running against openstack-dev (master branch) using the stackforge/networking-ovs-dpdk master branch (OVS GIT TAG 1e77bbe565bbf5ae7f4c47f481a4097d666d3d68), using the single-node local.conf file on Ubuntu 15.04.  I've had to patch a few things to get past ERRORs during start:
- disable/purge apparmor
- patch ovs-dpdk-init to find correct qemu group and launch ovs-dpdk with sg
- patch ovs_dvr_neutron_agent.py to change default datapath_type to be netdev
- modify ml2_conf.ini to have [ovs] datapath_type=netdev
- create a symlink between /usr/var/run/openvsitch and /var/run/openvswitch

Everything appears to be working from a horizon point of view, I can launch VMs, create routers/networks, etc.

However, I've tried to figure out how to get two vms using ovs-dpdk (ovs-vswitchd --dpdk ...) to be able to ping eachother (or do anything network related for that matter - like get an ipv4 address (via dhcp), ping the gateway/router, etc), but to no avail.

I'm wondering if there is a flow that is bogus as when I dump the flows:

# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=5234.334s, table=0, n_packets=0, n_bytes=0, idle_age=5234, priority=3,in_port=2,dl_vlan=2001 actions=mod_vlan_vid:1,NORMAL  cookie=0x0, duration=5246.680s, table=0, n_packets=0, n_bytes=0, idle_age=5246, priority=2,in_port=1 actions=drop  cookie=0x0, duration=5246.613s, table=0, n_packets=0, n_bytes=0, idle_age=5246, priority=2,in_port=2 actions=drop  cookie=0x0, duration=5246.744s, table=0, n_packets=46828985, n_bytes=2809779780, idle_age=0, priority=0 actions=NORMAL  cookie=0x0, duration=5246.740s, table=23, n_packets=0, n_bytes=0, idle_age=5246, priority=0 actions=drop  cookie=0x0, duration=5246.738s, table=24, n_packets=0, n_bytes=0, idle_age=5246, priority=0 actions=drop

There is only one flow that ever gets any packets, and it gets millions of them apparently.  Viewing the number of packets sent on an interface (via ifconfig) doesn't have any interfaces with near that many packets.

Dumping ports doesn't show any stats near those values either:
===========================================================================================
ovs-ofctl dump-ports br-int
OFPST_PORT reply (xid=0x2): 7 ports
  port  6: rx pkts=0, bytes=?, drop=?, errs=?, frame=?, over=?, crc=?
           tx pkts=636, bytes=?, drop=0, errs=?, coll=?
  port  4: rx pkts=?, bytes=?, drop=?, errs=?, frame=?, over=?, crc=?
           tx pkts=?, bytes=?, drop=?, errs=?, coll=?
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=874, bytes=98998, drop=0, errs=0, coll=0
  port  1: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=874, bytes=95570, drop=0, errs=0, coll=0
  port  5: rx pkts=?, bytes=?, drop=?, errs=?, frame=?, over=?, crc=?
           tx pkts=?, bytes=?, drop=?, errs=?, coll=?
  port  2: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=874, bytes=95570, drop=0, errs=0, coll=0
  port  3: rx pkts=?, bytes=?, drop=?, errs=?, frame=?, over=?, crc=?
           tx pkts=?, bytes=?, drop=?, errs=?, coll=?
===========================================================================================

One thing I find odd is that showing the br-int bridge (and all the other bridges for that matter) seem to show the port_down for almost all the interfaces:
===========================================================================================
ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:00009ab69b50904d n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(int-br-em1): addr:0a:41:20:a8:6b:50
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(int-br-p6p1): addr:2a:5d:62:23:0a:60
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(tapa32182b8-ee): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 4(qr-a510b75f-f7): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 5(qr-d2f1d4a0-a9): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 6(vhucf0a0213-68): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:9a:b6:9b:50:90:4d
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 ===========================================================================================

In fact, trying to bring them up (i.e. ovs-ofctl mod-port br-int 3 up) does not change anything for any of the ports other than for the vhost user (vhuXXYYZZ ) ports.  And then it only changes the config state to 0, it doesn't change the state status (that remains showing LINK_DOWN).

Any idea of an area I should be focusing on to debug?  I've seen some posts saying to use ovs-appctl but when I try to use it complains with the message 'cannot read pidfile "/usr/var/run/openvswitch/ovs-vswitchd.pid" (No such process)'.  Even if I copy that file (copied from /opt/stack/log/ovs-vswitchd.pid) to the location it is searching for it still complains with the same message, however strace shows that it did find it, but then I think it tries to query about the lock status on it.  I'm guessing it is resulting in that process being "locked" and that that is the real error.  I don't know if that is a red herring as well.

Anyway, I appreciate all the help you've given!  I hope you or someone on the mailing list might have some hints on where I might look next.

Gabriel Black