[openstack-dev] [networking-ovn] Re: [ovs-dev] Issue when using ovn with Openstack
Ryan Moats
rmoats at us.ibm.com
Wed Jul 20 04:42:04 UTC 2016
"dev" <dev-bounces at openvswitch.org> wrote on 07/19/2016 10:44:27 PM:
> From: Chen Li <lichen.hangzhou at gmail.com>
> To: dev at openvswitch.org
> Date: 07/19/2016 10:44 PM
> Subject: [ovs-dev] Issue when using ovn with Openstack
> Sent by: "dev" <dev-bounces at openvswitch.org>
>
> Hi list,
>
> I have an all-in-one devstack environment with ovn enabled.
> I create a neutron network.
> Create a port A from the network with secgroup A
> Create a vm from the network with secgroup B.
> Secgroup B has both ICMP and tcp 22 enabled.
>
> Then I try to ping the VM from the dhcp namespace, since the Secgroup B
has
> enabled ICMP, I suppose this should work. But, unfortunately, this do
not
> work. And, the ssh failed too.
>
> Anyone can help me to solve this issue ?
>
> I did some basic checks and looks like flows are missing in table 52.
>
> Here are flows in table 52:
>
> sudo ovs-ofctl dump-flows br-int |grep table=52
>
> cookie=0x0, duration=7766.195s, table=52, n_packets=0, n_bytes=0,
> idle_age=7766,
priority=65535,icmp6,metadata=0x4,icmp_type=135,icmp_code=0
> actions=resubmit(,53)
> cookie=0x0, duration=7766.195s, table=52, n_packets=0, n_bytes=0,
> idle_age=7766,
priority=65535,icmp6,metadata=0x4,icmp_type=136,icmp_code=0
> actions=resubmit(,53)
> cookie=0x0, duration=7766.195s, table=52, n_packets=4, n_bytes=1474,
> idle_age=7744, priority=2002,udp,reg15=0x2,metadata=0x4,nw_src=
> 192.168.0.0/24,tp_src=67,tp_dst=68
> actions=load:0x1->NXM_NX_REG0[1],resubmit(,53)
> cookie=0x0, duration=7557.209s, table=52, n_packets=2, n_bytes=759,
> idle_age=7548, priority=2002,udp,reg15=0x3,metadata=0x4,nw_src=
> 192.168.0.0/24,tp_src=67,tp_dst=68
> actions=load:0x1->NXM_NX_REG0[1],resubmit(,53)
> cookie=0x0, duration=7766.195s, table=52, n_packets=0, n_bytes=0,
> idle_age=7766, priority=2001,ipv6,reg15=0x2,metadata=0x4 actions=drop
> cookie=0x0, duration=7766.195s, table=52, n_packets=2, n_bytes=676,
> idle_age=7548, priority=2001,ip,reg15=0x2,metadata=0x4 actions=drop
> cookie=0x0, duration=7557.209s, table=52, n_packets=0, n_bytes=0,
> idle_age=7557, priority=2001,ipv6,reg15=0x3,metadata=0x4 actions=drop
> cookie=0x0, duration=7557.209s, table=52, n_packets=3979,
n_bytes=389774,
> idle_age=413, priority=2001,ip,reg15=0x3,metadata=0x4 actions=drop
> cookie=0x0, duration=7766.195s, table=52, n_packets=0, n_bytes=0,
> idle_age=7766, priority=1,ipv6,metadata=0x4
> actions=load:0x1->NXM_NX_REG0[1],resubmit(,53)
> cookie=0x0, duration=7766.195s, table=52, n_packets=8, n_bytes=2733,
> idle_age=7548, priority=1,ip,metadata=0x4
> actions=load:0x1->NXM_NX_REG0[1],resubmit(,53)
> cookie=0x0, duration=7926.354s, table=52, n_packets=0, n_bytes=0,
> idle_age=7926, priority=0,metadata=0x1 actions=resubmit(,53)
> cookie=0x0, duration=7790.771s, table=52, n_packets=129, n_bytes=5418,
> idle_age=408, priority=0,metadata=0x4 actions=resubmit(,53)
>
> Here are steps how I find flows are missing in table52:
>
> ovs-dpctl show
>
> port 0: ovs-system (internal)
> port 1: br-int (internal)
> port 2: tap446ef382-f0 (internal)
> port 3: tapc7c9f581-2d (internal) => the dhcp port for the testing
> network
> port 4: o-hm0 (internal) => the port create from the
> testing network with security group A
> port 5: tap275a5a25-79 => the port for the vm in the
> testing network with security group B
>
>
> sudo ip netns exec qdhcp-e8586b01-6441-4c3d-a90d-91bb0a54ec80 arp -n
>
> Address HWtype HWaddress Flags Mask
> Iface
> 192.168.0.6 ether fa:16:3e:40:85:41 C
> tapc7c9f581-2d
> 192.168.0.12 ether fa:16:3e:5c:fe:86 C
> tapc7c9f581-2d
>
>
> sudo ip netns exec qdhcp-e8586b01-6441-4c3d-a90d-91bb0a54ec80 ping
> 192.168.0.12 => This is the IP for the VM.
> PING 192.168.0.12 (192.168.0.12) 56(84) bytes of data.
>
>
> ovs-dpctl dump-flows
>
> recirc_id(0),in_port(3),eth(src=fa:16:3e:b6:f6:25,dst=fa:16:3e:
> 5c:fe:86),eth_type(0x0806),arp(sip=192.168.0.2,tip=192.168.0.
> 12,op=1/0xff,sha=fa:16:3e:b6:f6:25,tha=00:00:00:00:00:00),
> packets:0, bytes:0, used:never,
> actions:userspace(pid=4294958325,slow_path(action))
> recirc_id(0),in_port(3),eth(src=00:00:00:00:00:00/01:00:00:00:00:
> 00,dst=fa:16:3e:5c:fe:86),eth_type(0x0800),ipv4(src=
> 192.168.0.0/255.255.255.0,proto=1,frag=no), packets:14, bytes:1372,
> used:0.974s, actions:drop
>
>
> sudo ovs-appctl ofproto/trace
> "recirc_id(0),in_port(3),eth(src=00:00:00:00:00:00/01:00:00:00:00:
> 00,dst=fa:16:3e:5c:fe:86),eth_type(0x0800),ipv4(src=
> 192.168.0.0/255.255.255.0,proto=1,frag=no)"
> => This produce a long output, and here are the end of the output:
>
> OpenFlow actions=resubmit(,52)
>
> Resubmitted flow: unchanged
>
> Resubmitted regs: reg0=0x1 reg1=0x0 reg2=0x0 reg3=0x0 reg4=0x0 reg5=0x0
> reg6=0x0 reg7=0x0 reg8=0x0 reg9=0x0 reg10=0x0 reg11=0x0 reg12=0x0
> reg13=0x0 reg14=0x1 reg15=0x3
>
> Resubmitted odp: drop
>
> Resubmitted megaflow:
>
recirc_id=0,icmp,reg0=0,reg1=0,reg2=0,reg3=0,reg4=0,reg5=0,reg6=0,reg7=0,reg8=0,reg9=0,reg14=0,reg15=0,metadata=0,in_port=4,vlan_tci=0x0000/
> 0x1000
> ,dl_src=00:00:00:00:00:00/01:00:00:00:00:00,dl_dst=fa:16:3e:5c:fe:86,nw_src=
> 192.168.0.0/24,nw_frag=no
>
> Rule: table=52 cookie=0 priority=2001,ip,reg15=0x3,metadata=0x4
>
> OpenFlow actions=drop
> Final flow:
>
icmp,reg0=0x1,reg14=0x1,reg15=0x3,metadata=0x4,in_port=4,vlan_tci=0x0000,dl_src=00:
> 00:00:00:00:00,dl_dst=fa:16:3e:5c:fe:86,nw_src=192.168.0.0,nw_d
> st=0.0.0.0,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0
> Megaflow:
> recirc_id=0,icmp,in_port=4,vlan_tci=0x0000/0x1000,dl_src=00:00:00:
> 00:00:00/01:00:00:00:00:00,dl_dst=fa:16:3e:5c:fe:86,nw_src=
> 192.168.0.0/24,nw_fra
> g=no
> Datapath actions: drop
>
>
> Here are some output from OVN commands:
>
> sudo ovn-nbctl show
>
> switch 3ce05ec4-f591-4ca7-ba54-dc4fab2ffd1b
> (neutron-e8586b01-6441-4c3d-a90d-91bb0a54ec80)
> port 2c713237-ffc7-4ff1-9e4c-95c1337545e6
> addresses: ["fa:16:3e:40:85:41 192.168.0.6"]
> port c7c9f581-2db9-4b06-86c6-bde2d1aa8ffb
> addresses: ["fa:16:3e:b6:f6:25 192.168.0.2"]
> port 275a5a25-794f-47b9-9b04-8a8da053c143
> addresses: ["fa:16:3e:5c:fe:86 192.168.0.12"]
>
>
> ovn-nbctl acl-list 3ce05ec4-f591-4ca7-ba54-dc4fab2ffd1b
>
> from-lport 1002 (inport == "275a5a25-794f-47b9-9b04-8a8da053c143" &&
ip4)
> allow-related
> from-lport 1002 (inport == "275a5a25-794f-47b9-9b04-8a8da053c143" && ip4
> && (ip4.dst == 255.255.255.255 || ip4.dst == 192.168.0.0/24) && udp &&
> udp.src == 68 && udp.dst == 67) allow
> from-lport 1002 (inport == "275a5a25-794f-47b9-9b04-8a8da053c143" &&
ip6)
> allow-related
> from-lport 1002 (inport == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" &&
ip4)
> allow-related
> from-lport 1002 (inport == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" && ip4
> && (ip4.dst == 255.255.255.255 || ip4.dst == 192.168.0.0/24) && udp &&
> udp.src == 68 && udp.dst == 67) allow
> from-lport 1002 (inport == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" &&
ip6)
> allow-related
> from-lport 1001 (inport == "275a5a25-794f-47b9-9b04-8a8da053c143" && ip)
> drop
> from-lport 1001 (inport == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" && ip)
> drop
> to-lport 1002 (outport == "275a5a25-794f-47b9-9b04-8a8da053c143" &&
ip4
> && icmp4) allow-related
> to-lport 1002 (outport == "275a5a25-794f-47b9-9b04-8a8da053c143" &&
ip4
> && ip4.src == 192.168.0.0/24 && udp && udp.src == 67 && udp.dst == 68)
allow
> to-lport 1002 (outport == "275a5a25-794f-47b9-9b04-8a8da053c143" &&
ip4
> && tcp && tcp.dst == 22) allow-related
> to-lport 1002 (outport == "275a5a25-794f-47b9-9b04-8a8da053c143" &&
ip4
> && tcp && tcp.dst == 9443) allow-related
> to-lport 1002 (outport == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" &&
ip4
> && ip4.src == 192.168.0.0/24 && udp && udp.src == 67 && udp.dst == 68)
allow
> to-lport 1002 (outport == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" &&
ip4
> && udp && udp.dst == 5555) allow-related
> to-lport 1001 (outport == "275a5a25-794f-47b9-9b04-8a8da053c143" &&
ip)
> drop
> to-lport 1001 (outport == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" &&
ip)
> drop
>
>
> ovn-sbctl lflow-list | grep ls_out_acl
>
> table=4 (ls_out_acl ), priority=0 , match=(1), action=
(next;)
> table=4 (ls_out_acl ), priority=0 , match=(1), action=
(next;)
> table=4 (ls_out_acl ), priority=65535, match=(!ct.est && ct.rel
> && !ct.new && !ct.inv), action=(next;)
> table=4 (ls_out_acl ), priority=65535, match=(ct.est && !ct.rel
> && !ct.new && !ct.inv), action=(next;)
> table=4 (ls_out_acl ), priority=65535, match=(ct.inv),
> action=(drop;)
> table=4 (ls_out_acl ), priority=65535, match=(nd), action=
(next;)
> table=4 (ls_out_acl ), priority=2002 , match=(ct.new &&
(outport
> == "275a5a25-794f-47b9-9b04-8a8da053c143" && ip4 && icmp4)),
> action=(reg0[1] = 1; next;)
> table=4 (ls_out_acl ), priority=2002 , match=(ct.new &&
(outport
> == "275a5a25-794f-47b9-9b04-8a8da053c143" && ip4 && tcp && tcp.dst ==
22)),
> action=(reg0[1] = 1; next;)
> table=4 (ls_out_acl ), priority=2002 , match=(ct.new &&
(outport
> == "275a5a25-794f-47b9-9b04-8a8da053c143" && ip4 && tcp && tcp.dst ==
> 9443)), action=(reg0[1] = 1; next;)
> table=4 (ls_out_acl ), priority=2002 , match=(ct.new &&
(outport
> == "2c713237-ffc7-4ff1-9e4c-95c1337545e6" && ip4 && udp && udp.dst ==
> 5555)), action=(reg0[1] = 1; next;)
> table=4 (ls_out_acl ), priority=2002 , match=(outport ==
> "275a5a25-794f-47b9-9b04-8a8da053c143" && ip4 && ip4.src ==
192.168.0.0/24
> && udp && udp.src == 67 && udp.dst == 68), action=(reg0[1] = 1; next;)
> table=4 (ls_out_acl ), priority=2002 , match=(outport ==
> "2c713237-ffc7-4ff1-9e4c-95c1337545e6" && ip4 && ip4.src ==
192.168.0.0/24
> && udp && udp.src == 67 && udp.dst == 68), action=(reg0[1] = 1; next;)
> table=4 (ls_out_acl ), priority=2001 , match=(outport ==
> "275a5a25-794f-47b9-9b04-8a8da053c143" && ip), action=(drop;)
> table=4 (ls_out_acl ), priority=2001 , match=(outport ==
> "2c713237-ffc7-4ff1-9e4c-95c1337545e6" && ip), action=(drop;)
> table=4 (ls_out_acl ), priority=1 , match=(ip),
> action=(reg0[1] = 1; next;)
> table=4 (ls_out_acl ), priority=0 , match=(1), action=
(next;)
>
>
> The last commit in my ovs code:
>
> commit 7efb1e09bb06270248d29c787978593b57101d4f
> Author: Pravin B Shelar <pshelar at ovn.org>
> Date: Sun Jul 17 19:24:07 2016 -0700
>
> datapath: Add support for kernel 4.5
>
> Signed-off-by: Pravin B Shelar <pshelar at ovn.org>
> Acked-by: Jesse Gross <jesse at kernel.org>
>
>
> Here are some detailed information from openstack:
>
> 1. The vm:
>
> nova show test01
> +--------------------------------------
> +----------------------------------------------------------------+
> | Property |
> Value |
> +--------------------------------------
> +----------------------------------------------------------------+
> | OS-DCF:diskConfig |
> MANUAL |
> | OS-EXT-AZ:availability_zone |
> nova |
> | OS-EXT-SRV-ATTR:host | LB-dev-chenli
|
> | OS-EXT-SRV-ATTR:hostname |
> test01 |
> | OS-EXT-SRV-ATTR:hypervisor_hostname | LB-dev-chenli
|
> | OS-EXT-SRV-ATTR:instance_name |
> instance-00000001 |
> | OS-EXT-SRV-ATTR:kernel_id |
> 261ca209-430e-4b8f-ac39-0e397df30a46 |
> | OS-EXT-SRV-ATTR:launch_index |
> 0 |
> | OS-EXT-SRV-ATTR:ramdisk_id |
> 04b6a65d-3cff-4eaf-b30b-582caa2379d7 |
> | OS-EXT-SRV-ATTR:reservation_id |
> r-wehkr5gi |
> | OS-EXT-SRV-ATTR:root_device_name |
> /dev/vda |
> | OS-EXT-SRV-ATTR:user_data |
> - |
> | OS-EXT-STS:power_state |
> 1 |
> | OS-EXT-STS:task_state |
> - |
> | OS-EXT-STS:vm_state |
> active |
> | OS-SRV-USG:launched_at |
> 2016-07-20T01:18:48.000000 |
> | OS-SRV-USG:terminated_at |
> - |
> | accessIPv4
> | |
> | accessIPv6
> | |
> | config_drive |
> True |
> | created |
> 2016-07-20T01:18:42Z |
> | description |
> - |
> | flavor | m1.tiny
> (1) |
> | hostId |
> 36ef28d2b661e38d2d07645d814903a15d62da769828b57029306ec0 |
> | host_status |
> UP |
> | id |
> 27264d62-6a7c-4fe9-be81-c06fca56ec00 |
> | image | cirros-0.3.4-x86_64-uec
> (aa86e8b5-0699-46a0-a624-7af794b21404) |
> | key_name |
> - |
> | lb-mgmt-net network |
> 192.168.0.12 |
> | locked |
> False |
> | metadata |
> {} |
> | name |
> test01 |
> | os-extended-volumes:volumes_attached |
> [] |
> | progress |
> 0 |
> | security_groups |
> lb-mgmt-sec-grp |
> | status |
> ACTIVE |
> | tags |
> [] |
> | tenant_id |
> 73aebe8aa8ab41f58d5e375a03e279bf |
> | updated |
> 2016-07-20T01:18:48Z |
> | user_id |
> 53f8c8e491e94d2fa9210f3a8e6a85e4 |
> +--------------------------------------
> +----------------------------------------------------------------+
>
> 2. the security group:
>
> neutron security-group-show lb-mgmt-sec-grp
> +----------------------
> +--------------------------------------------------------------------+
> | Field |
> Value |
> +----------------------
> +--------------------------------------------------------------------+
> | description
> | |
> | id |
> fbc0c663-f32d-4ddf-9bef-eabac9dfd8ee |
> | name |
> lb-mgmt-sec-grp |
> | security_group_rules |
> { |
> | | "remote_group_id":
> null, |
> | | "direction":
> "ingress", |
> | | "protocol":
> "icmp", |
> | | "description":
> "", |
> | | "ethertype":
> "IPv4", |
> | | "remote_ip_prefix":
> null, |
> | | "port_range_max":
> null, |
> | | "security_group_id":
> "fbc0c663-f32d-4ddf-9bef-eabac9dfd8ee", |
> | | "port_range_min":
> null, |
> | | "tenant_id":
> "73aebe8aa8ab41f58d5e375a03e279bf", |
> | | "id":
> "140677a5-5308-48b2-a5a2-bb5e17994ed5" |
> | |
> } |
> | |
> { |
> | | "remote_group_id":
> null, |
> | | "direction":
> "ingress", |
> | | "protocol":
> "tcp", |
> | | "description":
> "", |
> | | "ethertype":
> "IPv4", |
> | | "remote_ip_prefix":
> null, |
> | | "port_range_max":
> 22, |
> | | "security_group_id":
> "fbc0c663-f32d-4ddf-9bef-eabac9dfd8ee", |
> | | "port_range_min":
> 22, |
> | | "tenant_id":
> "73aebe8aa8ab41f58d5e375a03e279bf", |
> | | "id":
> "39fccc0c-f832-497a-b03d-fa0e40e3f407" |
> | |
> } |
> | |
> { |
> | | "remote_group_id":
> null, |
> | | "direction":
> "egress", |
> | | "protocol":
> null, |
> | | "description":
> "", |
> | | "ethertype":
> "IPv6", |
> | | "remote_ip_prefix":
> null, |
> | | "port_range_max":
> null, |
> | | "security_group_id":
> "fbc0c663-f32d-4ddf-9bef-eabac9dfd8ee", |
> | | "port_range_min":
> null, |
> | | "tenant_id":
> "73aebe8aa8ab41f58d5e375a03e279bf", |
> | | "id":
> "54d134c0-f4bc-4f3d-bf49-0e1d0ac9ef1c" |
> | |
> } |
> | |
> { |
> | | "remote_group_id":
> null, |
> | | "direction":
> "ingress", |
> | | "protocol":
> "tcp", |
> | | "description":
> "", |
> | | "ethertype":
> "IPv4", |
> | | "remote_ip_prefix":
> null, |
> | | "port_range_max":
> 9443, |
> | | "security_group_id":
> "fbc0c663-f32d-4ddf-9bef-eabac9dfd8ee", |
> | | "port_range_min":
> 9443, |
> | | "tenant_id":
> "73aebe8aa8ab41f58d5e375a03e279bf", |
> | | "id":
> "b3e00b04-d398-450b-b1cf-b92fd3dc37a1" |
> | |
> } |
> | |
> { |
> | | "remote_group_id":
> null, |
> | | "direction":
> "egress", |
> | | "protocol":
> null, |
> | | "description":
> "", |
> | | "ethertype":
> "IPv4", |
> | | "remote_ip_prefix":
> null, |
> | | "port_range_max":
> null, |
> | | "security_group_id":
> "fbc0c663-f32d-4ddf-9bef-eabac9dfd8ee", |
> | | "port_range_min":
> null, |
> | | "tenant_id":
> "73aebe8aa8ab41f58d5e375a03e279bf", |
> | | "id":
> "c528b1cf-b065-4498-986c-13adac4c2a0a" |
> | |
> } |
> | tenant_id |
> 73aebe8aa8ab41f58d5e375a03e279bf |
> +----------------------
> +--------------------------------------------------------------------+
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
>
http://openvswitch.org/mailman/listinfo/dev
For issues between ovn and openstack, the openstack-dev mailing
list with a tag of [networking-ovn] is another mailing list that
you should try. I've added the same in the CC of this message to
help.
You've shared a lot of information about the ovv/ovs side of the
situation, but more information on the openstack side would also
help. Since you say "all-in-one" is this running with devstack?
Is this the master branch or a stable branch of neutron? What
is the latest commit in the networking-ovn repository? What
settings are there in the neutron.ini and networking-ovn.ini files?
I've personally seen tip of the tree neutron, networking-ovn,
and ovn work together in multi-node setups, so my initial thought
is to check the code versions or configurations you are using...
Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160719/a3bfe511/attachment-0001.html>
More information about the OpenStack-dev
mailing list