[Neutron] OVS forwarding issues
Dear colleagues,
just faced an issue with Openvswitch, which looks strange for me. The problem is that any particular VM receives a lot of packets, which are unicasted: - from other VMs which reside on the same host (let's name them "local VMs") - to other VMs which reside on other hosts (let's name them "remote VMs")
Long output from "ovs-ofctl dump-flows br-int" which, as far as I can narrow, ends there:
# ovs-ofctl dump-flows br-int |grep " table=94," |egrep "n_packets=[123456789]" cookie=0xaf6b1435fe826bdf, duration=2952350.695s, table=94, n_packets=291494723, n_bytes=40582103074, idle_age=0, hard_age=65534, priority=1 actions=NORMAL
coming to normal processing (classic MAC learning). Looking into br-int MAC-table (ovs-appctl fdb/show br-int) shows, that there are really no MAC addresses of remote VMs and br-int behaves in the right way, flooding unknown unicast to all ports in this L2 segment.
Of course, there is br-tun which connected over vxlan to all other hosts and to br-int:
Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-0a960008" Interface "vxlan-0a960008" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.150.0.5", out_key=flow, remote_ip="10.150.0.8"} [ ... ] Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun}
but MAC table on br-tun is empty as well:
# ovs-appctl fdb/show br-tun port VLAN MAC Age #
Finally, packets get to destination, while being copied to all ports on source host, which is serious security issue.
I do not think so conceived by design, I rather think we missed something in configuration. Can anybody point me where we're wrong and help with this issue?
We're using Openstack Rocky and OVS 2.10.0 under Ubuntu 16.04. Network configuration is:
@controller: # cat /etc/neutron/plugins/ml2/ml2_conf.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [ml2] type_drivers = flat,vxlan tenant_network_types = vxlan mechanism_drivers = l2population,openvswitch extension_drivers = port_security,qos,dns_domain_ports [ml2_type_flat] flat_networks = provider [ml2_type_geneve] [ml2_type_gre] [ml2_type_vlan] [ml2_type_vxlan] vni_ranges = 400:400000 [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true
@agent: # cat /etc/neutron/plugins/ml2/openvswitch_agent.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [agent] tunnel_types = vxlan l2_population = true arp_responder = true extensions = qos [ovs] local_ip = 10.150.0.5 bridge_mappings = provider:br-ex [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true [xenapi]
Thank you.
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
Hi,
This is a known issue with the openvswitch firewall[1].
firewall_driver = openvswitch
I recommend running iptables_hybrid until that is resolved.
[1] https://bugs.launchpad.net/neutron/+bug/1732067
James Denton Network Engineer Rackspace Private Cloud james.denton@rackspace.com
From: Volodymyr Litovka doka.ua@gmx.com Date: Monday, November 11, 2019 at 12:10 PM To: "openstack-discuss@lists.openstack.org" openstack-discuss@lists.openstack.org Cc: "doka.ua@gmx.com" doka.ua@gmx.com Subject: [Neutron] OVS forwarding issues
CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!
Dear colleagues,
just faced an issue with Openvswitch, which looks strange for me. The problem is that any particular VM receives a lot of packets, which are unicasted: - from other VMs which reside on the same host (let's name them "local VMs") - to other VMs which reside on other hosts (let's name them "remote VMs")
Long output from "ovs-ofctl dump-flows br-int" which, as far as I can narrow, ends there:
# ovs-ofctl dump-flows br-int |grep " table=94," |egrep "n_packets=[123456789]" cookie=0xaf6b1435fe826bdf, duration=2952350.695s, table=94, n_packets=291494723, n_bytes=40582103074, idle_age=0, hard_age=65534, priority=1 actions=NORMAL
coming to normal processing (classic MAC learning). Looking into br-int MAC-table (ovs-appctl fdb/show br-int) shows, that there are really no MAC addresses of remote VMs and br-int behaves in the right way, flooding unknown unicast to all ports in this L2 segment.
Of course, there is br-tun which connected over vxlan to all other hosts and to br-int:
Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-0a960008" Interface "vxlan-0a960008" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.150.0.5", out_key=flow, remote_ip="10.150.0.8"} [ ... ] Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun}
but MAC table on br-tun is empty as well:
# ovs-appctl fdb/show br-tun port VLAN MAC Age #
Finally, packets get to destination, while being copied to all ports on source host, which is serious security issue.
I do not think so conceived by design, I rather think we missed something in configuration. Can anybody point me where we're wrong and help with this issue?
We're using Openstack Rocky and OVS 2.10.0 under Ubuntu 16.04. Network configuration is:
@controller: # cat /etc/neutron/plugins/ml2/ml2_conf.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [ml2] type_drivers = flat,vxlan tenant_network_types = vxlan mechanism_drivers = l2population,openvswitch extension_drivers = port_security,qos,dns_domain_ports [ml2_type_flat] flat_networks = provider [ml2_type_geneve] [ml2_type_gre] [ml2_type_vlan] [ml2_type_vxlan] vni_ranges = 400:400000 [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true
@agent: # cat /etc/neutron/plugins/ml2/openvswitch_agent.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [agent] tunnel_types = vxlan l2_population = true arp_responder = true extensions = qos [ovs] local_ip = 10.150.0.5 bridge_mappings = provider:br-ex [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true [xenapi]
Thank you.
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
Hi colleagues,
thanks for the pointing on this. Can anybody _assume_ whether this bug affects also ML2/OVN implementation of networking?
I was looking into OVN sometimes ago, but due to lack of resources skipped this research, now I think it makes sense to return back to this question.
Thank you.
On 11.11.2019 19:38, James Denton wrote:
Hi,
This is a known issue with the openvswitch firewall[1].
firewall_driver = openvswitch
I recommend running iptables_hybrid until that is resolved.
[1] https://bugs.launchpad.net/neutron/+bug/1732067 https://bugs.launchpad.net/neutron/+bug/1732067
James Denton
Network Engineer
Rackspace Private Cloud
james.denton@rackspace.com
*From: *Volodymyr Litovka doka.ua@gmx.com *Date: *Monday, November 11, 2019 at 12:10 PM *To: *"openstack-discuss@lists.openstack.org" openstack-discuss@lists.openstack.org *Cc: *"doka.ua@gmx.com" doka.ua@gmx.com *Subject: *[Neutron] OVS forwarding issues
*CAUTION:*This message originated externally, please use caution when clicking on links or opening attachments!
Dear colleagues,
just faced an issue with Openvswitch, which looks strange for me. The problem is that any particular VM receives a lot of packets, which are unicasted:
- from other VMs which reside on the same host (let's name them "local
VMs")
- to other VMs which reside on other hosts (let's name them "remote VMs")
Long output from "ovs-ofctl dump-flows br-int" which, as far as I can narrow, ends there:
# ovs-ofctl dump-flows br-int |grep " table=94," |egrep "n_packets=[123456789]" cookie=0xaf6b1435fe826bdf, duration=2952350.695s, table=94, n_packets=291494723, n_bytes=40582103074, idle_age=0, hard_age=65534, priority=1 actions=NORMAL
coming to normal processing (classic MAC learning). Looking into br-int MAC-table (ovs-appctl fdb/show br-int) shows, that there are really no MAC addresses of remote VMs and br-int behaves in the right way, flooding unknown unicast to all ports in this L2 segment.
Of course, there is br-tun which connected over vxlan to all other hosts and to br-int:
Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-0a960008" Interface "vxlan-0a960008" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.150.0.5", out_key=flow, remote_ip="10.150.0.8"} [ ... ] Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun}
but MAC table on br-tun is empty as well:
# ovs-appctl fdb/show br-tun port VLAN MAC Age #
Finally, packets get to destination, while being copied to all ports on source host, which is serious security issue.
I do not think so conceived by design, I rather think we missed something in configuration. Can anybody point me where we're wrong and help with this issue?
We're using Openstack Rocky and OVS 2.10.0 under Ubuntu 16.04. Network configuration is:
@controller: # cat /etc/neutron/plugins/ml2/ml2_conf.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [ml2] type_drivers = flat,vxlan tenant_network_types = vxlan mechanism_drivers = l2population,openvswitch extension_drivers = port_security,qos,dns_domain_ports [ml2_type_flat] flat_networks = provider [ml2_type_geneve] [ml2_type_gre] [ml2_type_vlan] [ml2_type_vxlan] vni_ranges = 400:400000 [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true
@agent: # cat /etc/neutron/plugins/ml2/openvswitch_agent.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [agent] tunnel_types = vxlan l2_population = true arp_responder = true extensions = qos [ovs] local_ip = 10.150.0.5 bridge_mappings = provider:br-ex [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true [xenapi]
Thank you.
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
I seem to recall checking this when the issue was first discovered, and OVN did not appear to implement the same flow rules that resulted in the issue. I don’t have a live environment to test with, though.
James Denton Network Engineer Rackspace Private Cloud james.denton@rackspace.com
From: Volodymyr Litovka doka.ua@gmx.com Date: Friday, November 15, 2019 at 3:55 AM To: James Denton james.denton@rackspace.com, "openstack-discuss@lists.openstack.org" openstack-discuss@lists.openstack.org, Slawek Kaplonski skaplons@redhat.com Cc: "doka.ua@gmx.com" doka.ua@gmx.com Subject: Re: [Neutron] OVS forwarding issues
CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!
Hi colleagues,
thanks for the pointing on this. Can anybody _assume_ whether this bug affects also ML2/OVN implementation of networking?
I was looking into OVN sometimes ago, but due to lack of resources skipped this research, now I think it makes sense to return back to this question.
Thank you.
On 11.11.2019 19:38, James Denton wrote: Hi,
This is a known issue with the openvswitch firewall[1].
firewall_driver = openvswitch
I recommend running iptables_hybrid until that is resolved.
[1] https://bugs.launchpad.net/neutron/+bug/1732067
James Denton Network Engineer Rackspace Private Cloud james.denton@rackspace.commailto:james.denton@rackspace.com
From: Volodymyr Litovka doka.ua@gmx.commailto:doka.ua@gmx.com Date: Monday, November 11, 2019 at 12:10 PM To: "openstack-discuss@lists.openstack.org"mailto:openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org Cc: "doka.ua@gmx.com"mailto:doka.ua@gmx.com doka.ua@gmx.commailto:doka.ua@gmx.com Subject: [Neutron] OVS forwarding issues
CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!
Dear colleagues,
just faced an issue with Openvswitch, which looks strange for me. The problem is that any particular VM receives a lot of packets, which are unicasted: - from other VMs which reside on the same host (let's name them "local VMs") - to other VMs which reside on other hosts (let's name them "remote VMs")
Long output from "ovs-ofctl dump-flows br-int" which, as far as I can narrow, ends there:
# ovs-ofctl dump-flows br-int |grep " table=94," |egrep "n_packets=[123456789]" cookie=0xaf6b1435fe826bdf, duration=2952350.695s, table=94, n_packets=291494723, n_bytes=40582103074, idle_age=0, hard_age=65534, priority=1 actions=NORMAL
coming to normal processing (classic MAC learning). Looking into br-int MAC-table (ovs-appctl fdb/show br-int) shows, that there are really no MAC addresses of remote VMs and br-int behaves in the right way, flooding unknown unicast to all ports in this L2 segment.
Of course, there is br-tun which connected over vxlan to all other hosts and to br-int:
Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-0a960008" Interface "vxlan-0a960008" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.150.0.5", out_key=flow, remote_ip="10.150.0.8"} [ ... ] Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun}
but MAC table on br-tun is empty as well:
# ovs-appctl fdb/show br-tun port VLAN MAC Age #
Finally, packets get to destination, while being copied to all ports on source host, which is serious security issue.
I do not think so conceived by design, I rather think we missed something in configuration. Can anybody point me where we're wrong and help with this issue?
We're using Openstack Rocky and OVS 2.10.0 under Ubuntu 16.04. Network configuration is:
@controller: # cat /etc/neutron/plugins/ml2/ml2_conf.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [ml2] type_drivers = flat,vxlan tenant_network_types = vxlan mechanism_drivers = l2population,openvswitch extension_drivers = port_security,qos,dns_domain_ports [ml2_type_flat] flat_networks = provider [ml2_type_geneve] [ml2_type_gre] [ml2_type_vlan] [ml2_type_vxlan] vni_ranges = 400:400000 [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true
@agent: # cat /etc/neutron/plugins/ml2/openvswitch_agent.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [agent] tunnel_types = vxlan l2_population = true arp_responder = true extensions = qos [ovs] local_ip = 10.150.0.5 bridge_mappings = provider:br-ex [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true [xenapi]
Thank you.
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
Hi,
If You are using ovs firewall driver, it is known issue there. See bug [1] for details. There is proposal how to fix it in [2] but it's not perfect and still require some more work to do.
[1] https://bugs.launchpad.net/neutron/+bug/1732067 [2] https://bugs.launchpad.net/neutron/+bug/1841622
On Mon, Nov 11, 2019 at 07:06:43PM +0200, Volodymyr Litovka wrote:
Dear colleagues,
just faced an issue with Openvswitch, which looks strange for me. The problem is that any particular VM receives a lot of packets, which are unicasted:
- from other VMs which reside on the same host (let's name them "local VMs")
- to other VMs which reside on other hosts (let's name them "remote VMs")
Long output from "ovs-ofctl dump-flows br-int" which, as far as I can narrow, ends there:
# ovs-ofctl dump-flows br-int |grep " table=94," |egrep "n_packets=[123456789]" cookie=0xaf6b1435fe826bdf, duration=2952350.695s, table=94, n_packets=291494723, n_bytes=40582103074, idle_age=0, hard_age=65534, priority=1 actions=NORMAL
coming to normal processing (classic MAC learning). Looking into br-int MAC-table (ovs-appctl fdb/show br-int) shows, that there are really no MAC addresses of remote VMs and br-int behaves in the right way, flooding unknown unicast to all ports in this L2 segment.
Of course, there is br-tun which connected over vxlan to all other hosts and to br-int:
Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-0a960008" Interface "vxlan-0a960008" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.150.0.5", out_key=flow, remote_ip="10.150.0.8"} [ ... ] Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun}
but MAC table on br-tun is empty as well:
# ovs-appctl fdb/show br-tun port VLAN MAC Age #
Finally, packets get to destination, while being copied to all ports on source host, which is serious security issue.
I do not think so conceived by design, I rather think we missed something in configuration. Can anybody point me where we're wrong and help with this issue?
We're using Openstack Rocky and OVS 2.10.0 under Ubuntu 16.04. Network configuration is:
@controller: # cat /etc/neutron/plugins/ml2/ml2_conf.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [ml2] type_drivers = flat,vxlan tenant_network_types = vxlan mechanism_drivers = l2population,openvswitch extension_drivers = port_security,qos,dns_domain_ports [ml2_type_flat] flat_networks = provider [ml2_type_geneve] [ml2_type_gre] [ml2_type_vlan] [ml2_type_vxlan] vni_ranges = 400:400000 [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true
@agent: # cat /etc/neutron/plugins/ml2/openvswitch_agent.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [agent] tunnel_types = vxlan l2_population = true arp_responder = true extensions = qos [ovs] local_ip = 10.150.0.5 bridge_mappings = provider:br-ex [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true [xenapi]
Thank you.
-- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison
participants (3)
-
James Denton
-
Slawek Kaplonski
-
Volodymyr Litovka