Open Stack

In short term, we use veth pairs with namespace to fix the issue if performance is not impacted (Hopefully:)

If performance downgrade too much, we may consider the following:

1) DHCP agent: use veth pairs with namespace since it is not critical path.

2) L3 agent: don't create port in OSV.  Connect L3 agent without open switch

VM <---> Linux Bridge <---> open switch <--> (int-br-eth, phy-br-eth) <---> physical switch <----> vlan interface with namespace <---> L3agent

In long term, we may implement namespace and remove linux bridge.

-----Original Message-----
From: Ihar Hrachyshka [mailto:ihrachys at redhat.com] 
Sent: Friday, February 13, 2015 8:15 PM
To: openstack-dev
Cc: Jiri Benc; jpirko at redhat.com
Subject: [openstack-dev] [neutron] moving openvswitch ports between namespaces considered harmful

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi neutroners,

we** had several conversations recently with our Red Hat fellows who work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way neutron utilizes their software. Those were beneficial to both sides to understand what we do right and wrong. I was asked to share some of the points from those discussions with broader community.

===

One of the issues that came up during discussions is the way neutron connects ovs ports to namespaces. The short story is that openvswitch is not designed with namespaces in mind, and the fact that moving its ports into a different namespace works for neutron is mere coincidence, and is actually considered as a bug by openvswitch guys.

It's not just broken in theory from software design standpoint, but also in practice. Specifically,

1. ovsdb dump does not work properly for namespaces:
- - https://bugzilla.redhat.com/show_bug.cgi?id=1160340

This results in wrong statistics and other data collected for these ports;

2. We suspect that the following kernel crash is triggered because of our usage of the feature that is actually a bug:
- - https://bugs.launchpad.net/neutron/+bug/1418097

Quoting Jiri Benc,

"The problem is openvswitch does not support its ports to be moved to a different name space. The fact that it's possible to do that is a bug - such operation should have been denied. Unfortunately, due to a missing check, it's not been denied. Such setup does not work reliably, though, and leads to various issues from incorrect resource accounting to kernel crashes.

We're aware of the bug but we don't have any easy way to fix it. The most obvious option, disabling moving of ovs ports to different name spaces, would be easy to do but it would break Neutron. The other option, making ovs to work across net name spaces, is hard and will require addition of different kernel APIs and large changes in ovs user space daemons. This constitutes tremendous amount of work."

The tracker bug on openvswitch side is:
- - https://bugzilla.redhat.com/show_bug.cgi?id=1160340

So in the best case, we may expect openvswitch to properly support the feature in long term, but short term it won't work, especially while neutron expects other features implemented in openvswitch for it (like NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few).

We could try to handle the issue neutron side. We can fix it by utilizing veth pairs to get into namespaces, but it may mean worse performance, and will definitely require proper benchmarking to see whether we can live with performance drop.

===

There were other suggestions on how we can enhance our way of usage of openvswitch. Among those, getting rid of linux bridge used for security groups, with special attention on getting rid of ebtables
(sic!) for they are a lot slower than iptables; getting rid of veth pair for instance ports.

===

I really encourage everyone to check the following video from devconf.cz 2015 on all that and more at:

- - https://www.youtube.com/watch?v=BlLD-qh9EBQ

Among other things, you will find presentation of plotnetcfg tool to create nice graphs of openstack state.

If you're lazy enough and want to switch directly to the analysis of neutron problems, skip to ~18:00.

I also encourage to check our the video around 30:00 on the way out of openvswitch for neutron (tc/eBPF). As crazy as encouraging.

===

While at it, I encourage everyone interested in controlling switches the open source way to check out presentation of the new kernel subsystem for that (I guess vaporware atm):

- - https://www.youtube.com/watch?v=awekaJ7xWuw

===

So, that's what I have for you now. I really want to hear from everyone what is our plan to solve the problem neutron side.

Comments?

/Ihar

**Jakub Libosvar and me
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJU3eq+AAoJEC5aWaUY1u57uyIH/2MRnU7Xr2ivfzDsqg1T1djN
WgE6j87hVyIUnw/p/+vD1eDpOURPmZUcE/S7B6SCVv5KNB+j0pr22os5JM0cjCox
zt63xz4GR/LibiJhyPnWtmSOqYdGFeTIdOj2TvovvOqtmI4MRmHoZy4fwShq0jXd
RX00Z/o2DCxB+0KfJYQiWbFgXO43/zrdNGe9ME3XWI5TvVXQx49DMwv5K1jYN45Q
oHsyqIGovi1bpYDFCaqe+CPuRh5xO8SVrOHa+JHURgW8N0JHzDSN31zLT45roMp4
AqmBkZgjAG6WvsJWwYcQdoBEUdy5l0ml/86qysC5yFVdntHe2pfzMkpeZyLFNho=
=4OWY
-----END PGP SIGNATURE-----

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev