Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi neutroners,

we** had several conversations recently with our Red Hat fellows who
work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way
neutron utilizes their software. Those were beneficial to both sides
to understand what we do right and wrong. I was asked to share some of
the points from those discussions with broader community.

===

One of the issues that came up during discussions is the way neutron
connects ovs ports to namespaces. The short story is that openvswitch
is not designed with namespaces in mind, and the fact that moving its
ports into a different namespace works for neutron is mere
coincidence, and is actually considered as a bug by openvswitch guys.

It's not just broken in theory from software design standpoint, but
also in practice. Specifically,

1. ovsdb dump does not work properly for namespaces:
- - https://bugzilla.redhat.com/show_bug.cgi?id=1160340

This results in wrong statistics and other data collected for these ports;

2. We suspect that the following kernel crash is triggered because of
our usage of the feature that is actually a bug:
- - https://bugs.launchpad.net/neutron/+bug/1418097

Quoting Jiri Benc,

"The problem is openvswitch does not support its ports to be moved to
a different name space. The fact that it's possible to do that is a
bug - such operation should have been denied. Unfortunately, due to a
missing check, it's not been denied. Such setup does not work
reliably, though, and leads to various issues from incorrect resource
accounting to kernel crashes.

We're aware of the bug but we don't have any easy way to fix it. The
most obvious option, disabling moving of ovs ports to different name
spaces, would be easy to do but it would break Neutron. The other
option, making ovs to work across net name spaces, is hard and will
require addition of different kernel APIs and large changes in ovs
user space daemons. This constitutes tremendous amount of work."

The tracker bug on openvswitch side is:
- - https://bugzilla.redhat.com/show_bug.cgi?id=1160340

So in the best case, we may expect openvswitch to properly support the
feature in long term, but short term it won't work, especially while
neutron expects other features implemented in openvswitch for it (like
NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few).

We could try to handle the issue neutron side. We can fix it by
utilizing veth pairs to get into namespaces, but it may mean worse
performance, and will definitely require proper benchmarking to see
whether we can live with performance drop.

===

There were other suggestions on how we can enhance our way of usage of
openvswitch. Among those, getting rid of linux bridge used for
security groups, with special attention on getting rid of ebtables
(sic!) for they are a lot slower than iptables; getting rid of veth
pair for instance ports.

===

I really encourage everyone to check the following video from
devconf.cz 2015 on all that and more at:

- - https://www.youtube.com/watch?v=BlLD-qh9EBQ

Among other things, you will find presentation of plotnetcfg tool to
create nice graphs of openstack state.

If you're lazy enough and want to switch directly to the analysis of
neutron problems, skip to ~18:00.

I also encourage to check our the video around 30:00 on the way out of
openvswitch for neutron (tc/eBPF). As crazy as encouraging.

===

While at it, I encourage everyone interested in controlling switches
the open source way to check out presentation of the new kernel
subsystem for that (I guess vaporware atm):

- - https://www.youtube.com/watch?v=awekaJ7xWuw

===

So, that's what I have for you now. I really want to hear from
everyone what is our plan to solve the problem neutron side.

Comments?

/Ihar

**Jakub Libosvar and me
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJU3eq+AAoJEC5aWaUY1u57uyIH/2MRnU7Xr2ivfzDsqg1T1djN
WgE6j87hVyIUnw/p/+vD1eDpOURPmZUcE/S7B6SCVv5KNB+j0pr22os5JM0cjCox
zt63xz4GR/LibiJhyPnWtmSOqYdGFeTIdOj2TvovvOqtmI4MRmHoZy4fwShq0jXd
RX00Z/o2DCxB+0KfJYQiWbFgXO43/zrdNGe9ME3XWI5TvVXQx49DMwv5K1jYN45Q
oHsyqIGovi1bpYDFCaqe+CPuRh5xO8SVrOHa+JHURgW8N0JHzDSN31zLT45roMp4
AqmBkZgjAG6WvsJWwYcQdoBEUdy5l0ml/86qysC5yFVdntHe2pfzMkpeZyLFNho=
=4OWY
-----END PGP SIGNATURE-----