<div style="font-family: Helvetica; font-size: 14px;">Hi, Ihar & Jiri, thank you for pointing this out.<div><br></div><div>I’m working on the following items:</div><div><br></div><div>1) Doing Openflow traffic filtering (stateful firewall) based on OVS+CT[1] patch, which may</div><div> eventually merge. Here I want to build a good amount of benchmarks to be able to compare</div><div> the current network iptables+LB solution to just openflow.</div><div><br></div><div> Openflow filtering should be fast, as it’s quite smart at using hashes to match OF rules</div><div> in the kernel megaflows (thanks Jiri & T.Graf for explaining me this)</div><div> </div><div> The only bad part is that we would have to dynamically change more rules based on security</div><div>group changes (now we do that via ip sets without reloading all the rules).</div><div><br></div><div> To do this properly, we may want to make the OVS plugin a real OF controller to be able to</div><div>push OF rules to the bridge without the need of calling ovs-ofctl on the command line all the time.</div><div><br></div><div>2) Using OVS+OF to do QoS</div><div><br></div><div>other interesting stuff to look at:</div><div><br></div><div>3) Doing routing in OF too, thanks to the NAT capabilities of having OVS+CT </div><div><br></div><div>4) The namespace problem, what kinds of statistics get broken by moving ports into namespaces now?,</div><div> the short-term fix could be using vets, but “namespaceable” OVS ports would be perfect, yet I understand</div><div> the change is a big feature.</div><div><br></div><div> If we had 1 & 3, may be 4 wouldn’t be a problem anymore.</div><div><br></div><div>[1] <a href="https://github.com/justinpettit/ovs/tree/conntrack">https://github.com/justinpettit/ovs/tree/conntrack</a></div></div>
<div><div><br></div><div><span style="font-size: 10pt;">Miguel Ángel Ajo</span></div><div><br></div></div>
<p style="color: #A0A0A8;">On Friday, 13 de February de 2015 at 13:14, Ihar Hrachyshka wrote:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>-----BEGIN PGP SIGNED MESSAGE-----</div><div>Hash: SHA1</div><div><br></div><div>Hi neutroners,</div><div><br></div><div>we** had several conversations recently with our Red Hat fellows who</div><div>work on openvswitch (Jiri Benc and Jiri Pirko) regarding the way</div><div>neutron utilizes their software. Those were beneficial to both sides</div><div>to understand what we do right and wrong. I was asked to share some of</div><div>the points from those discussions with broader community.</div><div><br></div><div>===</div><div><br></div><div>One of the issues that came up during discussions is the way neutron</div><div>connects ovs ports to namespaces. The short story is that openvswitch</div><div>is not designed with namespaces in mind, and the fact that moving its</div><div>ports into a different namespace works for neutron is mere</div><div>coincidence, and is actually considered as a bug by openvswitch guys.</div><div><br></div><div>It's not just broken in theory from software design standpoint, but</div><div>also in practice. Specifically,</div><div><br></div><div>1. ovsdb dump does not work properly for namespaces:</div><div>- - <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1160340">https://bugzilla.redhat.com/show_bug.cgi?id=1160340</a></div><div><br></div><div>This results in wrong statistics and other data collected for these ports;</div><div><br></div><div>2. We suspect that the following kernel crash is triggered because of</div><div>our usage of the feature that is actually a bug:</div><div>- - <a href="https://bugs.launchpad.net/neutron/+bug/1418097">https://bugs.launchpad.net/neutron/+bug/1418097</a></div><div><br></div><div>Quoting Jiri Benc,</div><div><br></div><div>"The problem is openvswitch does not support its ports to be moved to</div><div>a different name space. The fact that it's possible to do that is a</div><div>bug - such operation should have been denied. Unfortunately, due to a</div><div>missing check, it's not been denied. Such setup does not work</div><div>reliably, though, and leads to various issues from incorrect resource</div><div>accounting to kernel crashes.</div><div><br></div><div>We're aware of the bug but we don't have any easy way to fix it. The</div><div>most obvious option, disabling moving of ovs ports to different name</div><div>spaces, would be easy to do but it would break Neutron. The other</div><div>option, making ovs to work across net name spaces, is hard and will</div><div>require addition of different kernel APIs and large changes in ovs</div><div>user space daemons. This constitutes tremendous amount of work."</div><div><br></div><div>The tracker bug on openvswitch side is:</div><div>- - <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1160340">https://bugzilla.redhat.com/show_bug.cgi?id=1160340</a></div><div><br></div><div>So in the best case, we may expect openvswitch to properly support the</div><div>feature in long term, but short term it won't work, especially while</div><div>neutron expects other features implemented in openvswitch for it (like</div><div>NAT, or connection tracking, or ipv6 tunnel endpoints, to name a few).</div><div><br></div><div>We could try to handle the issue neutron side. We can fix it by</div><div>utilizing veth pairs to get into namespaces, but it may mean worse</div><div>performance, and will definitely require proper benchmarking to see</div><div>whether we can live with performance drop.</div><div><br></div><div>===</div><div><br></div><div>There were other suggestions on how we can enhance our way of usage of</div><div>openvswitch. Among those, getting rid of linux bridge used for</div><div>security groups, with special attention on getting rid of ebtables</div><div>(sic!) for they are a lot slower than iptables; getting rid of veth</div><div>pair for instance ports.</div><div><br></div><div>===</div><div><br></div><div>I really encourage everyone to check the following video from</div><div><a href="http://devconf.cz">devconf.cz</a> 2015 on all that and more at:</div><div><br></div><div>- - <a href="https://www.youtube.com/watch?v=BlLD-qh9EBQ">https://www.youtube.com/watch?v=BlLD-qh9EBQ</a></div><div><br></div><div>Among other things, you will find presentation of plotnetcfg tool to</div><div>create nice graphs of openstack state.</div><div><br></div><div>If you're lazy enough and want to switch directly to the analysis of</div><div>neutron problems, skip to ~18:00.</div><div><br></div><div>I also encourage to check our the video around 30:00 on the way out of</div><div>openvswitch for neutron (tc/eBPF). As crazy as encouraging.</div><div><br></div><div>===</div><div><br></div><div>While at it, I encourage everyone interested in controlling switches</div><div>the open source way to check out presentation of the new kernel</div><div>subsystem for that (I guess vaporware atm):</div><div><br></div><div>- - <a href="https://www.youtube.com/watch?v=awekaJ7xWuw">https://www.youtube.com/watch?v=awekaJ7xWuw</a></div><div><br></div><div>===</div><div><br></div><div>So, that's what I have for you now. I really want to hear from</div><div>everyone what is our plan to solve the problem neutron side.</div><div><br></div><div>Comments?</div><div><br></div><div>/Ihar</div><div><br></div><div>**Jakub Libosvar and me</div><div>-----BEGIN PGP SIGNATURE-----</div><div>Version: GnuPG v1</div><div><br></div><div>iQEcBAEBAgAGBQJU3eq+AAoJEC5aWaUY1u57uyIH/2MRnU7Xr2ivfzDsqg1T1djN</div><div>WgE6j87hVyIUnw/p/+vD1eDpOURPmZUcE/S7B6SCVv5KNB+j0pr22os5JM0cjCox</div><div>zt63xz4GR/LibiJhyPnWtmSOqYdGFeTIdOj2TvovvOqtmI4MRmHoZy4fwShq0jXd</div><div>RX00Z/o2DCxB+0KfJYQiWbFgXO43/zrdNGe9ME3XWI5TvVXQx49DMwv5K1jYN45Q</div><div>oHsyqIGovi1bpYDFCaqe+CPuRh5xO8SVrOHa+JHURgW8N0JHzDSN31zLT45roMp4</div><div>AqmBkZgjAG6WvsJWwYcQdoBEUdy5l0ml/86qysC5yFVdntHe2pfzMkpeZyLFNho=</div><div>=4OWY</div><div>-----END PGP SIGNATURE-----</div><div><br></div><div>__________________________________________________________________________</div><div>OpenStack Development Mailing List (not for usage questions)</div><div>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>