<div style="font-family: Helvetica; font-size: 15px;"><span style="color: rgb(160, 160, 168); font-size: 13px;">On Monday, 18 de May de 2015 at 2:28, Kris G. Lindgren wrote:</span></div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>See inline.</div><div>____________________________________________</div><div> </div><div>Kris Lindgren</div><div>Senior Linux Systems Engineer</div><div>GoDaddy, LLC.</div><div><br></div><div><br></div><div><br></div><div><br></div><div>On 5/18/15, 1:40 AM, "Kevin Benton" <<a href="mailto:blak111@gmail.com">blak111@gmail.com</a>> wrote:</div><div><br></div><blockquote type="cite"><div><div>This whole discussion is basically pointless because the ebtables work</div><div>isn't even done. There is no merged ebtables fix to back-port.</div></div></blockquote><div><br></div><div>Fair.</div><blockquote type="cite"><div><div><br></div><blockquote type="cite"><div>Additionally, some of us don¹t want to run OVS</div></blockquote><div><br></div><div>Well OVS is the reference right now. If you choose to use something</div><div>else, there are going to be feature gaps like this. Did you consider</div><div>this before trying to remove OVS?</div><div><br></div><blockquote type="cite"><div>so an OVS only solution is effectively imho - crap</div></blockquote><div><br></div><div>You mean for people that choose not to use the gate-tested reference</div><div>driver?</div><div><br></div><div>Anyone can just as easily argue that an ebtables solution is crap</div><div>because it assumes a filtering bridge so it doesn't work with any</div><div>direct plugging setups. However, we try to discourage attacking</div><div>contributions that didn't happen to fit a narrow use-case. It's</div><div>unproductive and poisonous behavior that discourages people from doing</div><div>anything.</div></div></blockquote><div><br></div><div>You mean how security groups was/is currently and has always been</div><div>implemented in neutron + ovs. Creating linux bridge per vm so you can do</div><div>filtering via iptables over the bridge? To be fair, the OVS solution does</div><div>solve the problem for some people - so you are right I shouldn't have</div><div>called it crap. I guess it's when that solution is called out as an</div><div>acceptable work around that the more complete/robust solution wont be back</div><div>ported because it add's new dependancies, I guess that to me is what is</div><div>crap. BTW, to be clear that since we package our own packages - we will</div><div>pull this in and being running it as soon as it lands. But that doesn't</div><div>do anyone else any good.</div><blockquote type="cite"><div><div><br></div><blockquote type="cite"><div><div>you need to run OVS - seems like a stretch if you aren't using GRE/vxlan</div><div>tunneling for all of your guests.</div></div></blockquote><div><br></div><div>Tunneling has little to do with OVS. Linux bridge also supports vxlan</div><div>and GRE. You can just say why you don't like OVS (e.g. hard to debug,</div><div>tooling is different, admins don't know how to use it, etc). This can</div><div>help motivate the movement to restart development on the linux bridge</div><div>driver.</div></div></blockquote><div>The references configuration all say or at minimum imply that if you are</div><div>doing gre/overlay networks you need to be using OVS. I don¹t really want</div><div>to get into an ovs bashing/discussion here, but in general since neutron</div><div>creates a linux bridge to apply security group rules to for vms. We</div><div>effectively make an architecture that uses both OVS and linux bridge for</div><div>every vm. If I also have an architecture that happens to use vlans and</div><div>nothing else - what does ovs gain me? Complexity - yep, additonal</div><div>overhead, yep. Since a packet gets ran through kernel space for ovs +</div><div>possibly a trip to user space opensvswitchd + linux bridging code + bridge</div><div>mac learning code + iptables + tap code. Now tell me in that chain whats</div><div>the one thing that doesn¹t need to be there? I hope we both said</div><div>Openvswitch. Last I heard the megaflows that allowed for the huge speed</div><div>ups no longer work when you start doing specific L4 matches (IE</div><div>ip/port/porto filtering).</div></div></div></span></blockquote><div><br></div><div><span style="font-size: 15px;">That’s wrong since OVS 2.3, check conversation here with Ben Pfaff</span></div><div><br></div><div><a href="https://review.openstack.org/#/c/159840/1/doc/source/testing/openflow-firewall.rst">https://review.openstack.org/#/c/159840/1/doc/source/testing/openflow-firewall.rst</a> (line 26) </div><div> </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><blockquote type="cite"><div><div></div><blockquote type="cite"><div>Since, flat networks are a 100% supported</div></blockquote><div><br></div><div>Who is supporting Linux bridge used as a backend for shared networks</div><div>and is claiming they are secure? Whoever is doing that should be on</div><div>the receiving end of your complaints.</div></div></blockquote><div><br></div><div>We are using neutron + ovs + shared networks (on real vlans). But neutron</div><div>+ ml2 + linux bridge mechanism driver is totally a supported thing ;-).</div><div>We also aren't using any of the router stuff in neutron either. Because</div><div>neutron is suppose to apply "anti-spoofing rules" to prevent vm's from</div><div>doing bad things to other vm's. So I think the complaints are</div><div>appropriately justified, since the original commit that added security</div><div>groups to quantum had a method named: "_arp_spoofing_rule" [1] that the</div><div>original intent was to have neutron preventing arp spoofing. It just</div><div>never worked for the 3+ years, because as it turns out you can't filter</div><div>arp via iptables - since iptables is layer3-7 and arp is layer2. It was</div><div>also further hidden during the addition of allowed address-pairs [2],</div><div>which effectively reverse implemented the previous logic [3]</div><div><br></div><div>[1] - </div><div><a href="https://github.com/openstack/neutron/commit/f14af5dc755706c7297a96fa504acdf">https://github.com/openstack/neutron/commit/f14af5dc755706c7297a96fa504acdf</a></div><div>e15ac1957#diff-65b266f9e013df37c4934f0b1007897cR168</div><div>[2] - </div><div><a href="https://github.com/openstack/neutron/commit/b67b20832a5bfccd1bbf8d1e63ebcd7">https://github.com/openstack/neutron/commit/b67b20832a5bfccd1bbf8d1e63ebcd7</a></div><div>061856881</div><div>[3] - </div><div><a href="https://github.com/openstack/neutron/commit/0efce6195fa7be80e110bd841dc9b35">https://github.com/openstack/neutron/commit/0efce6195fa7be80e110bd841dc9b35</a></div><div>37a94c376#diff-abf220de4c2165d9e5bfd6dde12b3f4fR205</div><div><br></div><div>So, to be clear. I want the bug fix - dare I say security/vunerability</div><div>fix - when it lands to be back-ported to the current supported stable</div><div>release. This is to close a 3+ year old security issue in that neutron</div><div>tries to provide anti-spoofing rules - but fails to implement them in a</div><div>way that actual work. Yet at a minimum alludes to the fact that it does</div><div>stuff like nova to prevent vm's from doing arp spoofing/poisoning.</div><div>Additionally, I want done in a way that is as generic as possible, so it</div><div>works with the most amount of supported neutron plugins. (Ala linux</div><div>bridge+iptables+ now ebtables).</div><blockquote type="cite"><div><div><br></div><div>Please be clear about what you really want here. It sounds like you</div><div>want ARP filtering support in the Linux bridge driver. Is that</div><div>correct?</div><div><br></div><div>On Mon, May 18, 2015 at 12:22 AM, Kris G. Lindgren</div><div><<a href="mailto:klindgren@godaddy.com">klindgren@godaddy.com</a>> wrote:</div><blockquote type="cite"><div><div>I always thought that ebtables was below the stack in the iptables</div><div>schema -</div><div>but still part of netfilter - thus should be reasonably fast (I would</div><div>argue</div><div>faster than a user space lookup to openvswitchd). Considering the rules</div><div>being added are small in number and trivial (on this port allow src/dst</div><div>mac</div><div>of blah). Do you have any performance data showing that its slow? A</div><div>quick</div><div>google search shows nothing relevant :-). Additionally, libvirt</div><div>anti-spoofing rules configured via nova using nwfilter uses ebtables to</div><div>do</div><div>the anti spoofing rules. No one seems to complain about the</div><div>performance of</div><div>that...</div><div><br></div><div>Additionally, some of us don¹t want to run OVS, so an OVS only solution</div><div>is</div><div>effectively imho - crap. We are actively looking at removing OVS from</div><div>our</div><div>environment due to a number of reasons. So saying if you run neutron</div><div>and</div><div>care to have *REAL* network protection - you need to run OVS - seems</div><div>like a</div><div>stretch if you aren't using GRE/vxlan tunneling for all of your guests.</div><div><br></div><div>I personally agree with George, this stance of "We never said that</div><div>neutron</div><div>would provide anti-spoofing on flat networks thus we wont backport this,</div><div>because it now brings into scope ebtables", seems to be a pretty huge</div><div>gap in</div><div>what neutron says it does and the protection it actually provides. We</div><div>supply security group rules, but stealing someone else's IP or the</div><div>gateway</div><div>that doesn't belong to you - yep that totally cool with us. It strikes</div><div>me</div><div>that neutrons goal to deliver networking is basically two fold. 1.)</div><div>Provide</div><div>multi-tenant networking 2.) Make sure tenants cant stomp on each other.</div><div>Without number 2 (anti-spoofing) you kinda fail to provide number 1.</div><div>Since,</div><div>flat networks are a 100% supported and viable way of doing tenant</div><div>networking</div><div>I would say that this is a bug and should be backported to kilo/juno.</div><div><br></div><div>I don¹t personly buy the new dependency reason, new to neutron - maybe</div><div>- but</div><div>not new to people running nova/libvirt. Ebtables is used by libvirt for</div><div>nwfilter, assuming an pretty standard libvirt install ebtables should be</div><div>installed by default. Additionally, this would have already been a</div><div>dependency in nova-compute due to anti-spoofing support there.</div><div>____________________________________________</div><div><br></div><div>Kris Lindgren</div><div>Senior Linux Systems Engineer</div><div>GoDaddy, LLC.</div><div><br></div><div><br></div><div>From: Miguel Ángel Ajo <<a href="mailto:majopela@redhat.com">majopela@redhat.com</a>></div><div>Date: Sunday, May 17, 2015 at 6:33 AM</div><div>To: George Shuklin <<a href="mailto:george.shuklin@gmail.com">george.shuklin@gmail.com</a>></div><div>Cc: "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>"</div><div><<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>></div><div>Subject: Re: [Openstack-operators] Raising the degree of the scandal</div><div><br></div><div>Probably the solution is not selected to be backported because:</div><div><br></div><div> * It¹s an intrusive change</div><div> * Introduces new dependencies</div><div> * Probably it¹s going to introduce a performance penalty because</div><div>eatables</div><div>is slow.</div><div><br></div><div>I¹m asking in reviews for this feature to be enabled/disabled via a</div><div>flag.</div><div><br></div><div>In the future I hope OVS with connection tracking to be merged, so then</div><div>we</div><div>can</div><div>finally have a proper openvswitch_firewall_driver supporting stateful</div><div>firewalling</div><div>without reflective rules or flag matching (one is slow, the other is</div><div>insecureŠ)</div><div><br></div><div>Miguel Ángel Ajo</div><div><br></div><div>On Saturday, 16 de May de 2015 at 23:28, George Shuklin wrote:</div><div><br></div><div>On 05/15/2015 07:48 PM, Jay Pipes wrote:</div><div><br></div><div>On 05/15/2015 12:38 PM, George Shuklin wrote:</div><div><br></div><div>Just to let everyone know: broken antispoofing is not an 'security</div><div>issue' and the fix is not planned to be backported to Juno/kilo.</div><div><br></div><div><a href="https://bugs.launchpad.net/bugs/1274034">https://bugs.launchpad.net/bugs/1274034</a></div><div><br></div><div>What can I say? All hail devstack! Who care about production?</div><div><br></div><div><br></div><div>George, I can understand you are frustrated with this issue and feel</div><div>strongly about it. However, I don't think notes like this are all that</div><div>productive.</div><div><br></div><div>Would a more productive action be to tell the operator community a bit</div><div>about the vulnerability and suggest appropriate remedies to take?</div><div><br></div><div>Ok, sorry.</div><div><br></div><div>Short issue: If few tenants use same network (shared network) one tenant</div><div>may disrupt network activities of other tenant by sending a specially</div><div>crafted ARP packets on behave of the victim. Normally, Openstack</div><div>prohibit usage of unauthorized addresses (this feature is called</div><div>'antispoofing' and it is essential for multi-tenant clouds). This</div><div>feature were subtly broken (malicious tenant may not use other addresses</div><div>but still may disrupt activities of other tenants).</div><div><br></div><div>Finally, that bug has been fixed. But now they says 'oh, it is not that</div><div>important, we will not backport it to current releases, only to</div><div>"Libery"' because of new etables dependency.</div><div><br></div><div><br></div><div>_______________________________________________</div><div>OpenStack-operators mailing list</div><div><a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></div><div><br></div><div><br></div><div><br></div><div>_______________________________________________</div><div>OpenStack-operators mailing list</div><div><a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></div></div></blockquote><div><br></div><div><br></div><div><br></div><div>-- </div><div>Kevin Benton</div></div></blockquote><div><br></div><div><br></div><div>_______________________________________________</div><div>OpenStack-operators mailing list</div><div><a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>