Open Stack

See inline.
____________________________________________
 
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.




On 5/18/15, 1:40 AM, "Kevin Benton" <blak111 at gmail.com> wrote:

>This whole discussion is basically pointless because the ebtables work
>isn't even done. There is no merged ebtables fix to back-port.

Fair.
>
>>Additionally, some of us don¹t want to run OVS
>
>Well OVS is the reference right now. If you choose to use something
>else, there are going to be feature gaps like this. Did you consider
>this before trying to remove OVS?
>
>>so an OVS only solution is effectively imho - crap
>
>You mean for people that choose not to use the gate-tested reference
>driver?
>
>Anyone can just as easily argue that an ebtables solution is crap
>because it assumes a filtering bridge so it doesn't work with any
>direct plugging setups. However, we try to discourage attacking
>contributions that didn't happen to fit a narrow use-case. It's
>unproductive and poisonous behavior that discourages people from doing
>anything.

You mean how security groups was/is currently and has always been
implemented in neutron + ovs.  Creating linux bridge per vm so you can do
filtering via iptables over the bridge?  To be fair, the OVS solution does
solve the problem for some people - so you are right I shouldn't have
called it crap.  I guess it's when that solution is called out as an
acceptable work around that the more complete/robust solution wont be back
ported because it add's new dependancies, I guess that to me is what is
crap.  BTW, to be clear that since we package our own packages - we will
pull this in and being running it as soon as it lands.  But that doesn't
do anyone else any good.
>
>>you need to run OVS - seems like a stretch if you aren't using GRE/vxlan
>>tunneling for all of your guests.
>
>Tunneling has little to do with OVS. Linux bridge also supports vxlan
>and GRE. You can just say why you don't like OVS (e.g. hard to debug,
>tooling is different, admins don't know how to use it, etc). This can
>help motivate the movement to restart development on the linux bridge
>driver.
The references configuration all say or at minimum imply that if you are
doing gre/overlay networks you need to be using OVS.  I don¹t really want
to get into an ovs bashing/discussion here, but in general since neutron
creates a linux bridge to apply security group rules to for vms.  We
effectively make an architecture that uses both OVS and linux bridge for
every vm.  If I also have an architecture that happens to use vlans and
nothing else - what does ovs gain me?  Complexity - yep, additonal
overhead, yep.  Since a packet gets ran through kernel space for ovs +
possibly a trip to user space opensvswitchd + linux bridging code + bridge
mac learning code + iptables + tap code.  Now tell me in that chain whats
the one thing that doesn¹t need to be there?  I hope we both said
Openvswitch.  Last I heard the megaflows that allowed for the huge speed
ups no longer work when you start doing specific L4 matches (IE
ip/port/porto filtering).
>
>>Since, flat networks are a 100% supported
>
>Who is supporting Linux bridge used as a backend for shared networks
>and is claiming they are secure? Whoever is doing that should be on
>the receiving end of your complaints.

We are using neutron + ovs + shared networks (on real vlans).  But neutron
+ ml2 + linux bridge mechanism driver is totally a supported thing ;-).
We also aren't using any of the router stuff in neutron either.  Because
neutron is suppose to apply "anti-spoofing rules" to prevent vm's from
doing bad things to other vm's.  So I think the complaints are
appropriately justified, since the original commit that added security
groups to quantum had a method named: "_arp_spoofing_rule" [1] that the
original intent was to have neutron preventing arp spoofing.  It just
never worked for the 3+ years, because as it turns out you can't filter
arp via iptables - since iptables is layer3-7 and arp is layer2.  It was
also further hidden during the addition of allowed address-pairs [2],
which effectively reverse implemented the previous logic [3]

[1] - 
https://github.com/openstack/neutron/commit/f14af5dc755706c7297a96fa504acdf
e15ac1957#diff-65b266f9e013df37c4934f0b1007897cR168
[2] - 
https://github.com/openstack/neutron/commit/b67b20832a5bfccd1bbf8d1e63ebcd7
061856881
[3] - 
https://github.com/openstack/neutron/commit/0efce6195fa7be80e110bd841dc9b35
37a94c376#diff-abf220de4c2165d9e5bfd6dde12b3f4fR205

So, to be clear.  I want the bug fix - dare I say security/vunerability
fix - when it lands to be back-ported to the current supported stable
release.  This is to close a 3+ year old security issue in that neutron
tries to provide anti-spoofing rules - but fails to implement them in a
way that actual work.  Yet at a minimum alludes to the fact that it does
stuff like nova to prevent vm's from doing arp spoofing/poisoning.
Additionally, I want done in a way that is as generic as possible, so it
works with the most amount of supported neutron plugins.  (Ala linux
bridge+iptables+ now ebtables).
>
>Please be clear about what you really want here. It sounds like you
>want ARP filtering support in the Linux bridge driver. Is that
>correct?
>
>On Mon, May 18, 2015 at 12:22 AM, Kris G. Lindgren
><klindgren at godaddy.com> wrote:
>> I always thought that ebtables was below the stack in the iptables
>>schema -
>> but still part of netfilter - thus should be reasonably fast (I would
>>argue
>> faster than a user space lookup to openvswitchd).  Considering the rules
>> being added are small in number and trivial (on this port allow src/dst
>>mac
>> of blah).  Do you have any performance data showing that its slow?  A
>>quick
>> google search shows nothing relevant :-).  Additionally, libvirt
>> anti-spoofing rules configured via nova using nwfilter uses ebtables to
>>do
>> the anti spoofing rules.  No one seems to complain about the
>>performance of
>> that...
>>
>> Additionally, some of us don¹t want to run OVS, so an OVS only solution
>>is
>> effectively imho - crap.  We are actively looking at removing OVS from
>>our
>> environment due to a number of reasons.  So saying if you run neutron
>>and
>> care to have *REAL* network protection - you need to run OVS - seems
>>like a
>> stretch if you aren't using GRE/vxlan tunneling for all of your guests.
>>
>> I personally agree with George, this stance of "We never said that
>>neutron
>> would provide anti-spoofing on flat networks thus we wont backport this,
>> because it now brings into scope ebtables", seems to be a pretty huge
>>gap in
>> what neutron says it does and the protection it actually provides.  We
>> supply security group rules, but stealing someone else's IP or the
>>gateway
>> that doesn't belong to you - yep that totally cool with us.  It strikes
>>me
>> that neutrons goal to deliver networking is basically two fold. 1.)
>>Provide
>> multi-tenant networking 2.) Make sure tenants cant stomp on each other.
>> Without number 2 (anti-spoofing) you kinda fail to provide number 1.
>>Since,
>> flat networks are a 100% supported and viable way of doing tenant
>>networking
>> I would say that this is a bug and should be backported to kilo/juno.
>>
>> I don¹t personly buy the new dependency reason, new to neutron - maybe
>>- but
>> not new to people running nova/libvirt.  Ebtables is used by libvirt for
>> nwfilter, assuming an pretty standard libvirt install ebtables should be
>> installed by default.  Additionally, this would have already been a
>> dependency in nova-compute due to anti-spoofing support there.
>> ____________________________________________
>>
>> Kris Lindgren
>> Senior Linux Systems Engineer
>> GoDaddy, LLC.
>>
>>
>> From: Miguel Ángel Ajo <majopela at redhat.com>
>> Date: Sunday, May 17, 2015 at 6:33 AM
>> To: George Shuklin <george.shuklin at gmail.com>
>> Cc: "openstack-operators at lists.openstack.org"
>> <openstack-operators at lists.openstack.org>
>> Subject: Re: [Openstack-operators] Raising the degree of the scandal
>>
>> Probably the solution is not selected to be backported because:
>>
>>   * It¹s an intrusive change
>>   * Introduces new dependencies
>>   * Probably it¹s going to introduce a performance penalty because
>>eatables
>> is slow.
>>
>> I¹m asking in reviews for this feature to be enabled/disabled via a
>>flag.
>>
>> In the future I hope OVS with connection tracking to be merged, so then
>>we
>> can
>> finally have a proper openvswitch_firewall_driver supporting stateful
>> firewalling
>> without reflective rules or flag matching (one is slow, the other is
>> insecureŠ)
>>
>> Miguel Ángel Ajo
>>
>> On Saturday, 16 de May de 2015 at 23:28, George Shuklin wrote:
>>
>> On 05/15/2015 07:48 PM, Jay Pipes wrote:
>>
>> On 05/15/2015 12:38 PM, George Shuklin wrote:
>>
>> Just to let everyone know: broken antispoofing is not an 'security
>> issue' and the fix is not planned to be backported to Juno/kilo.
>>
>> https://bugs.launchpad.net/bugs/1274034
>>
>> What can I say? All hail devstack! Who care about production?
>>
>>
>> George, I can understand you are frustrated with this issue and feel
>> strongly about it. However, I don't think notes like this are all that
>> productive.
>>
>> Would a more productive action be to tell the operator community a bit
>> about the vulnerability and suggest appropriate remedies to take?
>>
>> Ok, sorry.
>>
>> Short issue: If few tenants use same network (shared network) one tenant
>> may disrupt network activities of other tenant by sending a specially
>> crafted ARP packets on behave of the victim. Normally, Openstack
>> prohibit usage of unauthorized addresses (this feature is called
>> 'antispoofing' and it is essential for multi-tenant clouds). This
>> feature were subtly broken (malicious tenant may not use other addresses
>> but still may disrupt activities of other tenants).
>>
>> Finally, that bug has been fixed. But now they says 'oh, it is not that
>> important, we will not backport it to current releases, only to
>> "Libery"' because of new etables dependency.
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
>
>-- 
>Kevin Benton