<html><body><p>I will be there for my lightning talk, and I think armax and kevin_benton will be there - it would be good to find some time for us to pow-wow, along with some teleconference so that carl_baldwin and mestery can join in...<br><br>Ryan Moats (regXboi)<br><br><tt>Mike Dorman <mdorman@godaddy.com> wrote on 08/03/2015 10:07:23 PM:<br><br>> From: Mike Dorman <mdorman@godaddy.com></tt><br><tt>> To: "OpenStack Development Mailing List (not for usage questions)" <br>> <openstack-dev@lists.openstack.org>, OpenStack Operators <openstack-<br>> operators@lists.openstack.org></tt><br><tt>> Date: 08/03/2015 10:08 PM</tt><br><tt>> Subject: Re: [Openstack-operators] [openstack-dev] [Neutron][L3] <br>> [Large Deployments Team] Representing a networks connected by routers</tt><br><tt>> <br>> I hope we can move this idea moving forward. I was disappointed to see <br>> the spec abandoned.<br>> <br>> Some of us from the large deployers group will be at the Ops Meetup. Will <br>> there be any representation from Neutron there that we could discuss with <br>> more?<br>> <br>> Thanks,<br>> Mike<br>> <br>> <br>> <br>> <br>> <br>> On 8/3/15, 12:27 PM, "Carl Baldwin" <carl@ecbaldwin.net> wrote:<br>> <br>> >Kevin, sorry for the delay in response. Keeping up on this thread was<br>> >getting difficult while on vacation.<br>> ><br>> >tl;dr: I think it is worth it to talk through the idea of inserting<br>> >some sort of a "subnet group thing" in the model to which floating ips<br>> >(and router external gateways) will associate. It has been on my mind<br>> >for a long time now. I didn't pursue it because a few informal<br>> >attempts to discuss it with others indicated to me that it would be a<br>> >difficult heavy-lifting job that others may not appreciate or<br>> >understand. Scroll to the bottom of this message for a little more on<br>> >this.<br>> ><br>> >Carl<br>> ><br>> >On Tue, Jul 28, 2015 at 1:15 AM, Kevin Benton <blak111@gmail.com> wrote:<br>> >>>Also, in my proposal, it is more the router that is the grouping <br>> >>>mechanism.<br>> >><br>> >> I can't reconcile this with all of the points you make in the rest of <br>> >>your<br>> >> email. You want the collection of subnets that a network represents, <br>> >>but you<br>> >> don't want any other properties of the network.<br>> ><br>> >This is closer to what I'm trying to say but isn't quite there. There<br>> >are some subnets that should be associated with the segments<br>> >themselves and there are some that should be associated with the<br>> >collection of segments. I want floating IPs that are not tied the an<br>> >L2 network. None of the alternate proposals that I'd heard addressed<br>> >this.<br>> ><br>> >>>that the network object is currently the closest thing we have to<br>> >>> representing the L3 part of the network.<br>> >><br>> >> The L3 part of a network is the subnets. You can't have IP addresses <br>> >>without<br>> >> the subnets, you can't have floating IPs without the subnets, etc.<br>> ><br>> >You're right but in the current model you can't have IP addresses<br>> >without the network either which is actually the point I'm trying to<br>> >make.<br>> ><br>> >> A Neutron network is an L2 construct that encapsulates L3 things. By<br>> >> encapsulating them it also provides an implicit grouping. The routed<br>> >> networks proposal basically wants that implicit grouping without the<br>> >> encapsulation or the L2 part.<br>> ><br>> >This sounds about right. I think it is wrong to assume that we need<br>> >an L2 network to encapsulate L3 things. I'm feeling restricted by the<br>> >model and the insistence that a neutron network is a purely L2<br>> >construct.<br>> ><br>> >>>We don't associate floating ips with a network because we want to arp <br>> >>>for<br>> >>> them. You're taking a consequence of the current model and its <br>> >>>constraints<br>> >>> and presenting that as the motivation for the model. We do so because <br>> >>>there<br>> >>> is no better L3 object to associate it to.<br>> >><br>> >> Don't make assumptions about how people use floating IPs now just <br>> >>because it<br>> >> doesn't fit your use-case well. When an external network is implemented <br>> >>as a<br>> >> real Neutron network (leaving external_network_bridge blank like we <br>> >>suggest<br>> >> in the networking guide), normal ports can be attached and can<br>> >> co-exist/communicate with the floating IPs because it behaves as an L2<br>> >> network exactly as implied by the API. The current model works quite <br>> >>well if<br>> >> your fabric can extend the external network everywhere it needs to be.<br>> ><br>> >Yes, "when an external network is implemented as a real Neutron<br>> >network" all of this is true and my proposal doesn't change any of<br>> >this. I'm wasn't making any such assumptions. I acknowledge that the<br>> >current model works well in this case and didn't intend to change it<br>> >for current use cases. It is precisely that because it does not fit<br>> >my use case well that I'm pursuing this.<br>> ><br>> >Notice that a network marked only as external doesn't allow normal<br>> >tenants to create ports. It must also be marked shared to allow it.<br>> >Unless tenants are creating regular ports then they really don't care<br>> >if arp or anything else L2 is involved because such an external<br>> >network is meant to give access external to the cloud where L2 is<br>> >really just an implementation detail. It is the deployer that cares<br>> >because of whatever infrastructure (like a gateway router) needs to<br>> >work with it. If the L2 is important, then the deployer will not<br>> >attempt to use an L3 only network, she will use the same kinds of<br>> >networks as always.<br>> ><br>> >The bad assumption here is that floating IPs need an explicit<br>> >association with an L2 only construct: tenant's allocate a floating<br>> >IP by selecting the Neutron network it is recorded in the DB that way.<br>> >Tenant's aren't even allowed to see the subnets on an external<br>> >network. This is counter-intuitive to me because I believe that, in<br>> >most cases, tenants want a floating IP to get L3 access to the world<br>> >(or a part of it) that is external to Openstack. Yet, they can only<br>> >see the L2 object? These are the factors that make me view the<br>> >Neutron network as an L2 + L3 construct.<br>> ><br>> >> If you don't want floating IPs to be reachable on the network they are<br>> >> associated with, then let's stop associating them with a network and <br>> >>instead<br>> >> start associating them with a group of subnets from which they allocate <br>> >>IPs.<br>> ><br>> >Okay. I'm willing to take a serious look at this. This isn't merely<br>> >associating a floating IP with a subnet. The tenant shouldn't need to<br>> >know about the individual cidrs of the L3 network and wonder if there<br>> >is some significant difference or "flavor" of each. I think we truly<br>> >need something that represents the group of subnets as you said.<br>> ><br>> >>>If we insist on a new object for the L3 part of a network then I'd say <br>> >>>we<br>> >>> had better have an L3 only port to connect to it.<br>> >><br>> >> I don't think a new port type is necessary. We can just make the <br>> >>network ID<br>> >> nullable for a port to indicate that it isn't attached to a Neutron <br>> >>network<br>> >> since it won't be. It would then have a relationship to its associated<br>> >> subnet via fixed_ips as it does now.<br>> ><br>> >Is this really so different from what I'm trying to do with networks?<br>> >Make the L2 part nullable.<br>> ><br>> >>>The subnet is not the L3 object that we're looking for. You may wish it<br>> >>> were but that does not make it so.<br>> >><br>> >> I never said a subnet is what we are looking for. The group of subnets <br>> >>is<br>> >> what we seem to be after.<br>> ><br>> >Agreed as I stated above.<br>> ><br>> >>>Ignoring the forced dependence on L2, the subnets still don't stand <br>> >>>alone<br>> >>> to describe just the L3 part, they must be linked to a network to make <br>> >>>any<br>> >>> sense.<br>> >><br>> >> They don't need to be. If we made the network_id nullable on ports and<br>> >> subnets, we could still have ports associated with subnets. The network<br>> >> portion is the L2 portion. You don't want L2 so don't ask for the <br>> >>network.<br>> ><br>> >In your model, should the port be associated with the "group of<br>> >subnets" at all? I'm not sure I see a need for it to be directly<br>> >associated but I haven't thought it all the way through.<br>> ><br>> >> I understand that we want a way to reference collections of subnets and<br>> >> create ports that allocate IPs from them. Networks provide that now, but<br>> >> they imply an L2 domain for the ports, which we don't want. So we are <br>> >>trying<br>> >> to change what a network implies for this one special case, which is <br>> >>going<br>> >> to have ripple effects everywhere.<br>> >><br>> >> Here are some areas where I can already see we will need special-casing:<br>> >><br>> >> DHCP agent scheduling - broadcast doesn't work on L3 networks, every <br>> >>compute<br>> >> node will need to use the direct tap attachment logic Neil brought up.<br>> ><br>> >My proposal only created the ports on the network segments.<br>> >Admittedly, that is the ugliest part of the proposal but it did<br>> >obviate the need for DHCP or arp for the port.<br>> ><br>> >> DHCP lease generation - a port can't get the normal subnet mask for the <br>> >>L3<br>> >> network it's attached to because it would try to ARP for addresses in <br>> >>the<br>> >> same subnet, which are actually somewhere else.<br>> ><br>> >See above.<br>> ><br>> >> Router interface attachment - what happens when a user attaches one<br>> >> interface to a regular network and one to an L3 network? Before they <br>> >>were<br>> >> all L2 networks so it didn't matter, but now none of the ports will be<br>> >> reachable on the L3 network without route table manipulation.<br>> >> (or) Router creation - to avoid the above you can have different router<br>> >> types that can only attach to one or the other.<br>> ><br>> >Not sure I'm following here. The ports are all created on the segments.<br>> ><br>> >> Every L2 attribute related to networks - we will need logic in the API <br>> >>that<br>> >> hides these fields or marks them as invalid and to generate an error if <br>> >>the<br>> >> user tries to update them.<br>> >> Multi-provider segments - We can't let a user add an L3 segment to a <br>> >>network<br>> >> with L2 segments (e.g. VXLAN, VLAN). Same goes for the inverse.<br>> >> Hierarchical port binding - coordinating ToRs for VXLAN+VLAN is l2 <br>> >>encap. L3<br>> >> would need route propagation logic instead.<br>> ><br>> >All the ports are still connected to L2 network segments so I don't<br>> >think this is an issue.<br>> ><br>> >> Every plugin, service, tool, etc, built on the assumption that a Neutron<br>> >> network is L2.<br>> ><br>> >ok<br>> ><br>> >> Port creation - If you aren't doing the full l3 like Neil's proposal, <br>> >>you<br>> >> need to intercept port creation requests and schedule the port to one <br>> >>of the<br>> >> underlying regular networks. The port then has a different network_id <br>> >>than<br>> >> the one requested, or we have more special-casing to hide that.<br>> ><br>> >ok, this was called out and I admit it is the ugliest part of the <br>> >proposal.<br>> ><br>> >> All of those will be branches in the codebase to handle current Neutron<br>> >> networks vs L3 networks. If we go down this route, it will be even worse<br>> >> than the conditionals we have to support DVR in ML2 because we are <br>> >>exposing<br>> >> it via the API. It's technical debt that we will not be able to get rid <br>> >>of.<br>> ><br>> >I don't think it is nearly as bad as you make it out to be.<br>> ><br>> >> I would rather see something to reference a group of subnets that can be<br>> >> used for floating IP allocation and port creation in lieu of a network <br>> >>ID<br>> >> than the technical debt that conditionally redefining a network will <br>> >>bring.<br>> ><br>> >I'm willing to discuss this further. In fact, it has been on my mind<br>> >for a while now. This is essentially where I started. I ended up<br>> >with my current proposal because I perceived a lot more difficulty in<br>> >doing this than in the proposal I created. But, your perspective from<br>> >the other side of the problem is worth considering.<br>> ><br>> >I'm glad to see that at least one other person seems to be<br>> >understanding the problem here. This will have API and end-user<br>> >impact is it changes the way that end users interact with floating IPs<br>> >at least. It will also affect the way that neutron routers are<br>> >associated to a network. Today's use case where a user connects a<br>> >router to an external network will also change. To what extent do we<br>> >support backward compatibility for existing work-flows? For example,<br>> >can a user port an existing work-flow to a cloud where the "external<br>> >network" is now a group of subnets routed among segments instead of a<br>> >neutron network?<br>> ><br>> >Carl<br>> ><br>> >__________________________________________________________________________<br>> >OpenStack Development Mailing List (not for usage questions)<br>> >Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>> ><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>> _______________________________________________<br>> OpenStack-operators mailing list<br>> OpenStack-operators@lists.openstack.org<br>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>> <br></tt><BR>
</body></html>