<div dir="ltr">><span style="font-size:12.8000001907349px">You're right but in the current model you can't have IP addresses </span><span style="font-size:12.8000001907349px">without the network either which is actually the point I'm trying to </span><span style="font-size:12.8000001907349px">make.</span><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">Definitely, but it's not because the network has properties that IPAM depends on. It's only because a subnet has a non-nullable foreign key relationship to a network. The network itself doesn't have anything IPAM depends on. </span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px">This sounds about right. I think it is wrong to assume that we need </span><span style="font-size:12.8000001907349px">an L2 network to encapsulate L3 things. I'm feeling restricted by the </span><span style="font-size:12.8000001907349px">model and the insistence that a neutron network is a purely L2 </span><span style="font-size:12.8000001907349px">construct.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I agree that it's wrong to assume we need L2 in the model for L3 stuff. Let's fix that by taking away the requirement for the L2 component, which is the network.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px">Notice that a network marked only as external doesn't allow normal </span><span style="font-size:12.8000001907349px">tenants to create ports.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">It does if the owner is a normal tenant. An admin (or a service tenant permitted by policy.json) also can as well, which is particularly relevant when you run service VMs in the external network (or VMs as routers).</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px">The bad assumption here is that floating IPs need an explicit </span><span style="font-size:12.8000001907349px">association with an L2 only construct: tenant's allocate a floating </span><span style="font-size:12.8000001907349px">IP by selecting the Neutron network it is recorded in the DB that way.</span><br></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I agree, but lets fix it by taking the floating IP off of the network. Not by trying to strip l2 out of the network for this one case.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px">These are the factors that make me view the </span><span style="font-size:12.8000001907349px">Neutron network as an L2 + L3 construct.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I just view it as an indication that we were missing a way to reference a group of subnets.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px">Is this really so different from what I'm trying to do with networks? </span><span style="font-size:12.8000001907349px">Make the L2 part nullable.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I think so, the difference is that I just want to eliminate a dependency between the objects. The L3 networks proposal essentially nullifies everything in the network except for the subnets it contains. There is nothing L3 related in a network except for those subnets.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div>><span style="font-size:12.8000001907349px">In your model, should the port be associated with the "group of </span><span style="font-size:12.8000001907349px">subnets" at all? I'm not sure I see a need for it to be directly </span><span style="font-size:12.8000001907349px">associated but I haven't thought it all the way through.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">Probably not. It would just be a part of the creation request where you could say that you want it allocated from a group of subnets. The actual association would just be to a specific subnet in the fixed_ip field I think.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div>><span style="font-size:12.8000001907349px">All the ports are still connected to L2 network segments so I don't </span><span style="font-size:12.8000001907349px">think this is an issue.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I was mixing your spec with Neil's needs as well. I had thought we were trying to solve both (which is what I want to do if we can). For his case, ports need to be directly associated to these new non-L2 things.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div>><span style="font-size:12.8000001907349px">I'm willing to discuss this further. In fact, it has been on my mind </span><span style="font-size:12.8000001907349px">for a while now. This is essentially where I started. I ended up </span><span style="font-size:12.8000001907349px">with my current proposal because I perceived a lot more difficulty in </span><span style="font-size:12.8000001907349px">doing this than in the proposal I created. But, your perspective from </span><span style="font-size:12.8000001907349px">the other side of the problem is worth considering.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I definitely see why you chose this path. I was even in agreement with it at the mid-cycle where we were thinking of using a provider network type of L3. But the more I've been looking through ML2 and the provider networks stuff to see how it would fit, the more I realized how much we are going to have to special-case this one particular type.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I see that you abandoned the spec for this cycle, but maybe we can still put something together towards the end of the cycle...</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 3, 2015 at 2:27 PM, Carl Baldwin <span dir="ltr"><<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Kevin, sorry for the delay in response. Keeping up on this thread was<br>
getting difficult while on vacation.<br>
<br>
tl;dr: I think it is worth it to talk through the idea of inserting<br>
some sort of a "subnet group thing" in the model to which floating ips<br>
(and router external gateways) will associate. It has been on my mind<br>
for a long time now. I didn't pursue it because a few informal<br>
attempts to discuss it with others indicated to me that it would be a<br>
difficult heavy-lifting job that others may not appreciate or<br>
understand. Scroll to the bottom of this message for a little more on<br>
this.<br>
<br>
Carl<br>
<span class=""><br>
On Tue, Jul 28, 2015 at 1:15 AM, Kevin Benton <<a href="mailto:blak111@gmail.com">blak111@gmail.com</a>> wrote:<br>
>>Also, in my proposal, it is more the router that is the grouping mechanism.<br>
><br>
> I can't reconcile this with all of the points you make in the rest of your<br>
> email. You want the collection of subnets that a network represents, but you<br>
> don't want any other properties of the network.<br>
<br>
</span>This is closer to what I'm trying to say but isn't quite there. There<br>
are some subnets that should be associated with the segments<br>
themselves and there are some that should be associated with the<br>
collection of segments. I want floating IPs that are not tied the an<br>
L2 network. None of the alternate proposals that I'd heard addressed<br>
this.<br>
<span class=""><br>
>>that the network object is currently the closest thing we have to<br>
>> representing the L3 part of the network.<br>
><br>
> The L3 part of a network is the subnets. You can't have IP addresses without<br>
> the subnets, you can't have floating IPs without the subnets, etc.<br>
<br>
</span>You're right but in the current model you can't have IP addresses<br>
without the network either which is actually the point I'm trying to<br>
make.<br>
<span class=""><br>
> A Neutron network is an L2 construct that encapsulates L3 things. By<br>
> encapsulating them it also provides an implicit grouping. The routed<br>
> networks proposal basically wants that implicit grouping without the<br>
> encapsulation or the L2 part.<br>
<br>
</span>This sounds about right. I think it is wrong to assume that we need<br>
an L2 network to encapsulate L3 things. I'm feeling restricted by the<br>
model and the insistence that a neutron network is a purely L2<br>
construct.<br>
<span class=""><br>
>>We don't associate floating ips with a network because we want to arp for<br>
>> them. You're taking a consequence of the current model and its constraints<br>
>> and presenting that as the motivation for the model. We do so because there<br>
>> is no better L3 object to associate it to.<br>
><br>
> Don't make assumptions about how people use floating IPs now just because it<br>
> doesn't fit your use-case well. When an external network is implemented as a<br>
> real Neutron network (leaving external_network_bridge blank like we suggest<br>
> in the networking guide), normal ports can be attached and can<br>
> co-exist/communicate with the floating IPs because it behaves as an L2<br>
> network exactly as implied by the API. The current model works quite well if<br>
> your fabric can extend the external network everywhere it needs to be.<br>
<br>
</span>Yes, "when an external network is implemented as a real Neutron<br>
network" all of this is true and my proposal doesn't change any of<br>
this. I'm wasn't making any such assumptions. I acknowledge that the<br>
current model works well in this case and didn't intend to change it<br>
for current use cases. It is precisely that because it does not fit<br>
my use case well that I'm pursuing this.<br>
<br>
Notice that a network marked only as external doesn't allow normal<br>
tenants to create ports. It must also be marked shared to allow it.<br>
Unless tenants are creating regular ports then they really don't care<br>
if arp or anything else L2 is involved because such an external<br>
network is meant to give access external to the cloud where L2 is<br>
really just an implementation detail. It is the deployer that cares<br>
because of whatever infrastructure (like a gateway router) needs to<br>
work with it. If the L2 is important, then the deployer will not<br>
attempt to use an L3 only network, she will use the same kinds of<br>
networks as always.<br>
<br>
The bad assumption here is that floating IPs need an explicit<br>
association with an L2 only construct: tenant's allocate a floating<br>
IP by selecting the Neutron network it is recorded in the DB that way.<br>
Tenant's aren't even allowed to see the subnets on an external<br>
network. This is counter-intuitive to me because I believe that, in<br>
most cases, tenants want a floating IP to get L3 access to the world<br>
(or a part of it) that is external to Openstack. Yet, they can only<br>
see the L2 object? These are the factors that make me view the<br>
Neutron network as an L2 + L3 construct.<br>
<span class=""><br>
> If you don't want floating IPs to be reachable on the network they are<br>
> associated with, then let's stop associating them with a network and instead<br>
> start associating them with a group of subnets from which they allocate IPs.<br>
<br>
</span>Okay. I'm willing to take a serious look at this. This isn't merely<br>
associating a floating IP with a subnet. The tenant shouldn't need to<br>
know about the individual cidrs of the L3 network and wonder if there<br>
is some significant difference or "flavor" of each. I think we truly<br>
need something that represents the group of subnets as you said.<br>
<span class=""><br>
>>If we insist on a new object for the L3 part of a network then I'd say we<br>
>> had better have an L3 only port to connect to it.<br>
><br>
> I don't think a new port type is necessary. We can just make the network ID<br>
> nullable for a port to indicate that it isn't attached to a Neutron network<br>
> since it won't be. It would then have a relationship to its associated<br>
> subnet via fixed_ips as it does now.<br>
<br>
</span>Is this really so different from what I'm trying to do with networks?<br>
Make the L2 part nullable.<br>
<span class=""><br>
>>The subnet is not the L3 object that we're looking for. You may wish it<br>
>> were but that does not make it so.<br>
><br>
> I never said a subnet is what we are looking for. The group of subnets is<br>
> what we seem to be after.<br>
<br>
</span>Agreed as I stated above.<br>
<span class=""><br>
>>Ignoring the forced dependence on L2, the subnets still don't stand alone<br>
>> to describe just the L3 part, they must be linked to a network to make any<br>
>> sense.<br>
><br>
> They don't need to be. If we made the network_id nullable on ports and<br>
> subnets, we could still have ports associated with subnets. The network<br>
> portion is the L2 portion. You don't want L2 so don't ask for the network.<br>
<br>
</span>In your model, should the port be associated with the "group of<br>
subnets" at all? I'm not sure I see a need for it to be directly<br>
associated but I haven't thought it all the way through.<br>
<span class=""><br>
> I understand that we want a way to reference collections of subnets and<br>
> create ports that allocate IPs from them. Networks provide that now, but<br>
> they imply an L2 domain for the ports, which we don't want. So we are trying<br>
> to change what a network implies for this one special case, which is going<br>
> to have ripple effects everywhere.<br>
><br>
> Here are some areas where I can already see we will need special-casing:<br>
><br>
> DHCP agent scheduling - broadcast doesn't work on L3 networks, every compute<br>
> node will need to use the direct tap attachment logic Neil brought up.<br>
<br>
</span>My proposal only created the ports on the network segments.<br>
Admittedly, that is the ugliest part of the proposal but it did<br>
obviate the need for DHCP or arp for the port.<br>
<span class=""><br>
> DHCP lease generation - a port can't get the normal subnet mask for the L3<br>
> network it's attached to because it would try to ARP for addresses in the<br>
> same subnet, which are actually somewhere else.<br>
<br>
</span>See above.<br>
<span class=""><br>
> Router interface attachment - what happens when a user attaches one<br>
> interface to a regular network and one to an L3 network? Before they were<br>
> all L2 networks so it didn't matter, but now none of the ports will be<br>
> reachable on the L3 network without route table manipulation.<br>
> (or) Router creation - to avoid the above you can have different router<br>
> types that can only attach to one or the other.<br>
<br>
</span>Not sure I'm following here. The ports are all created on the segments.<br>
<span class=""><br>
> Every L2 attribute related to networks - we will need logic in the API that<br>
> hides these fields or marks them as invalid and to generate an error if the<br>
> user tries to update them.<br>
> Multi-provider segments - We can't let a user add an L3 segment to a network<br>
> with L2 segments (e.g. VXLAN, VLAN). Same goes for the inverse.<br>
> Hierarchical port binding - coordinating ToRs for VXLAN+VLAN is l2 encap. L3<br>
> would need route propagation logic instead.<br>
<br>
</span>All the ports are still connected to L2 network segments so I don't<br>
think this is an issue.<br>
<span class=""><br>
> Every plugin, service, tool, etc, built on the assumption that a Neutron<br>
> network is L2.<br>
<br>
</span>ok<br>
<span class=""><br>
> Port creation - If you aren't doing the full l3 like Neil's proposal, you<br>
> need to intercept port creation requests and schedule the port to one of the<br>
> underlying regular networks. The port then has a different network_id than<br>
> the one requested, or we have more special-casing to hide that.<br>
<br>
</span>ok, this was called out and I admit it is the ugliest part of the proposal.<br>
<span class=""><br>
> All of those will be branches in the codebase to handle current Neutron<br>
> networks vs L3 networks. If we go down this route, it will be even worse<br>
> than the conditionals we have to support DVR in ML2 because we are exposing<br>
> it via the API. It's technical debt that we will not be able to get rid of.<br>
<br>
</span>I don't think it is nearly as bad as you make it out to be.<br>
<span class=""><br>
> I would rather see something to reference a group of subnets that can be<br>
> used for floating IP allocation and port creation in lieu of a network ID<br>
> than the technical debt that conditionally redefining a network will bring.<br>
<br>
</span>I'm willing to discuss this further. In fact, it has been on my mind<br>
for a while now. This is essentially where I started. I ended up<br>
with my current proposal because I perceived a lot more difficulty in<br>
doing this than in the proposal I created. But, your perspective from<br>
the other side of the problem is worth considering.<br>
<br>
I'm glad to see that at least one other person seems to be<br>
understanding the problem here. This will have API and end-user<br>
impact is it changes the way that end users interact with floating IPs<br>
at least. It will also affect the way that neutron routers are<br>
associated to a network. Today's use case where a user connects a<br>
router to an external network will also change. To what extent do we<br>
support backward compatibility for existing work-flows? For example,<br>
can a user port an existing work-flow to a cloud where the "external<br>
network" is now a group of subnets routed among segments instead of a<br>
neutron network?<br>
<div class="HOEnZb"><div class="h5"><br>
Carl<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>