Open Stack

On 20/07/15 18:36, Carl Baldwin wrote:
> I'm looking for feedback from anyone interest but, in particular, I'd
> like feedback from the following people for varying perspectives:
> Mark McClain (proposed alternate), John Belamaric (IPAM), Ryan Tidwell
> (BGP), Neil Jerram (L3 networks), Aaron Rosen (help understand
> multi-provider networks) and you if you're reading this list of names
> and thinking "he forgot me!"
>
> We have been struggling to develop a way to model a network which is
> composed of disjoint L2 networks connected by routers.  The intent of
> this email is to describe the two proposals and request input on the
> two in attempt to choose a direction forward.  But, first:
> requirements.
>
> Requirements:
>
> The network should appear to end users as a single network choice.
> They should not be burdened with choosing between segments.  It might
> interest them that L2 communications may not work between instances on
> this network but that is all.  This has been requested by numerous
> operators [1][4].  It can be useful for external networks and provider
> networks.
>
> The model needs to be flexible enough to support two distinct types of
> addresses:  1) address blocks which are statically bound to a single
> segment and 2) address blocks which are mobile across segments using
> some sort of dynamic routing capability like BGP or programmatically
> injecting routes in to the infrastructure's routers with a plugin.

FWIW, I hadn't previously realized (2) here.

>
> Overlay networks are not the answer to this.  The goal of this effort
> is to scale very large networks with many connected ports by doing L3
> routing (e.g. to the top of rack) instead of using a large continuous
> L2 fabric.  Also, the operators interested in this work do not want
> the complexity of overlay networks [4].
>
> Proposal 1:
>
> We refined this model [2] at the Neutron mid-cycle a couple of weeks
> ago.  This proposal has already resonated reasonably with operators,
> especially those from GoDaddy who attended the Neutron sprint.  Some
> key parts of this proposal are:
>
> 1.  The routed super network is called a front network.  The segments
> are called back(ing) networks.
> 2.  Backing networks are modeled as admin-owned private provider
> networks but otherwise are full-blown Neutron networks.
> 3.  The front network is marked with a new provider type.
> 4.  A Neutron router is created to link the backing networks with
> internal ports.  It represents the collective routing ability of the
> underlying infrastructure.
> 5.  Backing networks are associated with a subset of hosts.
> 6.  Ports created on the front network must have a host binding and
> are actually created on a backing network when all is said and done.
> They carry the ID of the backing network in the DB.
>
> Using Neutron networks to model the segments allows us to fully
> specify the details of each network using the regular Neutron model.
> They could be heterogeneous or homogeneous, it doesn't matter.

You've probably seen Robert Kukura's comment on the related bug at
https://bugs.launchpad.net/neutron/+bug/1458890/comments/30, and there
is a useful detailed description of how the multiprovider extension
works at
https://bugs.launchpad.net/openstack-api-site/+bug/1242019/comments/3. 
I believe it is correct to say that using multiprovider would be an
effective substitute for using multiple backing networks with different
{network_type, physical_network, segmentation_id}, and that logically
multiprovider is aiming to describe the same thing as this email thread
is, i.e. non-overlay mapping onto a physical network composed of
multiple segments.

However, I believe multiprovider does not (per se) address the IP
addressing requirement(s) of the multi-segment scenario.

>
> This proposal offers a clear separation between the statically bound
> and the mobile address blocks by associating the former with the
> backing networks and the latter with the front network.  The mobile
> addresses are modeled just like floating IPs are today but are
> implemented by some plugin code (possibly without NAT).

Couldn't the mobile addresses be _exactly_ like floating IPs already
are?  Why is anything different from floating IPs needed here?

>
> This proposal also provides some advantages for integrating dynamic
> routing.  Since each backing network will, by necessity, have a
> corresponding router in the infrastructure, the relationship between
> dynamic routing speaker, router, and network is clear in the model:
> network <-> speaker <-> router.

I'm not sure exactly what you mean here by 'dynamic routing', but I
think this touches on a key point: can IP routing happen anywhere in a
Neutron network, without being explicitly represented by a router object
in the model?

I think the answer to that should be yes.  It clearly already is in the
underlay if you are using tunnels - the tunnel between two compute hosts
may require multiple IP hops across the fabric.  At the network level
that Neutron networks currently model, the answer is currently no, but I
think it's interesting to consider changing that.

>
> Proposal 2:
>
> This alternate model has not been fully fleshed out.

I should begin by admitting the blame here.  Much of this is a
half-baked idea from me, that I haven't yet had time to explore
properly.  However....

>   Some parts of it
> are still unclear to me.  The basic idea is to give the IPAM system
> information about IP availability on a given host.  When creating a
> port, the binding information would be sent to the IPAM system and the
> system would choose an appropriate address block for the allocation.

Right.  A key requirement, for this to be possible, is that Nova's host
selection happens before the IPAM system is asked to allocate an IP
address.  I have an action to investigate that, but if anyone happens to
know already, please do say.

>
> 1. This alternate model offers no way to distinguish the two types of
> address blocks.

Agreed.  But I wonder if normal floating IPs can be used for the mobile
IP addresses (as also suggested above).

> 2. We don't have the benefit of modeling the segments with Neutron networks.

Agreed, but it appears that multiprovider has already taken a different
view here, and already provides the ability for a network to map to
multiple {network_type, physical_network, segmentation_id} tuples.

>
> It was suggested that hierarchical port binding could help here but I
> see it as orthogonal to this.  Hierarchical port binding extends the
> L2 properties of a port to a hierarchical infrastructure to achieve
> continuous L2 connectivity.  It is also intended for overlay networks.
> That isn't what we're doing here and I don't think it fits.
>
> I have also considered the multi-provider extension [3] for this.
> This is not yet clear to me either.  First, my understanding was that
> this extension describes multi-segment continuous L2 fabrics.

https://bugs.launchpad.net/openstack-api-site/+bug/1242019/comments/3 says:

"Note that, although ML2 can manage binding to multi-segment networks,
neutron does not manage bridging between the segments of a multi-segment
network. This is assumed to be done administratively."

So I think it is not intended for a multiprovider network to be
"continuous".

Again, this touches on the point above about routing happening without
being explicitly represented in the Neutron model...

>   Second,
> there doesn't seem to be any host binding aspect to the multi-provider
> extension.  Third, not all L2 plugins support this extension.  It
> seems silly to require L2 plugin support in order to enable routing
> between segments.

Good point.  If all plugins required the same kind of transformation to
support multiprovider, perhaps that's telling us that the multi-ness
should instead be in a layer above, more like your proposal 1.

>
> It isn't clear to me how a dynamic routing speaker will fit in to this
> model.  My first thought is that it must be integrated with IPAM
> because the IPAM system has the understanding of how to map address
> blocks to infrastructure.  This pushes even more infrastructure
> knowledge down to the IPAM system.  If dynamic routing is pushed down
> to the IPAM system, it will also be necessary to push the association
> of mobile IPs or routed tenant subnets down in to the IPAM system too.
> This means Neutron needs to tell IPAM about every floating IP
> association and every tenant subnet behind a Neutron router in the
> same address scope as the external network.  I'm not convinced that
> IPAM and routing really belong together like this.

I'm afraid I don't yet sufficiently understand the 'dynamic routing'
requirements here.  Can you say more about them?

>
> If you made it this far in this email, you must have some feedback.
> Please help us out.

There are a lot of moving parts here.  I'm afraid I don't yet see any
clarity, but perhaps if we talk about this enough, that will eventually
emerge!

Regards,
    Neil