[openstack-dev] [Neutron][L3] Representing a networks connected by routers

Assaf Muller amuller at redhat.com
Wed Jul 22 20:36:33 UTC 2015

I added a summary of my thoughts about the enhancements I think we could
make to the Nova scheduler in order to better support the Neutron provider
networks use case.

----- Original Message -----
> On Tue, Jul 21, 2015 at 1:11 PM, John Belamaric <jbelamaric at infoblox.com>
> wrote:
> > Wow, a lot to digest in these threads. If I can summarize my understanding
> > of the two proposals. Let me know whether I get this right. There are a
> > couple problems that need to be solved:
> >
> >  a. Scheduling based on host reachability to the segments
> >  b. Floating IP functionality across the segments. I am not sure I am clear
> > on this one but it sounds like you want the routers attached to the
> > segments
> > to advertise routes to the specific floating IPs. Presumably then they
> > would
> > do NAT or the instance would assign both the fixed IP and the floating IP
> > to
> > its interface?
> >
> > In Proposal 1, (a) is solved by associating segments to the front network
> > via a router - that association is used to provide a single hook into the
> > existing API that limits the scope of segment selection to those associated
> > with the front network. (b) is solved by tying the floating IP ranges to
> > the
> > same front network and managing the reachability with dynamic routing.
> >
> > In Proposal 2, (a) is solved by tagging each network with some meta-data
> > that the IPAM system uses to make a selection. This implies an IP
> > allocation
> > request that passes something other than a network/port to the IPAM
> > subsystem. This fine from the IPAM point of view but there is no
> > corresponding API for this right now. To solve (b) either the IPAM system
> > has to publish the routes or the higher level management has to ALSO be
> > aware of the mappings (rather than just IPAM).
> John, from your summary above, you seem to have the best understanding
> of the whole of what I was weakly attempting to communicate.  Thank
> you for summarizing.
> > To throw some fuel on the fire, I would argue also that (a) is not
> > sufficient and address availability needs to be considered as well (as
> > described in [1]). Selecting a host based on reachability alone will fail
> > when addresses are exhausted. Similarly, with (b) I think there needs to be
> > consideration during association of a floating IP to the effect on routing.
> > That is, rather than a huge number of host routes it would be ideal to
> > allocate the floating IPs in blocks that can be associated with the backing
> > networks (though we would want to be able to split these blocks as small as
> > a /32 if necessary - but avoid it/optimize as much as possible).
> Yes, address availability is a factor and must be considered in either
> case.  My email was getting long already and I thought that could be
> considered separately since I believe it applies regardless of the
> outcome of this thread.  But, since it seems to be an essential part
> of this conversation, let me say something about it.
> Ultimately, we need to match up the host scheduled by Nova to the
> addresses available to that host.  We could do this by delaying
> address assignment until after host binding or we could do it by
> including segment information from Neutron during scheduling.  The
> latter has the advantage that we can consider IP availability during
> scheduling.  That is why GoDaddy implemented it that way.
> > In fact, I think that these proposals are more or less the same - it's just
> > in #1 the meta-data used to tie the backing networks together is another
> > network. This allows it to fit in neatly with the existing APIs. You would
> > still need to implement something prior to IPAM or within IPAM that would
> > select the appropriate backing network.
> They are similar but to say they're the same is going a bit too far.
> If they were the same then we'd be done with this conversation.  ;)
> > As a (gulp) third alternative, we should consider that the front network
> > here is in essence a layer 3 domain, and we have modeled layer 3 domains as
> > address scopes in Liberty. The user is essentially saying "give me an
> > address that is routable in this scope" - they don't care which actual
> > subnet it gets allocated on. This is conceptually more in-line with [2] -
> > modeling L3 domain separately from the existing Neutron concept of a
> > network
> > being a broadcast domain.
> I will consider this some more.  This is an interesting thought.
> Address scopes and subnet pools could play a role here.  I don't yet
> see how it can all fit together but it is worth some thought.
> One nit:  the neutron network might have been conceived as being just
> "a broadcast domain" but, in practice, it is L2 and L3.  The Neutron
> subnet is not really an L3 construct; it is just a cidr and doesn't
> make sense on its own without considering its association with a
> network and the other subnets associated with the same network.
> > Fundamentally, however we associate the segments together, this comes down
> > to a scheduling problem. Nova needs to be able to incorporate data from
> > Neutron in its scheduling decision. Rather than solving this with a single
> > piece of meta-data like network_id as described in proposal 1, it probably
> > makes more sense to build out the general concept of utilizing network data
> > for nova scheduling. We could still model this as in #1, or using address
> > scopes, or some arbitrary data as in #2. But the harder problem to solve is
> > the scheduling, not how we tag these things to inform that scheduling.
> Yet how we tag these things seems to be a significant point of
> interest.  Maybe not with you but with Ian and Assaf it certainly is.
> As I said above, I agree that the scheduling part is very important
> and needs to be discussed but I still separate them in my mind from
> this question.

I'm basing these ideas off my understanding of the GoDaddy, YY and Yahoo requirements in
https://etherpad.openstack.org/p/Network_Segmentation_Usecases. I am purposely not looking
at the problems being presented by Calico or similar /32's BGP advertising implementations,
nor the idea of injecting floating IPs, as I believe those to be separate problems, and conflating
them with everything else presented in that Etherpad would be a mistake. In other words I'm not
trying to solve all of the problems that have ever existed, just some of them :) I'd love to get
feedback from the authors of that Etherpad to see how much progress we'd be making here and if it's in the right direction.

Neutron supports self service networking, often implemented by overlay networks. An overlay (GRE, VXLAN) based
network is not location-sensitive, that is, all compute nodes would have access to such a network, as long as
the compute nodes can ping each other (And this may be realized over layer 2 or via routing in your data center).
Some deployments opt out of this type of solution, and instead an admin pre-creates and shares a network(s) that is
realized via VLANs. Tenants connect their VMs to these pre-created networks and don't create networks of their own.
It may be the case where not all compute nodes would have access to such a network. Here's some pretty graphics:

Problem 1:
In this example VLANs 11 and 12, and subnets, are only available in rack 1.
In this case the admin would create four Neutron networks (VLANs 11, 12, 13 and 14 with their respective subnets).
However, the Nova scheduler is not exposed to this information. This means that if a VM is booted on network 1
(And an AZ is not specified), Nova may try to start it in rack 2, where network 1 is not available.
Neutron port binding would fail in this case and the VM will end up in the error state.

Tag Neutron networks with an AZ as detailed here http://specs.openstack.org/openstack/neutron-specs/specs/liberty/availability-zone.html.
This means that when the admin creates network 1 he'll put it in AZ1. When a VM is booted on network 1,
the Nova scheduler will only consider hosts in AZ 1. If an AZ is specified, then Nova will fail-fast and yell
if the specified AZ doesn't match the AZ the Neutron network is in. If a network is not in an AZ (AZ == None)
then the behavior is backwards compatible. Currently it is assumed that all hosts have access to all networks,
while now the assumption will be that all hosts in the same AZ have access to all networks in that AZ.

Problem & Solution 2:
With tenant networking it makes sense to select the network a VM would boot on.
For example if a tenant created three networks (Say: DB, backend and web tiers, each with its own network and security group)
then each VM would need to go on a specific network according to its role.
With provider networking, you may want to boot a VM and let Nova select the appropriate network for you.
To clarify, you would not specify a network_id or port_id when booting a VM. Nova would schedule the VM to host 1 in rack 1,
and then randomly select network 1 or 2 (Because those are available to the AZ that host 1 is in, and the Nova scheduler would know this with problem 1 solved).

Problem 3:
In case the 'nova selects a network for you' (Marked as problem/solution 2) proposal is implemented,
you could run in to an issue where the IP addresses on the Nova selected network are exhausted,
and that another network available in that rack/AZ should have been chosen instead.

The nova scheduler could depend on: https://review.openstack.org/#/c/180803/ - A new API to report IP availability per network.
Then, as an additional built-in scheduling filter, when choosing a network, make sure that the network has an IP address available.

Problem 4 (Disclaimer: This one isn't as well thought out):
When Nova selects a network (Either from a specific AZ or not), solution 2 suggests that it'll essentially be a random choice,
apart from IP availability as defined in the solution to problem 3. It may be the case where the user doesn't want to specify which
network the VM will be connected to, but specify a property that the network must satisfy, such as the security zone
(This is taken straight from the Etherpad).

A new 'tags' property will be added to the network model, as a list of strings (Or perhaps couples consisting of a tag and its description).
When creating a network you could specify arbitrary data to be placed in those tags. When a user boots a VM he could specify a tag (Or tags)
instead of a network/port_id, and the Nova scheduler will filter out any networks that do not have that tag(s).
Tags will be writable by the owner of the network (So an admin in case of provider networks) and readable by anyone else.
Here's some more pretty graphics that would perhaps explain things better: http://i.imgur.com/89apoA8.png.

> > The optimization of routing for floating IPs is also a scheduling problem,
> > though one that would require a lot more changes to how FIP are allocated
> > and associated to solve.
> >
> > John
> >
> > [1] https://review.openstack.org/#/c/180803/
> > [2] https://bugs.launchpad.net/neutron/+bug/1458890/comments/7
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list