[openstack-dev] Multinode testing with devstack and neutron broken

Clark Boylan cboylan at sapwetik.org
Wed Oct 12 01:16:05 UTC 2016


On Tue, Oct 11, 2016, at 05:58 PM, Carl Baldwin wrote:
> On Tue, Oct 11, 2016 at 6:26 PM, Clark Boylan <cboylan at sapwetik.org>
> wrote:
> 
> > On Tue, Oct 11, 2016, at 05:14 PM, Carl Baldwin wrote:
> > > I found this [1] added here [2]. Since the provider is using 10.0.0.0/8
> > > on
> > > the public interface, and the subnet pool is also 10.0.0.0/8 the call to
> > > ip
> > > route replace eliminates the route through the public interface.
> >
> > It is also an issue if you have multiple subnets within 10/8 and rely on
> > your default route to route to them (though I don't think we currently
> > have any of these in the gate currently since rackspace sets more
> > specific routes for the subnets used for private networking there). This
> > would be the case if using a neutron setup with tenant networking and
> > floating IPs like we had with the old hpcloud.
> >
> 
> Any time we fiddle with the host's routing table with address ranges that
> we make up, we run the risk of clashing with something "real". What do we
> do? I guess we can try our best to pick ranges that shouldn't clash and
> try
> to minimize how much of the space we use up. Then, for those that get a
> clash, we provide the option to override. That's kind of what we've been
> doing, right? We could try other tricks to pick more obscure IP ranges
> that
> shouldn't be used anywhere (like 100.64.0.0/10 or the ranges reserved for
> documentation, which we do for ipv6) but with so many people competing
> for
> such a small space, I don't know if we can ever pick anything that won't
> ever clash. It just doesn't feel like there is a good answer.

Right my whole argument behind reusing FIXED_RANGE is its the var that
we have set for every devstack test job to avoid such conflicts. Its
value is currently conflict free as far as I know and has been set
intentionally to avoid the various conflicts we have run into over time.
This doesn't mean we will never run into new conflicts but it gives us a
single place to very clearly signal "this is useable". Maybe we need to
add a meta var that FIXED_RANGE and SUBNET POOL ranges are derived from.
Essentially set a value that is a large subnet that isn't conflicting
with anything and have the other ranges split it evenly. Then maintain
backward compat by using any hard set sub ranges that are supplied.

> 
> Could we instead modify devstack to use a different network namespace (as
> in ip nets) to represent the "external" network? I think the real problem
> is that we're trying to use the host's real network as the "external
> network" for our testing. We are combining any combination of real world
> subnets with made up hard-coded default address ranges. If we separated
> the
> two, then there would be no chance for conflict. I'm just thinking out
> loud
> here. There could be all kinds of difficulty with this idea.

Yup that is indeed the problem. Another related issue that makes this
extra turtly is we are running neutron within clouds running neutron. As
a result chance for conflicts is probably higher than if you were just
running on a random host on a random subnet (since we are probably more
likely to leak defaults between neutrons). It would definitely be neat
to separate these things more though. We have been slowly separating the
layer 2 devices (was sort of forced on us initially by OVH's use of /32s
on eth0 and nova nets inability to properly handle that). Probably worth
further exploring that for layer 3 as well though as you say this could
be difficult.

Clark




More information about the OpenStack-dev mailing list