Open Stack

On 07/04/14 21:58, Nachi Ueno wrote:
> Hi Zane
>
> Thank you for your very valuable post.
> We should convert your suggest to multiple bps.
>
> 2014-04-07 17:28 GMT-07:00 Zane Bitter <zbitter at redhat.com>:
>> The Neutron API is a constant cause of pain for us as Heat developers, but
>> afaik we've never attempted to bring up the issues we have found in a
>> cross-project forum. I've recently been doing some more investigation and I
>> want to document the exact ways in which the current Neutron API breaks
>> orchestration, both in the hope that a future version of it might be better
>> and as a guide for other API authors.
>>
>> BTW it's my contention that an API that is bad for orchestration is also
>> hard to use for the ordinary user as well. When you're trying to figure out
>> the order of operations you need to do, there are two times at which you
>> could find out you've got it wrong:
>>
>> 1) Before you run the command, when you realise you don't have all of the
>> required data yet; or
>> 2) After you run the command, when you get a cryptic error message.
>>
>> Not only is (1) *mandatory* for a data-driven orchestration system like
>> Heat, it offers orders-of-magnitude better user experience for everyone.
>>
>> I should say at the outset that I know next to nothing about Neutron, and
>> one of the goals of this message is to find out which parts I am completely
>> wrong about. I did know a little bit about traditional networking at one
>> time, and even remember some of it ;)
>>
>>
>> Neutron has a little documentation on workflow, so let's begin there:
>> http://docs.openstack.org/api/openstack-network/2.0/content/Overview-d1e71.html#Theory
>>
>> (1) Create a network
>> Instinctively, I want a Network to be something like a virtual VRF (VVRF?):
>> a separate namespace with it's own route table, within which subnet prefixes
>> are not overlapping, but which is completely independent of other Networks
>> that may contain overlapping subnets. As far as I can tell, this basically
>> seems to be the case. The difference, of course, is that instead of having
>> to configure a VRF on every switch/router and make sure they're all in sync
>> and connected up in the right ways, I just define it in one place globally
>> and Neutron does the rest. I call this #winning. Nice work, Neutron.
>
> In Neutron,  "A network is an isolated virtual layer-2 broadcast domain"
> http://docs.openstack.org/api/openstack-network/2.0/content/Overview-d1e71.html#subnet
> so the model don't have any L3 stuffs.
>
>> (2) Associate a subnet with the network
>> Slightly odd choice of words, because you're actually creating a new Subnet
>> (there's no such thing as a Subnet not associated with a Network), but this
>> is probably just a minor documentation nit. Instinctively, I want a Subnet
>> to be something like a virtual VLAN (VVLAN?): at its most basic level, just
>> a group of ports that share a broadcast domain, but also having other
>> properties (e.g. if L3 is in use, all IP addresses in the subnet should be
>> in the same CIDR). This doesn't seem to be the case, though, it's just a
>> CIDR prefix, which leaves me wondering how L2 traffic will be treated, as
>> well as how I would do things like use both IPv4 and IPv6 on a single port
>> (by assigning a port to multiple Subnets?). Looking at the docs, there is a
>> much bigger emphasis on DHCP client settings than I expected - surely I
>> might want to want to give two sets of ports in the same Subnet different
>> DHCP configs? Still, this is not bad - the DHCP configuration is done by the
>> time the Subnet is created, so there's no problem in connecting stuff to it
>> immediately after.
>
> so, subnet has many meanings.
> In neutron, it means
> "A subnet represents an IP address block that can be used to assign IP
> addresses to virtual instances."
> http://docs.openstack.org/api/openstack-network/2.0/content/Overview-d1e71.html#subnet
>
> so "subnet" in your definition is more like "network" in neutron.

Thanks for explaining this :)

I'm trying to think of the possible reasons for wanting to create 
multiple Subnets on one Network:

(a) To provide different DHCP options to different sets of servers.
(b) To allow addresses from multiple families (i.e. IPv4 & IPv6) to be 
assigned on the same Network.
(c) To mix addresses of different prefixes on the same network segment.

(a) is definitely broken as far as I can tell, because you have to give 
up dynamic allocation of IP addresses to use it. The fact that the 
extra-dhcp-opt extension works on a per-port level looks like an 
admission of defeat here. (b) could be broken if Subnets work as has 
been described so far, but it's quite possible there is an exception 
where a port is assigned to multiple Subnets (one per address family, 
rather than just one total). I don't know if that's the case? Finally, I 
always thought that (c) is frowned-upon except as a very temporary measure.

In any event, these are 3 very different use cases (perhaps more 
exist?), and the current Subnet API doesn't seem to quite fit any of them.

>> (3) Boot a VM and attach it to the network
>> Here's where you completely lost me. I just created a Subnet - maybe a bunch
>> of Subnets. I don't want to attach my VM just anywhere in the *Network*, I
>> want to attach it to a *particular* Subnet. It's not at all obvious where my
>> instance will get attached (at random?), because this API just plain takes
>> the Wrong data type. As a user, I'm irritated and confused.
>
> +1 for specifying subnet on booting server.
> We should have a bp in nova side and neutron.
>
>> The situation for orchestration, though, is much, much worse. Because the
>> server takes a reference to a network, the dependency graph generated from
>> my template will look like this:
>>
>>     Network <---------- Subnet
>>           ^
>>           \
>>            ------------ Server
>>
>> And yet if the Server is created before the Subnet (as will happen ~50% of
>> the time), it will fail. And vice-versa on delete, where the server must be
>> removed before the subnet. The dependency graph we needed to create was
>> this:
>>
>>     Network <---------- Subnet <---------- Server
>>
>> The solution used here was to jury-rig the resource types in Heat with a
>> hidden dependency. We can't know which Subnet the server will end up
>> attached to, so we create hidden dependencies on all of the ones defined in
>> the same template. There's nothing we can do about Subnets defined in
>> different templates (Heat allows a tree of templates to be instantiated with
>> a single command) - I'm not sure, but it may be possible even now to create
>> a tree of stacks that in practice could never be successfully deleted.
>>
>> The Neutron models in Heat are so riddled with these kinds of invisible
>> special-case hacks that all of our public documentation about how Heat can
>> be expected to respond to a particular template is rendered effectively
>> meaningless with respect to Neutron.
>>
>> I should add that we can't blame Nova here, because explicitly creating a
>> Port doesn't help - it too takes only a network argument, despite
>> _requiring_ a Subnet that it will be attached to, presumably at random.
>
> We can create a port with subnet_id, so this looks like doc bug for this part.
> https://gist.github.com/nati/10080957

That is mentioned in the docs, but it only applies if you're assigning a 
static IP.

As far as I can tell, you can create multiple Subnets each with DHCP 
enabled and their own allocation pools, but only allocation pools from 
the first subnet to be created will ever be used. To assign a port to a 
different subnet, you need to specify a static IP.

>   In
>> fact using a Port makes things even worse, because although there is an API
>> for it Nova and Neutron seem to assume that nobody would ever use it, and
>> therefore even if you create a port explicitly and pass it to Nova to
>> connect a Server, when you disconnect the Server again the Port will be
>> deleted at the same time as if you had let Nova create it implicitly for
>> you. This issue is currently breaking stack updates because we tend to
>> assume that once we've explicitly created something, it stays created.
>
> I agree the use case for this.
> We should have a bp for prevent auto deletion of port.
>
> # However, an api user can also delete anything, so we can't assume
> anything stays created
> in the other hand. But this is a different topic.
>
>> Evidently there is a mechanism for associating a Port with a Subnet, and
>> that's by assigning a fixed IP - which is hardly ever what I want. There's
>> no middle ground that I can find between specifying the exact, fixed IP for
>> a port and just letting it end up somewhere - anywhere - on the network,
>> entirely at random.
>
> We should have a bp for this.
>
>>
>> Let's move on to the L3 extension, starting with Routers. There's kind of an
>> inconsistency here, because Routers are virtual devices that I need to
>> manage. Hitherto, the point of Neutron was to free me from managing
>> individual devices and let me manage the network as a whole. Is there a
>> reason I wouldn't want all of the Subnets in the Network to just do the
>> Right Thing and make sure everywhere is reachable efficiently from
>> everywhere else? If I want something separate, wouldn't I use a different
>> Network? (It's not like I have any control over where in a Network ports get
>> attached anyway.)
>
> We are working on this issue in Group based policy.
> If we could implement this, there is no need to think about virtual
> device any more.

\o/

>> Nonetheless, Routers exist and it appears I have to create one to route
>> packets between Subnets. From an orchestration perspective, I'd like Router
>> to take a list of Ports to attach to (and of course I'd like each Port to be
>> explicitly associated with a Subnet!). I'd be out of luck though, because
>> even though the Port list is a property of a Router, you can't set it at
>> creation time, only through an update. This is by definition possible to do
>> at creation time (if I can do a create call immediately followed by an
>> update call then the Neutron API can certainly do this internally), so it's
>> very strange to see it disallowed. Following this API led us to implement it
>> wrong in Heat as well, leading to headaches with floating IPs, about which
>> more later. We also mistakenly used a similar design for the Router's
>> external gateway, but later corrected it by making it a property of the
>> Router, as it is in the API (though we still have to live with a lengthy
>> deprecation period). We'll probably end up doing the same with the
>> interfaces.
>
> +1 for have a bp for specifying subnet in router creation.
>
>> Of course it goes without saying that the router gateway is just a reference
>> to another network and, once again, requires a hidden dependency on all of
>> the Subnets in the hopes of picking up the right one. BTW I'm just assuming
>> that the definition of the gateway is "interface to another Network over
>> which I will do NAT"? I assume that because of the generic way in which
>> Floating IPs are handled, with a reference to an external network (I guess
>> the operator provides the user with the Network UUID for the Internet?) It's
>> not exactly clear why the external gateway is special enough that you can
>> have only one interface of this type on a Router, but not so special that it
>> would be considered a separate thing. There is also a separate Network
>> Gateway, and I have no idea what that is...
>>
>> The big problem with Floating IPs is that you can't create them until all
>> the necessary hops in the internetwork have been set up. And, once again,
>> there's nothing in the creation parameters that would naturally order them -
>> you just pass a reference to the external network. We still have a bug open
>> on this, but what we will have to do is create a hidden dependency on any
>> RouterInterfaces that connect any Routers whose external gateway is the same
>> network where the floating IP is allocated. That's about as horrible as it
>> sounds. A Floating IP needs to take as an argument a reference to the
>> Router/Gateway which does the NAT:
>>
>> External       External
>> Network  <---- Subnet   <---- (gateway)
>>                              \
>>                              Router <---- Floating IP
>> Internal                     /               /
>> Network  <---- Subnet <------<---- Port <----
>
> Sorry, I couldn't get point here. I agree it is not flexible.
> so what's model you will be happy on here?

The model above is the one that I would be happy with. The current model 
looks something like:

External <-------------------------------- Floating IP
Network  <----<---- Subnet                   /
                \                            /
               Router <---- Port            /
                /           /              /
Internal <----<------------<--- Port <----
Network  <--------- Subnet

So that we currently need hidden dependencies to ensure that the 
Router/gateway is fully set up an connected before attempting to create 
the floating IP.

>> The bane of my existence during Icehouse development has been the
>> ExtraRoutes table. First off, this is broken in a way completely unrelated
>> to orchestration: you can't add, remove or change an entry in the table
>> without rewriting the whole table, so the whole API is a giant race
>> condition waiting to happen. (This can, and IMHO should, be fixed - at least
>> for those using the official client - with an ETags header and the 409
>> return code.) Everything about this API, though, is strange. It's another
>> one of those only-on-update properties of a Router, though in this case
>> that's forced by the fact that you can't attach the Router to its Subnets
>> during its creation. An extra route doesn't behave at all like a static RIB
>> entry (with a weight and an administrative distance), but much like a FIB
>> entry (i.e. it's for routes that have already been selected to be active).
>> That part makes sense, but the next hop for a FIB entry is a layer 2 address
>> and this takes an IP address. That makes no sense to me, since the IP
>> address(es) assigned to the nexthop play no part in how packets are
>> forwarded. And, of course, it creates massive dependency issues, because we
>> don't know which ports are going to end up with the IP addresses required.
>> This API should take a reference to a Port as the nexthop. I've been told we
>> can't even simulate this in Heat at the moment because a VPN connection
>> doesn't have a port associated with it. (If the API accepted _either_ a Port
>> or a VPN connection, that would be fine by me though.) So far we've been
>> unable to merge ExtraRoutes into Heat, except for a plugin in /contrib, for
>> want of a way to make this reliably work in the correct dependency order
>> without resorting to progressively worse hacks.
>
> Sorry, I proposed the extension. so please blame me. (hopefully, softly.. :) )
> That was a small first step. We expect a limited use case on the first
> usecase.
> This was implemented as a one attribute in the router, so something
> like security group list in port,
> we need to send everything for update.
>
> Fortunately, we have a bp proposed for this.
> https://blueprints.launchpad.net/neutron/+spec/extended-route-params

I see, so it will end up looking very much like a traditional RIB.

I'm happy to see a Port appearing as an alternative destination in that 
document :) Purely from Heat's perspective, it would be even better if 
it could be made compulsory.

>> I'm sure fresh horrors await in corners I have not yet dug into. I must say
>> that the VPN Service, happily, is one that seems to have done things right.
>> Firewall looks pretty good in itself, although the fact that it is
>> completely disjoint from any other configuration - i.e. you can't even
>> specify which network it applies to, let alone which gateway - is
>> incomprehensible.
>>
>>
>> Over the past couple of development cycles, we've seen a number of proposals
>> to push orchestration-like features into Neutron itself. It is now clear to
>> me why: because the Neutron API is illegible to external orchestration
>> tools, this leads to people wanting to do an end run around it.
>>
>> I don't expect that the current API can be fixed without breaking backwards
>> compatibility, but I hope that folks will take these concepts into account
>> the next time the Neutron API gets revised. (I also hope we won't see any
>> more proposals to effectively reimplement Heat behind the Neutron API ;)
>> Please fell free to include [Heat] in any discussion along those lines, we'd
>> be happy to give feedback on any given API designs. In exchange, if any
>> Neutron folks are able to explain the exact ways in which my ideas about how
>> the current Neutron API does and/or should work are wrong and/or crazy, I
>> would be most appreciative :)
>
> Best
> Nachi
>
>>
>> cheers,
>> Zane.
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>