Open Stack

>is definitely broken as far as I can tell, because you have to give up
dynamic allocation of IP addresses to use it

What do you mean you have to give up dynamic allocation of IP addresses? A
user is never required to enter an IP address for a port if they don't want
a specific address. Do you mean you want your instances IP addresses to
change when they are rebooted? The current model allows multiple DHCP
servers from different subnets operate in the same broadcast domain because
MAC addresses are bound to specific DHCP servers and they don't answer DHCP
requests from arbitrary MAC addresses.

>As far as I can tell, you can create multiple Subnets each with DHCP
enabled and their own allocation pools, but only allocation pools from the
first subnet to be created will ever be used. To assign a port to a
different subnet, you need to specify a static IP.

No. I tried to point this out in my first email. When creating a port, use
the subnet_id parameter after --fixed-ip to specify which subnet to connect
to. For example:
*neutron port-create some_network --fixed-ip
subnet_id=a512cdd3-489d-4317-a06b-10cf894cff5d*

That will dynamically allocate it an address from the specified subnet
instead of the first one.

--
Kevin Benton


On Wed, Apr 9, 2014 at 4:02 PM, Zane Bitter <zbitter at redhat.com> wrote:

> On 07/04/14 21:58, Nachi Ueno wrote:
>
>> Hi Zane
>>
>> Thank you for your very valuable post.
>> We should convert your suggest to multiple bps.
>>
>> 2014-04-07 17:28 GMT-07:00 Zane Bitter <zbitter at redhat.com>:
>>
>>> The Neutron API is a constant cause of pain for us as Heat developers,
>>> but
>>> afaik we've never attempted to bring up the issues we have found in a
>>> cross-project forum. I've recently been doing some more investigation
>>> and I
>>> want to document the exact ways in which the current Neutron API breaks
>>> orchestration, both in the hope that a future version of it might be
>>> better
>>> and as a guide for other API authors.
>>>
>>> BTW it's my contention that an API that is bad for orchestration is also
>>> hard to use for the ordinary user as well. When you're trying to figure
>>> out
>>> the order of operations you need to do, there are two times at which you
>>> could find out you've got it wrong:
>>>
>>> 1) Before you run the command, when you realise you don't have all of the
>>> required data yet; or
>>> 2) After you run the command, when you get a cryptic error message.
>>>
>>> Not only is (1) *mandatory* for a data-driven orchestration system like
>>> Heat, it offers orders-of-magnitude better user experience for everyone.
>>>
>>> I should say at the outset that I know next to nothing about Neutron, and
>>> one of the goals of this message is to find out which parts I am
>>> completely
>>> wrong about. I did know a little bit about traditional networking at one
>>> time, and even remember some of it ;)
>>>
>>>
>>> Neutron has a little documentation on workflow, so let's begin there:
>>> http://docs.openstack.org/api/openstack-network/2.0/content/
>>> Overview-d1e71.html#Theory
>>>
>>> (1) Create a network
>>> Instinctively, I want a Network to be something like a virtual VRF
>>> (VVRF?):
>>> a separate namespace with it's own route table, within which subnet
>>> prefixes
>>> are not overlapping, but which is completely independent of other
>>> Networks
>>> that may contain overlapping subnets. As far as I can tell, this
>>> basically
>>> seems to be the case. The difference, of course, is that instead of
>>> having
>>> to configure a VRF on every switch/router and make sure they're all in
>>> sync
>>> and connected up in the right ways, I just define it in one place
>>> globally
>>> and Neutron does the rest. I call this #winning. Nice work, Neutron.
>>>
>>
>> In Neutron,  "A network is an isolated virtual layer-2 broadcast domain"
>> http://docs.openstack.org/api/openstack-network/2.0/content/
>> Overview-d1e71.html#subnet
>> so the model don't have any L3 stuffs.
>>
>>  (2) Associate a subnet with the network
>>> Slightly odd choice of words, because you're actually creating a new
>>> Subnet
>>> (there's no such thing as a Subnet not associated with a Network), but
>>> this
>>> is probably just a minor documentation nit. Instinctively, I want a
>>> Subnet
>>> to be something like a virtual VLAN (VVLAN?): at its most basic level,
>>> just
>>> a group of ports that share a broadcast domain, but also having other
>>> properties (e.g. if L3 is in use, all IP addresses in the subnet should
>>> be
>>> in the same CIDR). This doesn't seem to be the case, though, it's just a
>>> CIDR prefix, which leaves me wondering how L2 traffic will be treated, as
>>> well as how I would do things like use both IPv4 and IPv6 on a single
>>> port
>>> (by assigning a port to multiple Subnets?). Looking at the docs, there
>>> is a
>>> much bigger emphasis on DHCP client settings than I expected - surely I
>>> might want to want to give two sets of ports in the same Subnet different
>>> DHCP configs? Still, this is not bad - the DHCP configuration is done by
>>> the
>>> time the Subnet is created, so there's no problem in connecting stuff to
>>> it
>>> immediately after.
>>>
>>
>> so, subnet has many meanings.
>> In neutron, it means
>> "A subnet represents an IP address block that can be used to assign IP
>> addresses to virtual instances."
>> http://docs.openstack.org/api/openstack-network/2.0/content/
>> Overview-d1e71.html#subnet
>>
>> so "subnet" in your definition is more like "network" in neutron.
>>
>
> Thanks for explaining this :)
>
> I'm trying to think of the possible reasons for wanting to create multiple
> Subnets on one Network:
>
> (a) To provide different DHCP options to different sets of servers.
> (b) To allow addresses from multiple families (i.e. IPv4 & IPv6) to be
> assigned on the same Network.
> (c) To mix addresses of different prefixes on the same network segment.
>
> (a) is definitely broken as far as I can tell, because you have to give up
> dynamic allocation of IP addresses to use it. The fact that the
> extra-dhcp-opt extension works on a per-port level looks like an admission
> of defeat here. (b) could be broken if Subnets work as has been described
> so far, but it's quite possible there is an exception where a port is
> assigned to multiple Subnets (one per address family, rather than just one
> total). I don't know if that's the case? Finally, I always thought that (c)
> is frowned-upon except as a very temporary measure.
>
> In any event, these are 3 very different use cases (perhaps more exist?),
> and the current Subnet API doesn't seem to quite fit any of them.
>
>
>  (3) Boot a VM and attach it to the network
>>> Here's where you completely lost me. I just created a Subnet - maybe a
>>> bunch
>>> of Subnets. I don't want to attach my VM just anywhere in the *Network*,
>>> I
>>> want to attach it to a *particular* Subnet. It's not at all obvious
>>> where my
>>> instance will get attached (at random?), because this API just plain
>>> takes
>>> the Wrong data type. As a user, I'm irritated and confused.
>>>
>>
>> +1 for specifying subnet on booting server.
>> We should have a bp in nova side and neutron.
>>
>>  The situation for orchestration, though, is much, much worse. Because the
>>> server takes a reference to a network, the dependency graph generated
>>> from
>>> my template will look like this:
>>>
>>>     Network <---------- Subnet
>>>           ^
>>>           \
>>>            ------------ Server
>>>
>>> And yet if the Server is created before the Subnet (as will happen ~50%
>>> of
>>> the time), it will fail. And vice-versa on delete, where the server must
>>> be
>>> removed before the subnet. The dependency graph we needed to create was
>>> this:
>>>
>>>     Network <---------- Subnet <---------- Server
>>>
>>> The solution used here was to jury-rig the resource types in Heat with a
>>> hidden dependency. We can't know which Subnet the server will end up
>>> attached to, so we create hidden dependencies on all of the ones defined
>>> in
>>> the same template. There's nothing we can do about Subnets defined in
>>> different templates (Heat allows a tree of templates to be instantiated
>>> with
>>> a single command) - I'm not sure, but it may be possible even now to
>>> create
>>> a tree of stacks that in practice could never be successfully deleted.
>>>
>>> The Neutron models in Heat are so riddled with these kinds of invisible
>>> special-case hacks that all of our public documentation about how Heat
>>> can
>>> be expected to respond to a particular template is rendered effectively
>>> meaningless with respect to Neutron.
>>>
>>> I should add that we can't blame Nova here, because explicitly creating a
>>> Port doesn't help - it too takes only a network argument, despite
>>> _requiring_ a Subnet that it will be attached to, presumably at random.
>>>
>>
>> We can create a port with subnet_id, so this looks like doc bug for this
>> part.
>> https://gist.github.com/nati/10080957
>>
>
> That is mentioned in the docs, but it only applies if you're assigning a
> static IP.
>
> As far as I can tell, you can create multiple Subnets each with DHCP
> enabled and their own allocation pools, but only allocation pools from the
> first subnet to be created will ever be used. To assign a port to a
> different subnet, you need to specify a static IP.
>
>
>    In
>>
>>> fact using a Port makes things even worse, because although there is an
>>> API
>>> for it Nova and Neutron seem to assume that nobody would ever use it, and
>>> therefore even if you create a port explicitly and pass it to Nova to
>>> connect a Server, when you disconnect the Server again the Port will be
>>> deleted at the same time as if you had let Nova create it implicitly for
>>> you. This issue is currently breaking stack updates because we tend to
>>> assume that once we've explicitly created something, it stays created.
>>>
>>
>> I agree the use case for this.
>> We should have a bp for prevent auto deletion of port.
>>
>> # However, an api user can also delete anything, so we can't assume
>> anything stays created
>> in the other hand. But this is a different topic.
>>
>>  Evidently there is a mechanism for associating a Port with a Subnet, and
>>> that's by assigning a fixed IP - which is hardly ever what I want.
>>> There's
>>> no middle ground that I can find between specifying the exact, fixed IP
>>> for
>>> a port and just letting it end up somewhere - anywhere - on the network,
>>> entirely at random.
>>>
>>
>> We should have a bp for this.
>>
>>
>>> Let's move on to the L3 extension, starting with Routers. There's kind
>>> of an
>>> inconsistency here, because Routers are virtual devices that I need to
>>> manage. Hitherto, the point of Neutron was to free me from managing
>>> individual devices and let me manage the network as a whole. Is there a
>>> reason I wouldn't want all of the Subnets in the Network to just do the
>>> Right Thing and make sure everywhere is reachable efficiently from
>>> everywhere else? If I want something separate, wouldn't I use a different
>>> Network? (It's not like I have any control over where in a Network ports
>>> get
>>> attached anyway.)
>>>
>>
>> We are working on this issue in Group based policy.
>> If we could implement this, there is no need to think about virtual
>> device any more.
>>
>
> \o/
>
>
>  Nonetheless, Routers exist and it appears I have to create one to route
>>> packets between Subnets. From an orchestration perspective, I'd like
>>> Router
>>> to take a list of Ports to attach to (and of course I'd like each Port
>>> to be
>>> explicitly associated with a Subnet!). I'd be out of luck though, because
>>> even though the Port list is a property of a Router, you can't set it at
>>> creation time, only through an update. This is by definition possible to
>>> do
>>> at creation time (if I can do a create call immediately followed by an
>>> update call then the Neutron API can certainly do this internally), so
>>> it's
>>> very strange to see it disallowed. Following this API led us to
>>> implement it
>>> wrong in Heat as well, leading to headaches with floating IPs, about
>>> which
>>> more later. We also mistakenly used a similar design for the Router's
>>> external gateway, but later corrected it by making it a property of the
>>> Router, as it is in the API (though we still have to live with a lengthy
>>> deprecation period). We'll probably end up doing the same with the
>>> interfaces.
>>>
>>
>> +1 for have a bp for specifying subnet in router creation.
>>
>>  Of course it goes without saying that the router gateway is just a
>>> reference
>>> to another network and, once again, requires a hidden dependency on all
>>> of
>>> the Subnets in the hopes of picking up the right one. BTW I'm just
>>> assuming
>>> that the definition of the gateway is "interface to another Network over
>>> which I will do NAT"? I assume that because of the generic way in which
>>> Floating IPs are handled, with a reference to an external network (I
>>> guess
>>> the operator provides the user with the Network UUID for the Internet?)
>>> It's
>>> not exactly clear why the external gateway is special enough that you can
>>> have only one interface of this type on a Router, but not so special
>>> that it
>>> would be considered a separate thing. There is also a separate Network
>>> Gateway, and I have no idea what that is...
>>>
>>> The big problem with Floating IPs is that you can't create them until all
>>> the necessary hops in the internetwork have been set up. And, once again,
>>> there's nothing in the creation parameters that would naturally order
>>> them -
>>> you just pass a reference to the external network. We still have a bug
>>> open
>>> on this, but what we will have to do is create a hidden dependency on any
>>> RouterInterfaces that connect any Routers whose external gateway is the
>>> same
>>> network where the floating IP is allocated. That's about as horrible as
>>> it
>>> sounds. A Floating IP needs to take as an argument a reference to the
>>> Router/Gateway which does the NAT:
>>>
>>> External       External
>>> Network  <---- Subnet   <---- (gateway)
>>>                              \
>>>                              Router <---- Floating IP
>>> Internal                     /               /
>>> Network  <---- Subnet <------<---- Port <----
>>>
>>
>> Sorry, I couldn't get point here. I agree it is not flexible.
>> so what's model you will be happy on here?
>>
>
> The model above is the one that I would be happy with. The current model
> looks something like:
>
> External <-------------------------------- Floating IP
> Network  <----<---- Subnet                   /
>                \                            /
>               Router <---- Port            /
>                /           /              /
> Internal <----<------------<--- Port <----
> Network  <--------- Subnet
>
> So that we currently need hidden dependencies to ensure that the
> Router/gateway is fully set up an connected before attempting to create the
> floating IP.
>
>
>  The bane of my existence during Icehouse development has been the
>>> ExtraRoutes table. First off, this is broken in a way completely
>>> unrelated
>>> to orchestration: you can't add, remove or change an entry in the table
>>> without rewriting the whole table, so the whole API is a giant race
>>> condition waiting to happen. (This can, and IMHO should, be fixed - at
>>> least
>>> for those using the official client - with an ETags header and the 409
>>> return code.) Everything about this API, though, is strange. It's another
>>> one of those only-on-update properties of a Router, though in this case
>>> that's forced by the fact that you can't attach the Router to its Subnets
>>> during its creation. An extra route doesn't behave at all like a static
>>> RIB
>>> entry (with a weight and an administrative distance), but much like a FIB
>>> entry (i.e. it's for routes that have already been selected to be
>>> active).
>>> That part makes sense, but the next hop for a FIB entry is a layer 2
>>> address
>>> and this takes an IP address. That makes no sense to me, since the IP
>>> address(es) assigned to the nexthop play no part in how packets are
>>> forwarded. And, of course, it creates massive dependency issues, because
>>> we
>>> don't know which ports are going to end up with the IP addresses
>>> required.
>>> This API should take a reference to a Port as the nexthop. I've been
>>> told we
>>> can't even simulate this in Heat at the moment because a VPN connection
>>> doesn't have a port associated with it. (If the API accepted _either_ a
>>> Port
>>> or a VPN connection, that would be fine by me though.) So far we've been
>>> unable to merge ExtraRoutes into Heat, except for a plugin in /contrib,
>>> for
>>> want of a way to make this reliably work in the correct dependency order
>>> without resorting to progressively worse hacks.
>>>
>>
>> Sorry, I proposed the extension. so please blame me. (hopefully, softly..
>> :) )
>> That was a small first step. We expect a limited use case on the first
>> usecase.
>> This was implemented as a one attribute in the router, so something
>> like security group list in port,
>> we need to send everything for update.
>>
>> Fortunately, we have a bp proposed for this.
>> https://blueprints.launchpad.net/neutron/+spec/extended-route-params
>>
>
> I see, so it will end up looking very much like a traditional RIB.
>
> I'm happy to see a Port appearing as an alternative destination in that
> document :) Purely from Heat's perspective, it would be even better if it
> could be made compulsory.
>
>
>  I'm sure fresh horrors await in corners I have not yet dug into. I must
>>> say
>>> that the VPN Service, happily, is one that seems to have done things
>>> right.
>>> Firewall looks pretty good in itself, although the fact that it is
>>> completely disjoint from any other configuration - i.e. you can't even
>>> specify which network it applies to, let alone which gateway - is
>>> incomprehensible.
>>>
>>>
>>> Over the past couple of development cycles, we've seen a number of
>>> proposals
>>> to push orchestration-like features into Neutron itself. It is now clear
>>> to
>>> me why: because the Neutron API is illegible to external orchestration
>>> tools, this leads to people wanting to do an end run around it.
>>>
>>> I don't expect that the current API can be fixed without breaking
>>> backwards
>>> compatibility, but I hope that folks will take these concepts into
>>> account
>>> the next time the Neutron API gets revised. (I also hope we won't see any
>>> more proposals to effectively reimplement Heat behind the Neutron API ;)
>>> Please fell free to include [Heat] in any discussion along those lines,
>>> we'd
>>> be happy to give feedback on any given API designs. In exchange, if any
>>> Neutron folks are able to explain the exact ways in which my ideas about
>>> how
>>> the current Neutron API does and/or should work are wrong and/or crazy, I
>>> would be most appreciative :)
>>>
>>
>> Best
>> Nachi
>>
>>
>>> cheers,
>>> Zane.
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140409/a55e91cc/attachment.html>