[neutron] parent ports for trunks being claimed by instances
Neutron developers,
I am currently working on an issue with trunk ports that has come up a few times in my direct experience, and I hope that we can create a long term solution. I am hoping that developers with experience in trunk ports can validate my approach here, especially regarding fixing current behavior without introducing an API regression.
By way of introduction to the specifics of the issue, let me blockquote from the LP bug I raised for this [1]:
----
When you create a trunk in Neutron you create a parent port for the trunk and attach the trunk to the parent. Then subports can be created on the trunk. When instances are created on the trunk, first a port is created and then an instance is associated with a free port. It looks to me that's this is the oversight in the logic.
From the perspective of the code, the parent port looks like any other port attached to the trunk bridge. It doesn't have an instance attached to it so it looks like it's not being used for anything (which is technically correct). So it becomes an eligible port for an instance to bind to. That is all fine and dandy until you go to delete the instance and you get the "Port [port-id] is currently a parent port for trunk [trunk-id]" exception just as happened here. Anecdotally, it's seems rare that an instance will actually bind to it, but that is what happened for the user in this case and I have had several pings over the past year about people in a similar state.
I propose that when a port is made parent port for a trunk, that the trunk be established as the owner of the port. That way it will be ineligible for instances seeking to bind to the port.
----
Clearly the above behavior indicates buggy issue that should be rectified in master and stable branches. Nobody wants a VM that can't be fully deleted because the port can't ever be deleted. This is especially egregious when it causes heat stack deletion failures.
I am mostly concerned that by adding the trunk as an owner of the parent port, then the trunk will need to be deleted before the parent port can be deleted, otherwise a PortInUse error will occur when the port is deleted (i.e. on tempest test teardown). That to me seems indicative of an inadvertent API change. Do you think it's all right to say that if you delete a port that is a parent port of a trunk, and that trunk has no other subports, that the trunk deletion is implicit? Is that the lowest impact to the API that we can incur to resolve this issue?
Your wisdom is appreciated,
Nate
Hi Nate,
I think I'm getting configure with the use of "instances" here. With instances, you refer to VMs? Note there is a need to first create the parent, then the trunk and then the instance using that parent.
Also, deleting the VM is not a problem, it will just move the port to down (and the trunk). The problem is deleting the parent port (if it is part of the trunk). I think the problem here is that the VM should not try to delete the port as port was existing before VM creation, right?
Note too that if the parent port is not "eligible" after being part of a trunk, how can you boot a VM with a parent port given that there is a need for having the trunk before booting the VM?
Cheers, Luis
On Tue, May 12, 2020 at 4:13 PM Nate Johnston nate.johnston@redhat.com wrote:
Neutron developers,
I am currently working on an issue with trunk ports that has come up a few times in my direct experience, and I hope that we can create a long term solution. I am hoping that developers with experience in trunk ports can validate my approach here, especially regarding fixing current behavior without introducing an API regression.
By way of introduction to the specifics of the issue, let me blockquote from the LP bug I raised for this [1]:
When you create a trunk in Neutron you create a parent port for thetrunk and attach the trunk to the parent. Then subports can be created on the trunk. When instances are created on the trunk, first a port is created and then an instance is associated with a free port. It looks to me that's this is the oversight in the logic.
From the perspective of the code, the parent port looks like any otherport attached to the trunk bridge. It doesn't have an instance attached to it so it looks like it's not being used for anything (which is technically correct). So it becomes an eligible port for an instance to bind to. That is all fine and dandy until you go to delete the instance and you get the "Port [port-id] is currently a parent port for trunk [trunk-id]" exception just as happened here. Anecdotally, it's seems rare that an instance will actually bind to it, but that is what happened for the user in this case and I have had several pings over the past year about people in a similar state.
I propose that when a port is made parent port for a trunk, that thetrunk be established as the owner of the port. That way it will be ineligible for instances seeking to bind to the port.
Clearly the above behavior indicates buggy issue that should be rectified in master and stable branches. Nobody wants a VM that can't be fully deleted because the port can't ever be deleted. This is especially egregious when it causes heat stack deletion failures.
I am mostly concerned that by adding the trunk as an owner of the parent port, then the trunk will need to be deleted before the parent port can be deleted, otherwise a PortInUse error will occur when the port is deleted (i.e. on tempest test teardown). That to me seems indicative of an inadvertent API change. Do you think it's all right to say that if you delete a port that is a parent port of a trunk, and that trunk has no other subports, that the trunk deletion is implicit? Is that the lowest impact to the API that we can incur to resolve this issue?
Your wisdom is appreciated,
Nate
Hi,
On Wed, May 13, 2020 at 09:42:28AM +0200, Luis Tomas Bolivar wrote:
Hi Nate,
I think I'm getting configure with the use of "instances" here. With instances, you refer to VMs? Note there is a need to first create the parent, then the trunk and then the instance using that parent.
Also, deleting the VM is not a problem, it will just move the port to down (and the trunk). The problem is deleting the parent port (if it is part of the trunk). I think the problem here is that the VM should not try to delete the port as port was existing before VM creation, right?
It is like that if You create port in neutron, and then pass this port to the Nova when booting instance. But if You first create instance by giving network_id to Nova, it will create port on this network for You and that port will be deleted by Nova when instance will be deleted.
Note too that if the parent port is not "eligible" after being part of a trunk, how can you boot a VM with a parent port given that there is a need for having the trunk before booting the VM?
Cheers, Luis
On Tue, May 12, 2020 at 4:13 PM Nate Johnston nate.johnston@redhat.com wrote:
Neutron developers,
I am currently working on an issue with trunk ports that has come up a few times in my direct experience, and I hope that we can create a long term solution. I am hoping that developers with experience in trunk ports can validate my approach here, especially regarding fixing current behavior without introducing an API regression.
By way of introduction to the specifics of the issue, let me blockquote from the LP bug I raised for this [1]:
When you create a trunk in Neutron you create a parent port for thetrunk and attach the trunk to the parent. Then subports can be created on the trunk. When instances are created on the trunk, first a port is created and then an instance is associated with a free port. It looks to me that's this is the oversight in the logic.
From the perspective of the code, the parent port looks like any otherport attached to the trunk bridge. It doesn't have an instance attached to it so it looks like it's not being used for anything (which is technically correct). So it becomes an eligible port for an instance to bind to. That is all fine and dandy until you go to delete the instance and you get the "Port [port-id] is currently a parent port for trunk [trunk-id]" exception just as happened here. Anecdotally, it's seems rare that an instance will actually bind to it, but that is what happened for the user in this case and I have had several pings over the past year about people in a similar state.
I propose that when a port is made parent port for a trunk, that thetrunk be established as the owner of the port. That way it will be ineligible for instances seeking to bind to the port.
Clearly the above behavior indicates buggy issue that should be rectified in master and stable branches. Nobody wants a VM that can't be fully deleted because the port can't ever be deleted. This is especially egregious when it causes heat stack deletion failures.
I am mostly concerned that by adding the trunk as an owner of the parent port, then the trunk will need to be deleted before the parent port can be deleted, otherwise a PortInUse error will occur when the port is deleted (i.e. on tempest test teardown). That to me seems indicative of an inadvertent API change. Do you think it's all right to say that if you delete a port that is a parent port of a trunk, and that trunk has no other subports, that the trunk deletion is implicit? Is that the lowest impact to the API that we can incur to resolve this issue?
Your wisdom is appreciated,
Nate
-- LUIS TOMÃS BOLÃVAR Senior Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.com
On Wed, May 13, 2020 at 1:18 PM Slawek Kaplonski skaplons@redhat.com wrote:
Hi,
On Wed, May 13, 2020 at 09:42:28AM +0200, Luis Tomas Bolivar wrote:
Hi Nate,
I think I'm getting configure with the use of "instances" here. With instances, you refer to VMs? Note there is a need to first create the parent, then the trunk and then the instance using that parent.
Also, deleting the VM is not a problem, it will just move the port to
down
(and the trunk). The problem is deleting the parent port (if it is part
of
the trunk). I think the problem here is that the VM should not try to delete the port as port was existing before VM creation, right?
It is like that if You create port in neutron, and then pass this port to the Nova when booting instance. But if You first create instance by giving network_id to Nova, it will create port on this network for You and that port will be deleted by Nova when instance will be deleted.
You mean that it is possible to later make that VM port (created by nova) a parent port of a trunk? That is not supported in ml2/ovs (or it was not before), I never tried with OVN
Note too that if the parent port is not "eligible" after being part of a trunk, how can you boot a VM with a parent port given that there is a
need
for having the trunk before booting the VM?
Cheers, Luis
On Tue, May 12, 2020 at 4:13 PM Nate Johnston nate.johnston@redhat.com wrote:
Neutron developers,
I am currently working on an issue with trunk ports that has come up a
few
times in my direct experience, and I hope that we can create a long term solution. I am hoping that developers with experience in trunk ports can validate
my
approach here, especially regarding fixing current behavior without introducing an API regression.
By way of introduction to the specifics of the issue, let me blockquote from the LP bug I raised for this [1]:
When you create a trunk in Neutron you create a parent port for thetrunk and attach the trunk to the parent. Then subports can be created on the trunk. When instances are created on the trunk, first a port is created and
then
an instance is associated with a free port. It looks to me that's this is the oversight in the logic.
From the perspective of the code, the parent port looks like anyother
port attached to the trunk bridge. It doesn't have an instance attached
to
it so it looks like it's not being used for anything (which is technically correct). So it becomes an eligible port for an instance to bind to. That is all fine and dandy until you go to delete the instance and you get the "Port [port-id] is currently a parent port for trunk [trunk-id]" exception just as happened here. Anecdotally, it's seems rare that an instance will actually bind to it, but that is what happened for the user in this case and I have had several pings over the past year about people in a similar state.
I propose that when a port is made parent port for a trunk, thatthe
trunk be established as the owner of the port. That way it will be
ineligible
for instances seeking to bind to the port.
Clearly the above behavior indicates buggy issue that should be
rectified
in master and stable branches. Nobody wants a VM that can't be fully
deleted
because the port can't ever be deleted. This is especially egregious
when
it causes heat stack deletion failures.
I am mostly concerned that by adding the trunk as an owner of the
parent
port, then the trunk will need to be deleted before the parent port can be deleted, otherwise a PortInUse error will occur when the port is deleted (i.e.
on
tempest test teardown). That to me seems indicative of an inadvertent API change. Do you think it's all right to say that if you delete a port that is a
parent
port of a trunk, and that trunk has no other subports, that the trunk
deletion
is implicit? Is that the lowest impact to the API that we can incur to resolve this issue?
Your wisdom is appreciated,
Nate
-- LUIS TOM�S BOL�VAR Senior Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.com
-- Slawek Kaplonski Senior software engineer Red Hat
On Thu, 2020-05-14 at 08:45 +0200, Luis Tomas Bolivar wrote:
On Wed, May 13, 2020 at 1:18 PM Slawek Kaplonski skaplons@redhat.com wrote:
Hi,
On Wed, May 13, 2020 at 09:42:28AM +0200, Luis Tomas Bolivar wrote:
Hi Nate,
I think I'm getting configure with the use of "instances" here. With instances, you refer to VMs? Note there is a need to first create the parent, then the trunk and then the instance using that parent.
Also, deleting the VM is not a problem, it will just move the port to
down
(and the trunk). The problem is deleting the parent port (if it is part
of
the trunk). I think the problem here is that the VM should not try to delete the port as port was existing before VM creation, right?
It is like that if You create port in neutron, and then pass this port to the Nova when booting instance. But if You first create instance by giving network_id to Nova, it will create port on this network for You and that port will be deleted by Nova when instance will be deleted.
You mean that it is possible to later make that VM port (created by nova) a parent port of a trunk? That is not supported in ml2/ovs (or it was not before), I never tried with OVN
the behavior should not depend on the backend used.
i.e. form an interoperabltiy point of view there should be no obsverable difference in how the trunk ports api works if you are using ml2/ovs or ml2/ovn.
my recollection was we do not allow a standar port to be used as a parent for a trunk port if its attached to a vm currently. so i think you would have to first detach the port form the vm then make it a trunk port partent and then reattach it to the vm.
i think that workflow is "supported" but only in the sense that it is valid to detach a port and make it a trunk port. if that ortininal port was created by nova in the boot request you should still expect it to get deleted and no clean up of the sub ports to be done. hence my "suported" in scare quotes comment above since realistically its a user error to convert a nova created port into a trunk port parent. at least form a nova point of view we dont support that.
it might technically work but its not inteneded to an any odd behavior as a result is not a bug.
Note too that if the parent port is not "eligible" after being part of a trunk, how can you boot a VM with a parent port given that there is a
need
for having the trunk before booting the VM?
Cheers, Luis
On Tue, May 12, 2020 at 4:13 PM Nate Johnston nate.johnston@redhat.com wrote:
Neutron developers,
I am currently working on an issue with trunk ports that has come up a
few
times in my direct experience, and I hope that we can create a long term solution. I am hoping that developers with experience in trunk ports can validate
my
approach here, especially regarding fixing current behavior without introducing an API regression.
By way of introduction to the specifics of the issue, let me blockquote from the LP bug I raised for this [1]:
When you create a trunk in Neutron you create a parent port for thetrunk and attach the trunk to the parent. Then subports can be created on the trunk. When instances are created on the trunk, first a port is created and
then
an instance is associated with a free port. It looks to me that's this is the oversight in the logic.
From the perspective of the code, the parent port looks like anyother
port attached to the trunk bridge. It doesn't have an instance attached
to
it so it looks like it's not being used for anything (which is technically correct). So it becomes an eligible port for an instance to bind to. That is all fine and dandy until you go to delete the instance and you get the "Port [port-id] is currently a parent port for trunk [trunk-id]" exception just as happened here. Anecdotally, it's seems rare that an instance will actually bind to it, but that is what happened for the user in this case and I have had several pings over the past year about people in a similar state.
I propose that when a port is made parent port for a trunk, thatthe
trunk be established as the owner of the port. That way it will be
ineligible
for instances seeking to bind to the port.
Clearly the above behavior indicates buggy issue that should be
rectified
in master and stable branches. Nobody wants a VM that can't be fully
deleted
because the port can't ever be deleted. This is especially egregious
when
it causes heat stack deletion failures.
I am mostly concerned that by adding the trunk as an owner of the
parent
port, then the trunk will need to be deleted before the parent port can be deleted, otherwise a PortInUse error will occur when the port is deleted (i.e.
on
tempest test teardown). That to me seems indicative of an inadvertent API change. Do you think it's all right to say that if you delete a port that is a
parent
port of a trunk, and that trunk has no other subports, that the trunk
deletion
is implicit? Is that the lowest impact to the API that we can incur to resolve this issue?
Your wisdom is appreciated,
Nate
-- LUIS TOM�S BOL�VAR Senior Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.com
-- Slawek Kaplonski Senior software engineer Red Hat
Hi,
You mean that it is possible to later make that VM port (created by nova) a parent port of a trunk? That is not supported in ml2/ovs (or it was not before), I never tried with OVN
the behavior should not depend on the backend used.
But it does and for good reason.
Ideally we wanted all drivers to allow putting a trunk even on a bound port. But for ovs that would have required unfeasible complexity of re-wiring upgrades (and network downtime while that re-wiring happens).
Even in the api-ref we have error code 409 with reason "A system configuration prevents the operation from succeeding":
https://docs.openstack.org/api-ref/network/v2/?expanded=create-trunk-detail
The gist of my point in the other sub-thread was that an already bound port can be made parent of a trunk with most drivers (except ml2/ovs). But if you do this on a nova-created port (as opposed to a port created manually or by anybody but nova) then nova won't be able to delete that port when deleting the vm.
Cheers, Bence
Hi Nate,
We did have a chat on #neutron yesterday [1], but let me summarize my thoughts here for a broader audience and add some details I tried since:
Let me differentiate three API workflows:
A)
workflow 1)
in order: * create net and subnet for parent port * create parent port (more precisely a port that will become a parent port in the next step) * create trunk with parent port (and optionally subports) * boot vm with parent port (server create --nic port-id=)
side note: Neither the trunk, nor subports should ever be passed to 'server create'. Nova only ever knows about the parent port. Generally nova does not treat this port specially in any way, except for ovs, where (based on what neutron tells nova in port attributes) nova plugs the vif into a trunk bridge instead of br-int.
In this case the parent port existed before nova ever learned about it, therefore nova does not want to delete it when deleting the server.
As with other use cases where a port needs to be somehow special, but we did not want to change nova to support it, this is the canonical way to use trunk ports.
Here you lose the convenience of '--nic net-id='. And you lose some flexibility by not (always) being able to turn a plain old port into a trunk parent.
I think the bug is not present when using this workflow so this is a possible workaround.
workflow 2)
* create net and subnet * boot vm with network (server --nic port-id=) * delete vm
Here no trunk is created at all, so the bug is obviously not present. I just wanted to mention that in this case, nova creates the port for the server and it also wants to delete the port when deleting the server.
Originally I think we considered this as unsupported (for trunks), and that not being a problem since it is only for convenience.
workflow 3)
in order: * create net and subnet * boot vm with net (server create --nic net-id=) * find the port nova created * create trunk with port found as parent (and optionally subports) * delete the vm
This workflow reproduces (part of) the bug. Please see [2] for exact commands. Not all (ml2+trunk) drivers allow trunk creation on an already bound port. Here I used ovn which allows it, but ovs would not.
While we may consider this a bug, and fix it, please note that the error would not occur if a port had been pre-created.
IIRC we considered the ability to turn a bound port into a trunk parent a feature, but we considered the use of '--nic net-id=' only a convenience.
Obviusly we may argue that we lost some flexibility since the user must create this port before boot and before he may have realized he wanted a trunk.
B)
The bug report mentions a "second interface", which never happens even in workflow 3) above. So I suspect there's something else going on beyond workflow 3). I have no proof but a hunch that the API is used in some way that it is not supposed to. Unfortunately the trunk API is not (and could not be made) intuitive and some important error conditions are silent, because of the enormous work they would have entailed. One such situation I see quite frequently is that people try to add subports to a server boot. But that is not needed and should never be done. That is an error but unfortunately with no immediate error message.
But this is just based on a hunch. It still would be nice to know what actual API workflow led to the errors reported in the bug. I'd be happy to review it and incorporate the learnings into our docs.
Cheers, Bence
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neu... [2] https://etherpad.opendev.org/p/lp-1878031
participants (5)
-
Bence Romsics
-
Luis Tomas Bolivar
-
Nate Johnston
-
Sean Mooney
-
Slawek Kaplonski