[neutron][ops] pre-created port remains in use after instance is deleted
Hello again, this is version Caracal, recently upgraded from Antelope. We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error: # openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a) I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried: - openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something? Thanks! Eugen
It looks like this is again policy related, but I don't fully understand yet. I also can't detach a port neither via horizon nor cli, nova-compute reports: 2025-06-03 14:54:24.858 2161878 ERROR nova.network.neutron nova.exception.Forbidden: ((((rule:update_port and rule:update_port:device_id) and rule:update_port:device_owner) and rule:update_port:binding:host_id) and rule:update_port:binding:profile) is disallowed by policy I found this report: https://bugs.launchpad.net/neutron/+bug/2052937 My user is admin only in this project, so I figured I should be allowed to do that, no? So I'm not really sure where I need to fix what, I don't find any of the above rules in the docs. Could anyone point me in the right direction here? Zitat von Eugen Block <eblock@nde.ag>:
Hello again,
this is version Caracal, recently upgraded from Antelope.
We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error:
# openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a)
I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried:
- openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf
and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something?
Thanks! Eugen
On 03/06/2025 15:02, Eugen Block wrote:
It looks like this is again policy related, but I don't fully understand yet. I also can't detach a port neither via horizon nor cli, nova-compute reports:
2025-06-03 14:54:24.858 2161878 ERROR nova.network.neutron nova.exception.Forbidden: ((((rule:update_port and rule:update_port:device_id) and rule:update_port:device_owner) and rule:update_port:binding:host_id) and rule:update_port:binding:profile) is disallowed by policy
I found this report: https://bugs.launchpad.net/neutron/+bug/2052937
My user is admin only in this project, so I figured I should be allowed to do that, no? So I'm not really sure where I need to fix what, I don't find any of the above rules in the docs. Could anyone point me in the right direction here?
Zitat von Eugen Block <eblock@nde.ag>:
Hello again,
this is version Caracal, recently upgraded from Antelope.
We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error:
# openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a)
I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried:
- openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf
and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something?
so detaching port by clering the --device has never been supported by nova. when you pass in a port to nova with --port then nova will not delete it when the vm is deleted but it should be unbound form the previous vm and the device-id and device owner field should be cleared as part of that process. if it is still referencing the previous vms it sound like that is not happening properly or the port was partly detached. is there anything special about how the original vm was deleted. i.e. was the compute agent on the relevant host down?
Thanks! Eugen
Hello o/ This reminds me of an "edge case” I stumbled on recently. In Nova it raises a PortInUse exception in the neutron network driver if the device_id on a port that you try to use for an instance is still set [1]. I proposed —device and —device-id to the `openstack port unset` command [2]. You could try removing/unsetting the device ID on the port. Not sure about the permission issue, one issue we’ve seen when trying to move our service users to not have the admin role is that the service role did not have permissions to create/update the device_id field on ports, that was fixed with [3]. /Tobias [1] https://github.com/openstack/nova/blob/a1c47fc242b6002d2be60fc41176ce29d19d9... [2] https://review.opendev.org/c/openstack/python-openstackclient/+/944130 [3] https://review.opendev.org/c/openstack/neutron/+/947003 On 3 Jun 2025, at 14:10, Eugen Block <eblock@nde.ag> wrote: Hello again, this is version Caracal, recently upgraded from Antelope. We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error: # openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a) I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried: - openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something? Thanks! Eugen
On 03/06/2025 15:37, Tobias Urdin - Binero wrote:
Hello o/
This reminds me of an "edge case” I stumbled on recently.
In Nova it raises a PortInUse exception in the neutron network driver if the device_id on a port that you try to use for an instance is still set [1].
I proposed —device and —device-id to the `openstack port unset` command [2]. You could try removing/unsetting the device ID on the port.
setting or unsetting those field are ment to be resticted to the service role only i.e. not even admin are ment to set those as only nova, zun and ironic should ever set those value.
Not sure about the permission issue, one issue we’ve seen when trying to move our service users to not have the admin role is that the service role did not have permissions to create/update the device_id field on ports, that was fixed with [3].
right so im not sure how much of the transation to using only service user for this has been completed right now i think the nova user need both admin and service ot be able to correctly talk to all service over time nova should be able to stop usign admin and only have serive but that still in progress i believe. nova still requires neutron and cinder to have admin as we have not completed https://review.opendev.org/c/openstack/nova-specs/+/951218 yet.
/Tobias
[1] https://github.com/openstack/nova/blob/a1c47fc242b6002d2be60fc41176ce29d19d9... [2] https://review.opendev.org/c/openstack/python-openstackclient/+/944130 [3] https://review.opendev.org/c/openstack/neutron/+/947003
On 3 Jun 2025, at 14:10, Eugen Block <eblock@nde.ag> wrote:
Hello again,
this is version Caracal, recently upgraded from Antelope.
We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error:
# openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a)
I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried:
- openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf
and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something?
Thanks! Eugen
On 3 Jun 2025, at 17:19, Sean Mooney <smooney@redhat.com> wrote:
On 03/06/2025 15:37, Tobias Urdin - Binero wrote:
Hello o/
This reminds me of an "edge case” I stumbled on recently.
In Nova it raises a PortInUse exception in the neutron network driver if the device_id on a port that you try to use for an instance is still set [1].
I proposed —device and —device-id to the `openstack port unset` command [2]. You could try removing/unsetting the device ID on the port.
setting or unsetting those field are ment to be resticted to the service role only
i.e. not even admin are ment to set those as only nova, zun and ironic should ever set those value.
One would think that, but last time I checked even an end user can meddle with the device owner and device ID fields, which IMO is a design flaw, I was pushing for [1] to restrict that on device_id but it was hard to get consensus on the behaviour change. [1] https://review.opendev.org/c/openstack/neutron/+/861169
Not sure about the permission issue, one issue we’ve seen when trying to move our service users to not have the admin role is that the service role did not have permissions to create/update the device_id field on ports, that was fixed with [3].
right so im not sure how much of the transation to using only service user for this has been completed
right now i think the nova user need both admin and service ot be able to correctly talk to all service
over time nova should be able to stop usign admin and only have serive but that still in progress i believe.
nova still requires neutron and cinder to have admin as we have not completed https://review.opendev.org/c/openstack/nova-specs/+/951218
yet.
/Tobias
[1] https://github.com/openstack/nova/blob/a1c47fc242b6002d2be60fc41176ce29d19d9... [2] https://review.opendev.org/c/openstack/python-openstackclient/+/944130 [3] https://review.opendev.org/c/openstack/neutron/+/947003
On 3 Jun 2025, at 14:10, Eugen Block <eblock@nde.ag> wrote:
Hello again,
this is version Caracal, recently upgraded from Antelope.
We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error:
# openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a)
I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried:
- openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf
and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something?
Thanks! Eugen
On 03/06/2025 18:24, Tobias Urdin - Binero wrote:
On 3 Jun 2025, at 17:19, Sean Mooney <smooney@redhat.com> wrote:
On 03/06/2025 15:37, Tobias Urdin - Binero wrote:
Hello o/
This reminds me of an "edge case” I stumbled on recently.
In Nova it raises a PortInUse exception in the neutron network driver if the device_id on a port that you try to use for an instance is still set [1].
I proposed —device and —device-id to the `openstack port unset` command [2]. You could try removing/unsetting the device ID on the port. setting or unsetting those field are ment to be resticted to the service role only
i.e. not even admin are ment to set those as only nova, zun and ironic should ever set those value. One would think that, but last time I checked even an end user can meddle with the device owner and device ID fields, which IMO is a design flaw, I was pushing for [1] to restrict that on device_id but it was hard to get consensus on the behaviour change.
right so we discssed this as a secuirty hardening opportunity in neutorn a few PTGs ago im kind of lost track fo where that was but we had tentiivly agreed that it should require the service or at least the admin role to set that filed just like admin is required to set the binding:host-id i am not currently a aware of a way to exploit the mutablity of this field howver since nova is not designed for this to every be modifed directly in neutron directly it may actilly be a security bug today not just a hardening opportunity. i had tought it was agreed to update the polic to admin_or_service a few releases ago. the portbining api has been updated https://github.com/openstack/neutron/blob/master/neutron/conf/policies/port_... as part of https://github.com/openstack/neutron/commit/428f7a8418447e75d6a9245dbaf7ccc1... the device owner filed seams to have kept memebr access https://github.com/openstack/neutron/commit/428f7a8418447e75d6a9245dbaf7ccc1... https://github.com/openstack/neutron/blob/21a0588ed89b79f65b73d55ef3ba99cce5... so those seam to still be incorrect. it should have the same policy as binding:host_id https://github.com/openstack/neutron/blob/master/neutron/conf/policies/port.... setable and readable only be admin or service. nova has never supported attaching or detacing a vm via neutron by changing the device-id filed so its invalid for a user to modify that filed if the port is associated with a nova instance. we will not fix any issue related to taht on the nova side unless its a security bug. when neutron is used standalone (with ironic or or zun) and not with nova it may be valid to allow member but even in those cases i think its a strech. i think the device-id would be managed by ironic or zun not the user in those cases.
[1] https://review.opendev.org/c/openstack/neutron/+/861169
Not sure about the permission issue, one issue we’ve seen when trying to move our service users to not have the admin role is that the service role did not have permissions to create/update the device_id field on ports, that was fixed with [3]. right so im not sure how much of the transation to using only service user for this has been completed
right now i think the nova user need both admin and service ot be able to correctly talk to all service
over time nova should be able to stop usign admin and only have serive but that still in progress i believe.
nova still requires neutron and cinder to have admin as we have not completed https://review.opendev.org/c/openstack/nova-specs/+/951218
yet.
/Tobias
[1] https://github.com/openstack/nova/blob/a1c47fc242b6002d2be60fc41176ce29d19d9... [2] https://review.opendev.org/c/openstack/python-openstackclient/+/944130 [3] https://review.opendev.org/c/openstack/neutron/+/947003
On 3 Jun 2025, at 14:10, Eugen Block <eblock@nde.ag> wrote:
Hello again,
this is version Caracal, recently upgraded from Antelope.
We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error:
# openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a)
I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried:
- openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf
and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something?
Thanks! Eugen
Thanks for your responses, I appreciate it. So I did unset the device_id from the port, but that's not enough, the port is still reported as in use. The VM is deleted either via dashboard or via CLI, doesn't make a difference. It's a tiny cirros VM (which probably doesn't really matter). The nova user has admin and service roles: root@controller01:~# openstack role assignment list --names | grep nova | admin | nova@Default | | SERVICE@Default | | | False | | service | nova@Default | | SERVICE@Default | | | False | I'm currently not able to identify where that message exactly comes from since there are multiple places where this exception can come from. I don't see too many references for that port in the various databases/tables, nothing stands out who is using it. As I already mentioned before, I did clear more entries like host binding etc. And I don't expect nova to delete a pre-created port, I just want to be able to re-use it again (which worked until the Caracal upgrade, I believe). So the question remains, what is missing here? Zitat von Sean Mooney <smooney@redhat.com>:
On 03/06/2025 18:24, Tobias Urdin - Binero wrote:
On 3 Jun 2025, at 17:19, Sean Mooney <smooney@redhat.com> wrote:
On 03/06/2025 15:37, Tobias Urdin - Binero wrote:
Hello o/
This reminds me of an "edge case” I stumbled on recently.
In Nova it raises a PortInUse exception in the neutron network driver if the device_id on a port that you try to use for an instance is still set [1].
I proposed —device and —device-id to the `openstack port unset` command [2]. You could try removing/unsetting the device ID on the port. setting or unsetting those field are ment to be resticted to the service role only
i.e. not even admin are ment to set those as only nova, zun and ironic should ever set those value. One would think that, but last time I checked even an end user can meddle with the device owner and device ID fields, which IMO is a design flaw, I was pushing for [1] to restrict that on device_id but it was hard to get consensus on the behaviour change.
right so we discssed this as a secuirty hardening opportunity in neutorn a few PTGs ago im kind of lost track fo where that was but we had tentiivly agreed that it should require the service or at least the admin role to set that filed just like admin is required to set the binding:host-id
i am not currently a aware of a way to exploit the mutablity of this field howver since nova is not designed for this to every be modifed directly in neutron directly it may actilly be a security bug today not just a hardening opportunity.
i had tought it was agreed to update the polic to admin_or_service a few releases ago.
the portbining api has been updated https://github.com/openstack/neutron/blob/master/neutron/conf/policies/port_...
as part of https://github.com/openstack/neutron/commit/428f7a8418447e75d6a9245dbaf7ccc1...
the device owner filed seams to have kept memebr access
https://github.com/openstack/neutron/commit/428f7a8418447e75d6a9245dbaf7ccc1...
https://github.com/openstack/neutron/blob/21a0588ed89b79f65b73d55ef3ba99cce5...
so those seam to still be incorrect.
it should have the same policy as binding:host_id
https://github.com/openstack/neutron/blob/master/neutron/conf/policies/port....
setable and readable only be admin or service.
nova has never supported attaching or detacing a vm via neutron by changing the device-id filed so its invalid for a user to modify that filed if the port is associated with a nova instance. we will not fix any issue related to taht on the nova side unless its a security bug. when neutron is used standalone (with ironic or or zun) and not with nova it may be valid to allow member but even in those cases i think its a strech.
i think the device-id would be managed by ironic or zun not the user in those cases.
[1] https://review.opendev.org/c/openstack/neutron/+/861169
Not sure about the permission issue, one issue we’ve seen when trying to move our service users to not have the admin role is that the service role did not have permissions to create/update the device_id field on ports, that was fixed with [3]. right so im not sure how much of the transation to using only service user for this has been completed
right now i think the nova user need both admin and service ot be able to correctly talk to all service
over time nova should be able to stop usign admin and only have serive but that still in progress i believe.
nova still requires neutron and cinder to have admin as we have not completed https://review.opendev.org/c/openstack/nova-specs/+/951218
yet.
/Tobias
[1] https://github.com/openstack/nova/blob/a1c47fc242b6002d2be60fc41176ce29d19d9... [2] https://review.opendev.org/c/openstack/python-openstackclient/+/944130 [3] https://review.opendev.org/c/openstack/neutron/+/947003
On 3 Jun 2025, at 14:10, Eugen Block <eblock@nde.ag> wrote:
Hello again,
this is version Caracal, recently upgraded from Antelope.
We're currently facing an issue with pre-created ports. This used to work just fine for years, now we have to delete the port, recreate it and try again. For example, we launch an instance with a pre-created port, notice that something is wrong and delete the instance. Previously, we could just launch a new instance and use the same port-id, but now we get this error:
# openstack server create --image <image> --flavor 1 --port d10298ea-6dd9-40af-af7e-507f4d83a8cf test-port4 Port d10298ea-6dd9-40af-af7e-507f4d83a8cf is still in use. (HTTP 409) (Request-ID: req-e6cef8b4-d8af-4312-852b-d44ec92cf27a)
I noticed that the device_id was still pointing to the previous instance, so I tried a couple of things to clear the port state, but to no avail. I tried:
- openstack port set --device None d10298ea-6dd9-40af-af7e-507f4d83a8cf - openstack port set --device "" d10298ea-6dd9-40af-af7e-507f4d83a8cf
and a couple of more set commands. I also don't see any option for port creation to allow reusing a pre-created port. Am I missing something?
Thanks! Eugen
participants (3)
-
Eugen Block
-
Sean Mooney
-
Tobias Urdin - Binero