I didn't see a response to this email. Trying again with a better subject: On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden <ozzzo@yahoo.com> wrote: One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface: 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device) For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

Hi, maybe some more history and details could help understand what might be the issue here. So you have VMs with one or more interfaces that you try to delete. Do you see those interfaces in 'nova interface-list {UUID}'? And while deleting the VMs, ovs takes a long time becaus it can't find those devices, correct? This reminds me of our reinstallation a few months ago as well (importing the previous DB dump). After migrating from openSUSE to Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps the 'nova-manage db online_data_migrations' failed because of a few instances with a weird port state. A user attached a new interface according to the event list, but we don't see a detach in the logs although the instances had only one remaining NIC (as before). We had to dig really deep, including manipulating the DB (mark that interface as deleted). We didn't try to delete those instances in that weird state, so I'm not sure if we could have ended up in the same situation as you. But maybe you have a similar thing going on with those undeleted interfaces in the DB that OVS tries to delete? Hard to tell... Regards, Eugen Zitat von Albert Braden <ozzzo@yahoo.com>:

I didn't see a response to this email. Trying again with a better subject: On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden <ozzzo@yahoo.com> wrote:

One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)

For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

Hi, the short answer is, I don't know. It really depends on what exactly the issue is here. It requires some investigation to be able to tell if any openstack tools can help, e. g. neutron related commands, or if you need to tackle this from ovs directly, or if it's a database inconsistency or maybe something else entirely. For example, what I referred to was an interface that wasn't attached to an instance anymore, and the port didn't exist anymore, but the database had not been properly updated. I didn't see any other way than to manually update the DB. But you might be facing something different here, maybe someone else has experienced the same as you, but since nobody else responded yet, you'll have to dig on your own. Zitat von Albert Braden <ozzzo@yahoo.com>:

Hi Eugen,

Thanks for your email. I'm not aware of a VM that corresponds to the stuck interface. Every VM deletion on the affected hypervisor is delayed; this has been happening for a few weeks now. Looking at the logs, it appears that an interface is created when a VM is created, and deleted with that VM. I'm guessing that the interface may have become stuck when a customer deleted a VM during a maintenance, and the interface failed to delete. Is it possible to delete an interface without editing the database? On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block <eblock@nde.ag> wrote:

Hi,

maybe some more history and details could help understand what might  be the issue here. So you have VMs with one or more interfaces that you try to delete. Do  you see those interfaces in 'nova interface-list {UUID}'? And while  deleting the VMs, ovs takes a long time becaus it can't find those  devices, correct?

This reminds me of our reinstallation a few months ago as well  (importing the previous DB dump). After migrating from openSUSE to  Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps  the 'nova-manage db online_data_migrations' failed because of a few  instances with a weird port state. A user attached a new interface  according to the event list, but we don't see a detach in the logs  although the instances had only one remaining NIC (as before). We had  to dig really deep, including manipulating the DB (mark that interface  as deleted). We didn't try to delete those instances in that weird state, so I'm  not sure if we could have ended up in the same situation as you. But  maybe you have a similar thing going on with those undeleted  interfaces in the DB that OVS tries to delete? Hard to tell...

Regards, Eugen

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I didn't see a response to this email. Trying again with a better subject:       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden  <ozzzo@yahoo.com> wrote:

  One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)

For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

You can see the current interfaces with: nova interface-list {UUID} Zitat von Albert Braden <ozzzo@yahoo.com>:

I poked around in the Neutron documentation, but I can't find the commands interfaces. I looked at the Neutron API but the only "interface" commands there relate to routers. Is there an Openstack tool that will let me view or delete interfaces? On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block <eblock@nde.ag> wrote:

Hi,

the short answer is, I don't know. It really depends on what exactly  the issue is here. It requires some investigation to be able to tell  if any openstack tools can help, e. g. neutron related commands, or if  you need to tackle this from ovs directly, or if it's a database  inconsistency or maybe something else entirely. For example, what I referred to was an interface that wasn't attached  to an instance anymore, and the port didn't exist anymore, but the  database had not been properly updated. I didn't see any other way  than to manually update the DB. But you might be facing something  different here, maybe someone else has experienced the same as you,  but since nobody else responded yet, you'll have to dig on your own.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

Hi Eugen,

Thanks for your email. I'm not aware of a VM that corresponds to the  stuck interface. Every VM deletion on the affected hypervisor is  delayed; this has been happening for a few weeks now. Looking at the  logs, it appears that an interface is created when a VM is created,  and deleted with that VM. I'm guessing that the interface may have  become stuck when a customer deleted a VM during a maintenance, and  the interface failed to delete. Is it possible to delete an  interface without editing the database?       On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

maybe some more history and details could help understand what might  be the issue here. So you have VMs with one or more interfaces that you try to delete. Do  you see those interfaces in 'nova interface-list {UUID}'? And while  deleting the VMs, ovs takes a long time becaus it can't find those  devices, correct?

This reminds me of our reinstallation a few months ago as well  (importing the previous DB dump). After migrating from openSUSE to  Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps  the 'nova-manage db online_data_migrations' failed because of a few  instances with a weird port state. A user attached a new interface  according to the event list, but we don't see a detach in the logs  although the instances had only one remaining NIC (as before). We had  to dig really deep, including manipulating the DB (mark that interface  as deleted). We didn't try to delete those instances in that weird state, so I'm  not sure if we could have ended up in the same situation as you. But  maybe you have a similar thing going on with those undeleted  interfaces in the DB that OVS tries to delete? Hard to tell...

Regards, Eugen

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I didn't see a response to this email. Trying again with a better subject:       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden  <ozzzo@yahoo.com> wrote:

  One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)

For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

Basically, it's a port, so you could try to show|delete|modify it: openstack port list | grep {whatever} openstack port show|delete|set {PORT_UUID} Zitat von Albert Braden <ozzzo@yahoo.com>:

I apologize for the vague question. This particular interface is orphaned. It looks like the VM may have been deleted during a maintenance and the interface failed to delete. The symptom is that every time I delete a VM, on the hypervisor with the orphaned interface, it takes substantially longer than other hypervisors, so I'm guessing that the orphaned interface is confusing Neutron.

Is is possible to perform operations on an interface, without reference to a VM? I'd like to delete this orphaned interface, hopefully without manually editing the DB. On Monday, January 13, 2025 at 01:54:19 PM EST, Eugen Block <eblock@nde.ag> wrote:

You can see the current interfaces with:

nova interface-list {UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I poked around in the Neutron documentation, but I can't find the  commands interfaces. I looked at the Neutron API but the only  "interface" commands there relate to routers. Is there an Openstack  tool that will let me view or delete interfaces?       On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

the short answer is, I don't know. It really depends on what exactly  the issue is here. It requires some investigation to be able to tell  if any openstack tools can help, e. g. neutron related commands, or if  you need to tackle this from ovs directly, or if it's a database  inconsistency or maybe something else entirely. For example, what I referred to was an interface that wasn't attached  to an instance anymore, and the port didn't exist anymore, but the  database had not been properly updated. I didn't see any other way  than to manually update the DB. But you might be facing something  different here, maybe someone else has experienced the same as you,  but since nobody else responded yet, you'll have to dig on your own.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

Hi Eugen,

Thanks for your email. I'm not aware of a VM that corresponds to the  stuck interface. Every VM deletion on the affected hypervisor is  delayed; this has been happening for a few weeks now. Looking at the  logs, it appears that an interface is created when a VM is created,  and deleted with that VM. I'm guessing that the interface may have  become stuck when a customer deleted a VM during a maintenance, and  the interface failed to delete. Is it possible to delete an  interface without editing the database?       On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

maybe some more history and details could help understand what might  be the issue here. So you have VMs with one or more interfaces that you try to delete. Do  you see those interfaces in 'nova interface-list {UUID}'? And while  deleting the VMs, ovs takes a long time becaus it can't find those  devices, correct?

This reminds me of our reinstallation a few months ago as well  (importing the previous DB dump). After migrating from openSUSE to  Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps  the 'nova-manage db online_data_migrations' failed because of a few  instances with a weird port state. A user attached a new interface  according to the event list, but we don't see a detach in the logs  although the instances had only one remaining NIC (as before). We had  to dig really deep, including manipulating the DB (mark that interface  as deleted). We didn't try to delete those instances in that weird state, so I'm  not sure if we could have ended up in the same situation as you. But  maybe you have a similar thing going on with those undeleted  interfaces in the DB that OVS tries to delete? Hard to tell...

Regards, Eugen

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I didn't see a response to this email. Trying again with a better subject:       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden  <ozzzo@yahoo.com> wrote:

  One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)

For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

Can you check this command: openstack port list | grep 0430a2f7 This could help identify the port, but only if it's really still present in neutron. Zitat von Albert Braden <ozzzo@yahoo.com>:

This is what I see in the log:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)

When I do "openstack port list --long" I get ID, Name, MAC, IPs, Status, Security Groups, Owner and Tags, but I don't see anything that matches qvo0430a2f7-cd. How do I translate qvo0430a2f7-cd to a port UUID? On Tuesday, January 14, 2025 at 01:30:43 AM EST, Eugen Block <eblock@nde.ag> wrote:

Basically, it's a port, so you could try to show|delete|modify it:

openstack port list | grep {whatever} openstack port show|delete|set {PORT_UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I apologize for the vague question. This particular interface is  orphaned. It looks like the VM may have been deleted during a  maintenance and the interface failed to delete. The symptom is that  every time I delete a VM, on the hypervisor with the orphaned  interface, it takes substantially longer than other hypervisors, so  I'm guessing that the orphaned interface is confusing Neutron.

Is is possible to perform operations on an interface, without  reference to a VM? I'd like to delete this orphaned interface,  hopefully without manually editing the DB.       On Monday, January 13, 2025 at 01:54:19 PM EST, Eugen Block  <eblock@nde.ag> wrote:

  You can see the current interfaces with:

nova interface-list {UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I poked around in the Neutron documentation, but I can't find the  commands interfaces. I looked at the Neutron API but the only  "interface" commands there relate to routers. Is there an Openstack  tool that will let me view or delete interfaces?       On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

the short answer is, I don't know. It really depends on what exactly  the issue is here. It requires some investigation to be able to tell  if any openstack tools can help, e. g. neutron related commands, or if  you need to tackle this from ovs directly, or if it's a database  inconsistency or maybe something else entirely. For example, what I referred to was an interface that wasn't attached  to an instance anymore, and the port didn't exist anymore, but the  database had not been properly updated. I didn't see any other way  than to manually update the DB. But you might be facing something  different here, maybe someone else has experienced the same as you,  but since nobody else responded yet, you'll have to dig on your own.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

Hi Eugen,

Thanks for your email. I'm not aware of a VM that corresponds to the  stuck interface. Every VM deletion on the affected hypervisor is  delayed; this has been happening for a few weeks now. Looking at the  logs, it appears that an interface is created when a VM is created,  and deleted with that VM. I'm guessing that the interface may have  become stuck when a customer deleted a VM during a maintenance, and  the interface failed to delete. Is it possible to delete an  interface without editing the database?       On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

maybe some more history and details could help understand what might  be the issue here. So you have VMs with one or more interfaces that you try to delete. Do  you see those interfaces in 'nova interface-list {UUID}'? And while  deleting the VMs, ovs takes a long time becaus it can't find those  devices, correct?

This reminds me of our reinstallation a few months ago as well  (importing the previous DB dump). After migrating from openSUSE to  Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps  the 'nova-manage db online_data_migrations' failed because of a few  instances with a weird port state. A user attached a new interface  according to the event list, but we don't see a detach in the logs  although the instances had only one remaining NIC (as before). We had  to dig really deep, including manipulating the DB (mark that interface  as deleted). We didn't try to delete those instances in that weird state, so I'm  not sure if we could have ended up in the same situation as you. But  maybe you have a similar thing going on with those undeleted  interfaces in the DB that OVS tries to delete? Hard to tell...

Regards, Eugen

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I didn't see a response to this email. Trying again with a better subject:       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden  <ozzzo@yahoo.com> wrote:

  One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)

For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

Yeah, that matches the log output (no such device). The port mapping is in this file on the hypervisor (not sure where else): /var/lib/openvswitch/conf.db Do you see your "orphaned" port there? But don't change anything (yet). I'm not sure which tool to use for cleanup yet. Zitat von Albert Braden <ozzzo@yahoo.com>:

No output. It looks like that port may not exist anymore. On Tuesday, January 14, 2025 at 09:02:49 AM EST, Eugen Block <eblock@nde.ag> wrote:

Can you check this command:

openstack port list | grep 0430a2f7

This could help identify the port, but only if it's really still  present in neutron.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

This is what I see in the log:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network  device qvo0430a2f7-cd (No such device)

When I do "openstack port list --long" I get ID, Name, MAC, IPs,  Status, Security Groups, Owner and Tags, but I don't see anything  that matches qvo0430a2f7-cd. How do I translate qvo0430a2f7-cd to a  port UUID?    On Tuesday, January 14, 2025 at 01:30:43 AM EST,  Eugen Block <eblock@nde.ag> wrote:

  Basically, it's a port, so you could try to show|delete|modify it:

openstack port list | grep {whatever} openstack port show|delete|set {PORT_UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I apologize for the vague question. This particular interface is  orphaned. It looks like the VM may have been deleted during a  maintenance and the interface failed to delete. The symptom is that  every time I delete a VM, on the hypervisor with the orphaned  interface, it takes substantially longer than other hypervisors, so  I'm guessing that the orphaned interface is confusing Neutron.

Is is possible to perform operations on an interface, without  reference to a VM? I'd like to delete this orphaned interface,  hopefully without manually editing the DB.       On Monday, January 13, 2025 at 01:54:19 PM EST, Eugen Block  <eblock@nde.ag> wrote:

  You can see the current interfaces with:

nova interface-list {UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I poked around in the Neutron documentation, but I can't find the  commands interfaces. I looked at the Neutron API but the only  "interface" commands there relate to routers. Is there an Openstack  tool that will let me view or delete interfaces?       On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

the short answer is, I don't know. It really depends on what exactly  the issue is here. It requires some investigation to be able to tell  if any openstack tools can help, e. g. neutron related commands, or if  you need to tackle this from ovs directly, or if it's a database  inconsistency or maybe something else entirely. For example, what I referred to was an interface that wasn't attached  to an instance anymore, and the port didn't exist anymore, but the  database had not been properly updated. I didn't see any other way  than to manually update the DB. But you might be facing something  different here, maybe someone else has experienced the same as you,  but since nobody else responded yet, you'll have to dig on your own.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

Hi Eugen,

Thanks for your email. I'm not aware of a VM that corresponds to the  stuck interface. Every VM deletion on the affected hypervisor is  delayed; this has been happening for a few weeks now. Looking at the  logs, it appears that an interface is created when a VM is created,  and deleted with that VM. I'm guessing that the interface may have  become stuck when a customer deleted a VM during a maintenance, and  the interface failed to delete. Is it possible to delete an  interface without editing the database?       On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

maybe some more history and details could help understand what might  be the issue here. So you have VMs with one or more interfaces that you try to delete. Do  you see those interfaces in 'nova interface-list {UUID}'? And while  deleting the VMs, ovs takes a long time becaus it can't find those  devices, correct?

This reminds me of our reinstallation a few months ago as well  (importing the previous DB dump). After migrating from openSUSE to  Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps  the 'nova-manage db online_data_migrations' failed because of a few  instances with a weird port state. A user attached a new interface  according to the event list, but we don't see a detach in the logs  although the instances had only one remaining NIC (as before). We had  to dig really deep, including manipulating the DB (mark that interface  as deleted). We didn't try to delete those instances in that weird state, so I'm  not sure if we could have ended up in the same situation as you. But  maybe you have a similar thing going on with those undeleted  interfaces in the DB that OVS tries to delete? Hard to tell...

Regards, Eugen

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I didn't see a response to this email. Trying again with a  better subject:       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden  <ozzzo@yahoo.com> wrote:

  One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)

For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

That seems to have fixed the problem. Thanks for helping me figure that out! On Tuesday, January 14, 2025 at 09:43:22 AM EST, Eugen Block <eblock@nde.ag> wrote: If the orphaned port is still in the conf.db, you could try to remove  it with (assuming the corresponding bridge is br-int): root@compute002:~# ovs-vsctl del-port br-int qvo0430a2f7-cd To confirm that it is the br-int, you can run: root@compute002:~# ovs-vsctl iface-to-br qvo0430a2f7-cd br-int You can display the db contents a little more structured with this command: root@compute002:~# ovsdb-client dump Zitat von Eugen Block <eblock@nde.ag>:

Yeah, that matches the log output (no such device). The port mapping  is in this file on the hypervisor (not sure where else):

/var/lib/openvswitch/conf.db

Do you see your "orphaned" port there? But don't change anything  (yet). I'm not sure which tool to use for cleanup yet.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

No output. It looks like that port may not exist anymore.     On Tuesday, January 14, 2025 at 09:02:49 AM EST, Eugen Block  <eblock@nde.ag> wrote:

Can you check this command:

openstack port list | grep 0430a2f7

This could help identify the port, but only if it's really still  present in neutron.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

This is what I see in the log:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network  device qvo0430a2f7-cd (No such device)

When I do "openstack port list --long" I get ID, Name, MAC, IPs,  Status, Security Groups, Owner and Tags, but I don't see anything  that matches qvo0430a2f7-cd. How do I translate qvo0430a2f7-cd to a  port UUID?    On Tuesday, January 14, 2025 at 01:30:43 AM EST,  Eugen Block <eblock@nde.ag> wrote:

  Basically, it's a port, so you could try to show|delete|modify it:

openstack port list | grep {whatever} openstack port show|delete|set {PORT_UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I apologize for the vague question. This particular interface is  orphaned. It looks like the VM may have been deleted during a  maintenance and the interface failed to delete. The symptom is that  every time I delete a VM, on the hypervisor with the orphaned  interface, it takes substantially longer than other hypervisors, so  I'm guessing that the orphaned interface is confusing Neutron.

Is is possible to perform operations on an interface, without  reference to a VM? I'd like to delete this orphaned interface,  hopefully without manually editing the DB.       On Monday, January 13, 2025 at 01:54:19 PM EST, Eugen Block  <eblock@nde.ag> wrote:

  You can see the current interfaces with:

nova interface-list {UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I poked around in the Neutron documentation, but I can't find the  commands interfaces. I looked at the Neutron API but the only  "interface" commands there relate to routers. Is there an Openstack  tool that will let me view or delete interfaces?       On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

the short answer is, I don't know. It really depends on what exactly  the issue is here. It requires some investigation to be able to tell  if any openstack tools can help, e. g. neutron related commands, or if  you need to tackle this from ovs directly, or if it's a database  inconsistency or maybe something else entirely. For example, what I referred to was an interface that wasn't attached  to an instance anymore, and the port didn't exist anymore, but the  database had not been properly updated. I didn't see any other way  than to manually update the DB. But you might be facing something  different here, maybe someone else has experienced the same as you,  but since nobody else responded yet, you'll have to dig on your own.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

Hi Eugen,

Thanks for your email. I'm not aware of a VM that corresponds to the  stuck interface. Every VM deletion on the affected hypervisor is  delayed; this has been happening for a few weeks now. Looking at the  logs, it appears that an interface is created when a VM is created,  and deleted with that VM. I'm guessing that the interface may have  become stuck when a customer deleted a VM during a maintenance, and  the interface failed to delete. Is it possible to delete an  interface without editing the database?       On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

maybe some more history and details could help understand what might  be the issue here. So you have VMs with one or more interfaces that you try to delete. Do  you see those interfaces in 'nova interface-list {UUID}'? And while  deleting the VMs, ovs takes a long time becaus it can't find those  devices, correct?

This reminds me of our reinstallation a few months ago as well  (importing the previous DB dump). After migrating from openSUSE to  Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps  the 'nova-manage db online_data_migrations' failed because of a few  instances with a weird port state. A user attached a new interface  according to the event list, but we don't see a detach in the logs  although the instances had only one remaining NIC (as before). We had  to dig really deep, including manipulating the DB (mark that interface  as deleted). We didn't try to delete those instances in that weird state, so I'm  not sure if we could have ended up in the same situation as you. But  maybe you have a similar thing going on with those undeleted  interfaces in the DB that OVS tries to delete? Hard to tell...

Regards, Eugen

Zitat von Albert Braden <ozzzo@yahoo.com>:

> I didn't see a response to this email. Trying again with a  > better subject: >       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden  > <ozzzo@yahoo.com> wrote: > >   > One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface: > > 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device) > > For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

Yes, deletion time went back to normal. On Tuesday, January 14, 2025 at 11:19:13 AM EST, Eugen Block <eblock@nde.ag> wrote: Awesome! So deleting VMs now progresses without any delays? Zitat von Albert Braden <ozzzo@yahoo.com>:

That seems to have fixed the problem. Thanks for helping me figure that out!       On Tuesday, January 14, 2025 at 09:43:22 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  If the orphaned port is still in the conf.db, you could try to remove  it with (assuming the corresponding bridge is br-int):

root@compute002:~# ovs-vsctl del-port br-int qvo0430a2f7-cd

To confirm that it is the br-int, you can run:

root@compute002:~# ovs-vsctl iface-to-br qvo0430a2f7-cd br-int

You can display the db contents a little more structured with this command:

root@compute002:~# ovsdb-client dump

Zitat von Eugen Block <eblock@nde.ag>:

...

Yeah, that matches the log output (no such device). The port mapping  is in this file on the hypervisor (not sure where else):

/var/lib/openvswitch/conf.db

Do you see your "orphaned" port there? But don't change anything  (yet). I'm not sure which tool to use for cleanup yet.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

No output. It looks like that port may not exist anymore.     On Tuesday, January 14, 2025 at 09:02:49 AM EST, Eugen Block  <eblock@nde.ag> wrote:

Can you check this command:

openstack port list | grep 0430a2f7

This could help identify the port, but only if it's really still  present in neutron.

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

This is what I see in the log:

2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network  device qvo0430a2f7-cd (No such device)

When I do "openstack port list --long" I get ID, Name, MAC, IPs,  Status, Security Groups, Owner and Tags, but I don't see anything  that matches qvo0430a2f7-cd. How do I translate qvo0430a2f7-cd to a  port UUID?    On Tuesday, January 14, 2025 at 01:30:43 AM EST,  Eugen Block <eblock@nde.ag> wrote:

  Basically, it's a port, so you could try to show|delete|modify it:

openstack port list | grep {whatever} openstack port show|delete|set {PORT_UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I apologize for the vague question. This particular interface is  orphaned. It looks like the VM may have been deleted during a  maintenance and the interface failed to delete. The symptom is that  every time I delete a VM, on the hypervisor with the orphaned  interface, it takes substantially longer than other hypervisors, so  I'm guessing that the orphaned interface is confusing Neutron.

Is is possible to perform operations on an interface, without  reference to a VM? I'd like to delete this orphaned interface,  hopefully without manually editing the DB.       On Monday, January 13, 2025 at 01:54:19 PM EST, Eugen Block  <eblock@nde.ag> wrote:

  You can see the current interfaces with:

nova interface-list {UUID}

Zitat von Albert Braden <ozzzo@yahoo.com>:

...

I poked around in the Neutron documentation, but I can't find the  commands interfaces. I looked at the Neutron API but the only  "interface" commands there relate to routers. Is there an Openstack  tool that will let me view or delete interfaces?       On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block  <eblock@nde.ag> wrote:

  Hi,

the short answer is, I don't know. It really depends on what exactly  the issue is here. It requires some investigation to be able to tell  if any openstack tools can help, e. g. neutron related commands, or if  you need to tackle this from ovs directly, or if it's a database  inconsistency or maybe something else entirely. For example, what I referred to was an interface that wasn't attached  to an instance anymore, and the port didn't exist anymore, but the  database had not been properly updated. I didn't see any other way  than to manually update the DB. But you might be facing something  different here, maybe someone else has experienced the same as you,  but since nobody else responded yet, you'll have to dig on your own.

Zitat von Albert Braden <ozzzo@yahoo.com>:

> Hi Eugen, > > Thanks for your email. I'm not aware of a VM that corresponds to the  > stuck interface. Every VM deletion on the affected hypervisor is  > delayed; this has been happening for a few weeks now. Looking at the  > logs, it appears that an interface is created when a VM is created,  > and deleted with that VM. I'm guessing that the interface may have  > become stuck when a customer deleted a VM during a maintenance, and  > the interface failed to delete. Is it possible to delete an  > interface without editing the database? >       On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block  > <eblock@nde.ag> wrote: > >   Hi, > > maybe some more history and details could help understand what might  > be the issue here. > So you have VMs with one or more interfaces that you try to delete. Do  > you see those interfaces in 'nova interface-list {UUID}'? And while  > deleting the VMs, ovs takes a long time becaus it can't find those  > devices, correct? > > This reminds me of our reinstallation a few months ago as well  > (importing the previous DB dump). After migrating from openSUSE to  > Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps  > the 'nova-manage db online_data_migrations' failed because of a few  > instances with a weird port state. A user attached a new interface  > according to the event list, but we don't see a detach in the logs  > although the instances had only one remaining NIC (as before). We had  > to dig really deep, including manipulating the DB (mark that interface  > as deleted). > We didn't try to delete those instances in that weird state, so I'm  > not sure if we could have ended up in the same situation as you. But  > maybe you have a similar thing going on with those undeleted  > interfaces in the DB that OVS tries to delete? Hard to tell... > > Regards, > Eugen > > Zitat von Albert Braden <ozzzo@yahoo.com>: > >> I didn't see a response to this email. Trying again with a  >> better subject: >>       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden  >> <ozzzo@yahoo.com> wrote: >> >>   >> One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface: >> >> 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device) >> >> For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?