Zitat von Eugen Block <eblock@nde.ag>:

> Yeah, that matches the log output (no such device). The port mapping 
> is in this file on the hypervisor (not sure where else):
>
> /var/lib/openvswitch/conf.db
>
> Do you see your "orphaned" port there? But don't change anything 
> (yet). I'm not sure which tool to use for cleanup yet.
>
> Zitat von Albert Braden <ozzzo@yahoo.com>:
>
>> No output. It looks like that port may not exist anymore.
>>    On Tuesday, January 14, 2025 at 09:02:49 AM EST, Eugen Block 
>> <eblock@nde.ag> wrote:
>>
>> Can you check this command:
>>
>> openstack port list | grep 0430a2f7
>>
>> This could help identify the port, but only if it's really still 
>> present in neutron.
>>
>> Zitat von Albert Braden <ozzzo@yahoo.com>:
>>
>>> This is what I see in the log:
>>>
>>> 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network 
>>> device qvo0430a2f7-cd (No such device)
>>>
>>> When I do "openstack port list --long" I get ID, Name, MAC, IPs, 
>>> Status, Security Groups, Owner and Tags, but I don't see anything 
>>> that matches qvo0430a2f7-cd. How do I translate qvo0430a2f7-cd to a 
>>> port UUID?    On Tuesday, January 14, 2025 at 01:30:43 AM EST, 
>>> Eugen Block <eblock@nde.ag> wrote:
>>>
>>>   Basically, it's a port, so you could try to show|delete|modify it:
>>>
>>> openstack port list | grep {whatever}
>>> openstack port show|delete|set {PORT_UUID}
>>>
>>> Zitat von Albert Braden <ozzzo@yahoo.com>:
>>>
>>>> I apologize for the vague question. This particular interface is 
>>>> orphaned. It looks like the VM may have been deleted during a 
>>>> maintenance and the interface failed to delete. The symptom is that 
>>>> every time I delete a VM, on the hypervisor with the orphaned 
>>>> interface, it takes substantially longer than other hypervisors, so 
>>>> I'm guessing that the orphaned interface is confusing Neutron.
>>>>
>>>> Is is possible to perform operations on an interface, without 
>>>> reference to a VM? I'd like to delete this orphaned interface, 
>>>> hopefully without manually editing the DB.
>>>>       On Monday, January 13, 2025 at 01:54:19 PM EST, Eugen Block 
>>>> <eblock@nde.ag> wrote:
>>>>
>>>>   You can see the current interfaces with:
>>>>
>>>> nova interface-list {UUID}
>>>>
>>>> Zitat von Albert Braden <ozzzo@yahoo.com>:
>>>>
>>>>> I poked around in the Neutron documentation, but I can't find the 
>>>>> commands interfaces. I looked at the Neutron API but the only 
>>>>> "interface" commands there relate to routers. Is there an Openstack 
>>>>> tool that will let me view or delete interfaces?
>>>>>       On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block 
>>>>> <eblock@nde.ag> wrote:
>>>>>
>>>>>   Hi,
>>>>>
>>>>> the short answer is, I don't know. It really depends on what exactly 
>>>>> the issue is here. It requires some investigation to be able to tell 
>>>>> if any openstack tools can help, e. g. neutron related commands, or if 
>>>>> you need to tackle this from ovs directly, or if it's a database 
>>>>> inconsistency or maybe something else entirely.
>>>>> For example, what I referred to was an interface that wasn't attached 
>>>>> to an instance anymore, and the port didn't exist anymore, but the 
>>>>> database had not been properly updated. I didn't see any other way 
>>>>> than to manually update the DB. But you might be facing something 
>>>>> different here, maybe someone else has experienced the same as you, 
>>>>> but since nobody else responded yet, you'll have to dig on your own.
>>>>>
>>>>> Zitat von Albert Braden <ozzzo@yahoo.com>:
>>>>>
>>>>>> Hi Eugen,
>>>>>>
>>>>>> Thanks for your email. I'm not aware of a VM that corresponds to the 
>>>>>> stuck interface. Every VM deletion on the affected hypervisor is 
>>>>>> delayed; this has been happening for a few weeks now. Looking at the 
>>>>>> logs, it appears that an interface is created when a VM is created, 
>>>>>> and deleted with that VM. I'm guessing that the interface may have 
>>>>>> become stuck when a customer deleted a VM during a maintenance, and 
>>>>>> the interface failed to delete. Is it possible to delete an 
>>>>>> interface without editing the database?
>>>>>>       On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block 
>>>>>> <eblock@nde.ag> wrote:
>>>>>>
>>>>>>   Hi,
>>>>>>
>>>>>> maybe some more history and details could help understand what might 
>>>>>> be the issue here.
>>>>>> So you have VMs with one or more interfaces that you try to delete. Do 
>>>>>> you see those interfaces in 'nova interface-list {UUID}'? And while 
>>>>>> deleting the VMs, ovs takes a long time becaus it can't find those 
>>>>>> devices, correct?
>>>>>>
>>>>>> This reminds me of our reinstallation a few months ago as well 
>>>>>> (importing the previous DB dump). After migrating from openSUSE to 
>>>>>> Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps 
>>>>>> the 'nova-manage db online_data_migrations' failed because of a few 
>>>>>> instances with a weird port state. A user attached a new interface 
>>>>>> according to the event list, but we don't see a detach in the logs 
>>>>>> although the instances had only one remaining NIC (as before). We had 
>>>>>> to dig really deep, including manipulating the DB (mark that interface 
>>>>>> as deleted).
>>>>>> We didn't try to delete those instances in that weird state, so I'm 
>>>>>> not sure if we could have ended up in the same situation as you. But 
>>>>>> maybe you have a similar thing going on with those undeleted 
>>>>>> interfaces in the DB that OVS tries to delete? Hard to tell...
>>>>>>
>>>>>> Regards,
>>>>>> Eugen
>>>>>>
>>>>>> Zitat von Albert Braden <ozzzo@yahoo.com>:
>>>>>>
>>>>>>> I didn't see a response to this email. Trying again with a 
>>>>>>> better subject:
>>>>>>>       On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden 
>>>>>>> <ozzzo@yahoo.com> wrote:
>>>>>>>
>>>>>>>  
>>>>>>> One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:
>>>>>>>
>>>>>>> 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)
>>>>>>>
>>>>>>> For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?