Zitat von Eugen Block <
eblock@nde.ag>:
> Yeah, that matches the log output (no such device). The port mapping
> is in this file on the hypervisor (not sure where else):
>
> /var/lib/openvswitch/conf.db
>
> Do you see your "orphaned" port there? But don't change anything
> (yet). I'm not sure which tool to use for cleanup yet.
>
> Zitat von Albert Braden <
ozzzo@yahoo.com>:
>
>> No output. It looks like that port may not exist anymore.
>> On Tuesday, January 14, 2025 at 09:02:49 AM EST, Eugen Block
>> <
eblock@nde.ag> wrote:
>>
>> Can you check this command:
>>
>> openstack port list | grep 0430a2f7
>>
>> This could help identify the port, but only if it's really still
>> present in neutron.
>>
>> Zitat von Albert Braden <
ozzzo@yahoo.com>:
>>
>>> This is what I see in the log:
>>>
>>> 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network
>>> device qvo0430a2f7-cd (No such device)
>>>
>>> When I do "openstack port list --long" I get ID, Name, MAC, IPs,
>>> Status, Security Groups, Owner and Tags, but I don't see anything
>>> that matches qvo0430a2f7-cd. How do I translate qvo0430a2f7-cd to a
>>> port UUID? On Tuesday, January 14, 2025 at 01:30:43 AM EST,
>>> Eugen Block <
eblock@nde.ag> wrote:
>>>
>>> Basically, it's a port, so you could try to show|delete|modify it:
>>>
>>> openstack port list | grep {whatever}
>>> openstack port show|delete|set {PORT_UUID}
>>>
>>> Zitat von Albert Braden <
ozzzo@yahoo.com>:
>>>
>>>> I apologize for the vague question. This particular interface is
>>>> orphaned. It looks like the VM may have been deleted during a
>>>> maintenance and the interface failed to delete. The symptom is that
>>>> every time I delete a VM, on the hypervisor with the orphaned
>>>> interface, it takes substantially longer than other hypervisors, so
>>>> I'm guessing that the orphaned interface is confusing Neutron.
>>>>
>>>> Is is possible to perform operations on an interface, without
>>>> reference to a VM? I'd like to delete this orphaned interface,
>>>> hopefully without manually editing the DB.
>>>> On Monday, January 13, 2025 at 01:54:19 PM EST, Eugen Block
>>>> <
eblock@nde.ag> wrote:
>>>>
>>>> You can see the current interfaces with:
>>>>
>>>> nova interface-list {UUID}
>>>>
>>>> Zitat von Albert Braden <
ozzzo@yahoo.com>:
>>>>
>>>>> I poked around in the Neutron documentation, but I can't find the
>>>>> commands interfaces. I looked at the Neutron API but the only
>>>>> "interface" commands there relate to routers. Is there an Openstack
>>>>> tool that will let me view or delete interfaces?
>>>>> On Friday, January 10, 2025 at 02:30:02 AM EST, Eugen Block
>>>>> <
eblock@nde.ag> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> the short answer is, I don't know. It really depends on what exactly
>>>>> the issue is here. It requires some investigation to be able to tell
>>>>> if any openstack tools can help, e. g. neutron related commands, or if
>>>>> you need to tackle this from ovs directly, or if it's a database
>>>>> inconsistency or maybe something else entirely.
>>>>> For example, what I referred to was an interface that wasn't attached
>>>>> to an instance anymore, and the port didn't exist anymore, but the
>>>>> database had not been properly updated. I didn't see any other way
>>>>> than to manually update the DB. But you might be facing something
>>>>> different here, maybe someone else has experienced the same as you,
>>>>> but since nobody else responded yet, you'll have to dig on your own.
>>>>>
>>>>> Zitat von Albert Braden <
ozzzo@yahoo.com>:
>>>>>
>>>>>> Hi Eugen,
>>>>>>
>>>>>> Thanks for your email. I'm not aware of a VM that corresponds to the
>>>>>> stuck interface. Every VM deletion on the affected hypervisor is
>>>>>> delayed; this has been happening for a few weeks now. Looking at the
>>>>>> logs, it appears that an interface is created when a VM is created,
>>>>>> and deleted with that VM. I'm guessing that the interface may have
>>>>>> become stuck when a customer deleted a VM during a maintenance, and
>>>>>> the interface failed to delete. Is it possible to delete an
>>>>>> interface without editing the database?
>>>>>> On Thursday, January 9, 2025 at 07:54:46 AM EST, Eugen Block
>>>>>> <
eblock@nde.ag> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> maybe some more history and details could help understand what might
>>>>>> be the issue here.
>>>>>> So you have VMs with one or more interfaces that you try to delete. Do
>>>>>> you see those interfaces in 'nova interface-list {UUID}'? And while
>>>>>> deleting the VMs, ovs takes a long time becaus it can't find those
>>>>>> devices, correct?
>>>>>>
>>>>>> This reminds me of our reinstallation a few months ago as well
>>>>>> (importing the previous DB dump). After migrating from openSUSE to
>>>>>> Ubuntu (Victoria), we upgraded to Wallaby. In the post-upgrade steps
>>>>>> the 'nova-manage db online_data_migrations' failed because of a few
>>>>>> instances with a weird port state. A user attached a new interface
>>>>>> according to the event list, but we don't see a detach in the logs
>>>>>> although the instances had only one remaining NIC (as before). We had
>>>>>> to dig really deep, including manipulating the DB (mark that interface
>>>>>> as deleted).
>>>>>> We didn't try to delete those instances in that weird state, so I'm
>>>>>> not sure if we could have ended up in the same situation as you. But
>>>>>> maybe you have a similar thing going on with those undeleted
>>>>>> interfaces in the DB that OVS tries to delete? Hard to tell...
>>>>>>
>>>>>> Regards,
>>>>>> Eugen
>>>>>>
>>>>>> Zitat von Albert Braden <
ozzzo@yahoo.com>:
>>>>>>
>>>>>>> I didn't see a response to this email. Trying again with a
>>>>>>> better subject:
>>>>>>> On Friday, January 3, 2025 at 10:20:56 AM EST, Albert Braden
>>>>>>> <
ozzzo@yahoo.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface:
>>>>>>>
>>>>>>> 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device)
>>>>>>>
>>>>>>> For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?