April 2025 - openstack-discuss - lists.openstack.org

Bug deputy week 15 (April 7 - April 11)
by Sahid Orentino Ferdjaoui 11 Apr '25

11 Apr '25

Hello Guys, Here is a quick summary of Neutron related bugs this week == neutron-server error with ml2_distributed_port_bindings https://bugs.launchpad.net/neutron/+bug/2106374/ Status: LOW This issue appears to be related to tenant network/database handling. It throws an unexpected database exception which ideally should be handled more gracefully. Possibly a low-hanging fruit, that is said more investigation may be needed to understand the root cause. == [ovn-octavia-provider] Loadbalancer member list show operating status as ONLINE when member is created for a SHUTOFF state vm https://bugs.launchpad.net/neutron/+bug/2106357/ Status: LOW This seems to be a corner case where a stopped VM can be added to a load balancer, and it incorrectly appears as ONLINE. Unfortunately, I didn’t get the chance to reproduce it this week, apologies for that. == ovn-sb leader constantly changing https://bugs.launchpad.net/neutron/+bug/2106690/ Status: INCOMPLETE but reported is active and will share more detail soon Seem like point of instability with OVN SB database leader election. Waiting for more details. == [OVN] No network type availability zone https://bugs.launchpad.net/neutron/+bug/2106719/ Status: NEW This one may require someone with better picture of ML2/OVN and AZ to properly evaluate and move forward. == after running ``os-nova-install.yml`` lost instances connectivity https://bugs.launchpad.net/neutron/+bug/2106793 Status: INCOMPLETE at that point The issue seems to occur after a Nova restart, where instances lose network connectivity. Needs more info and confirmation. == Unable to enable/disable src_nat per external gateway https://bugs.launchpad.net/neutron/+bug/2106776 Status: TRIAGED This is related to OVN and occurs in scenarios involving multiple external gateways. == Have a good day! I unfortunately wont be available for driver meeting. s.

1 0

Question about patch to fix problem
by Franck VEDEL (UGA) 11 Apr '25

11 Apr '25

Hello, A few days ago, I asked a question about applying a patch to resolve a problem in my OpenStack environment. I didn't get a response. My question was probably stupid and not appropriate for this list. I apologize. Having to resolve this problem, I searched high and low using search engines. It seems that the solution, with kolla-ansible, is to upgrade to Epoxy, which takes the patch into account. Since this update could cause other problems, I don't want to do it for the moment. Is there another solution? Is it possible to modify the code (it's simply a line to modify in a file (https://review.opendev.org/c/openstack/horizon/+/942946)) in the two Horizon containers in my cluster? I'm really sorry if this is a stupid question, but I've been searching and can't find a method for applying these patches. Franck

5 8

[kolla] IRC meeting cancelled - 9-th of April 2025
by Bartosz Bezak 10 Apr '25

10 Apr '25

Hi, Kolla IRC meeting today is cancelled due to ongoing Project Teams Gathering (PTG) this week. Best regards, Bartosz Bezak

2 3

[octavia] Point releases
by Mikhail Okhrimenko 10 Apr '25

10 Apr '25

Hello community! We found that the latest octavia bobcat release packages for different OS like Rocky Linux 9, Debian 12, Ubuntu 22.04 do not contain the latest bug fixes from the release notes https://docs.openstack.org/releasenotes/octavia/2023.2.html of version 13.0.0-37. And there is no tag with these fixes in the octavia repository https://github.com/openstack/octavia/tags. So could you please create a point release in the octavia repository so that we can build packages with these changes. We also noticed the same situation with the release versions of caracal and dalmatian.

4 3

[nova][NFS] Changing disk permission on reboot, permission denied
by Nell Jerram 10 Apr '25

10 Apr '25

Hi; I'm seeing a problem with a hard reboot operation, when /var/lib/nova/instances for the relevant compute node is on an NFS share exported from another compute node. I'm hoping someone here might be able to point to where to debug this next. 3 compute nodes: A, B and C. A exports its /var/lib/nova/instances over NFS v4. B and C mount that at /var/lib/nova/instances. (This is the kind of setup that I understand to be required for live migration to be possible.) If I create a VM on A, wait for it to be ACTIVE, then `nova reboot --hard testvm1`, it's fine, i.e. it reboots and returns to ACTIVE state. If I create a VM on B, wait for it to be ACTIVE, then `nova reboot --hard testvm2`, it goes permanently into ERROR state. The Nova compute log has Stderr: "qemu-img: Could not open '/var/lib/nova/instances/02bf258b-41bc-451a-9d3c-af6741c6060f/disk': Could not open '/var/lib/nova/instances/02bf258b-41bc-451a-9d3c-af6741c6060f/disk': Permission denied\n": nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /var/lib/nova/instances/02bf258b-41bc-451a-9d3c-af6741c6060f/disk : Unexpected error while running command. and the ownership on that file is in fact now: -rw-r----- 1 root root 29491200 Apr 7 18:27 disk Before the reboot operation it was: -rw-r----- 1 libvirt-qemu kvm 29491200 Apr 7 18:25 disk There isn't a similar ownership change when rebooting a VM on node A, so presumably this is something to do with NFS being involved. But I believe I've followed standard NFS setup advice, and my user and group IDs are the same on all compute nodes: $ getent group nova nova:x:1003:libvirt-qemu $ getent passwd nova nova:x:1002:1003::/home/nova:/bin/bash $ getent passwd libvirt-qemu libvirt-qemu:x:64055:109:Libvirt Qemu,,,:/var/lib/libvirt:/usr/sbin/nologin $ getent group kvm kvm:x:109:nova Any clues on how to debug or fix this next? Many thanks - Nell

3 9

Re: [openstack-magnum ui] Cannot choose an external network ID when creating a cluster template
by Eugen Block 10 Apr '25

10 Apr '25

Please don't drop the list from your responses. Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw(a)gmail.com>: > Hello, They r different, > I check and see this happen because horizon switched to SDK > > https://github.com/openstack/horizon/commit/56f0e2685e2692467442d5be846b863… > > and Magnum-UI still keep old > > https://github.com/openstack/magnum-ui/blob/master/magnum_ui/static/dashboa… > > Thank you for replying to me. > > Nguyen Huu Khoi > > > On Thu, Apr 10, 2025 at 5:11 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Could it be this bug? >> >> https://bugs.launchpad.net/horizon/+bug/2077329 >> >> Zitat von Nguyễn Hữu Khôi <nguyenhuukhoinw(a)gmail.com>: >> >> > Hello, When I create a cluster template, I cannot choose an external >> > network ID, it disappears from horizon. >> > My env: >> > >> > Ubuntu 22.04 >> > Openstack 2024.2 >> > >> > Thank you. Regards >> > Nguyen Huu Khoi >> >> >> >>

2 1

[openstack-magnum ui] Cannot choose an external network ID when creating a cluster template
by Nguyễn Hữu Khôi 10 Apr '25

10 Apr '25

Hello, When I create a cluster template, I cannot choose an external network ID, it disappears from horizon. My env: Ubuntu 22.04 Openstack 2024.2 Thank you. Regards Nguyen Huu Khoi

2 1

[PTL][release][stable] 2023.2 Bobcat transition to End of Life
by Elõd Illés 10 Apr '25

10 Apr '25

Hi teams, As 2025.1 Epoxy was released last week and we are in a less busy period (except the PTG :)), now is the good time to call your attention to the following: In 3 weeks 2023.2 Bobcat is planned to transition to End of Life [1] (planned date: 2025-04-30). Note that 2023.2 Bobcat is not a SLURP release, hence it is not eligible to transition to Unmaintained [2] and will move to End of Life directly and the tip of stable/2023.2 branches will be tagged with 2023.2-eol tag and stable/2023.2 branches will be deleted afterwards. To projects with cycle-with-trailing release model (Kolla, OpenStack Ansible, Puppet OpenStack): transition of your projects have an extended deadline, ~2 months after the planned EOL date (2025-04-30). I have generated the list of the current *open* and *unreleased* changes in stable/2023.2 for every repositories [3] (where there are such patches). These lists could help the teams who are planning to do a *final* release on Bobcat before deleting stable/2023.2 branches. Feel free to edit and extend these lists to track your team's progress! *NOTE*: teams, please focus on wrapping up your libraries first if there is any concern about the changes, in order to avoid broken (final!!) releases! Also please try to avoid any regression in the final stable release and do not propose any release that has only non-functional code changes. [1] https://releases.openstack.org/ [2] https://docs.openstack.org/project-team-guide/stable-branches.html#unmainta… [3] https://etherpad.opendev.org/p/2023.2-bobcat-final-release-before-eol Thanks, Előd Illés irc: elodilles @ #openstack-release

1 0

[swift][ptg] Ops Feedback Session - Th 10 Apr at 14:00 UTC
by Tim Burke 09 Apr '25

09 Apr '25

It's that time again! Let's get devs and ops together to talk about Swift: what's working, what isn't, and what would be most helpful to improve. We're meeting on meetpad (https://meetpad.opendev.org/p/swift-flamingo-ops-feedback) on Wednesday at 14:00UTC -- if you run a Swift cluster, we hope to see you there! Even if you can't make it, we'd appreciate it if you can offer some feedback on the etherpad (https://etherpad.opendev.org/p/swift-flamingo-ops-feedback) This is always=A0a highlight at every PTG for the swift developers. Have your say and help make the next Swift the best one yet! Tim

1 0

GPU PCI passthrough woes.
by Andy Speagle 09 Apr '25

09 Apr '25

Hey Folks, I could use a little assistance getting GPU passthrough working. I had this working already for one flavor of nvidia gpu... and I've added some hosts with a much newer gpu... I've updated the pci_alias and pci_passthrough variables and those seem to be getting set properly in nova.conf passthrough_whitelist = [{"vendor_id":"10de", "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": "10de"} I believe I have all of the iommu stuff configured and have the pci- stub module entries... dmesg output shows that the GPUs are being claimed by the stub module. $ openstack flavor show t1.small_gpu_l40s +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 0 | | id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | | name | t1.small_gpu_l40s | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='gpu-l40s:1' | | ram | 2048 | | rxtx_factor | 1.0 | | swap | 0 | | vcpus | 1 | +----------------------------+--------------------------------------+ Yet... I can't seem to get an instance to run using that new flavor... keeps complaining about there not being enough hosts available. Fault: code: 500 created: 2024-10-30T02:04:33Z message: "No valid host was found. There are not enough hosts available." details: | Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Any clues on how/where to dig into this more to see what might be missing? Thanks. -- Andy Speagle

3 2