September 2025 - openstack-discuss

[largescale-sig] Next meeting: September 23, 14utc
by Thierry Carrez 24 Sep '25

24 Sep '25

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC at 14UTC. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20250923T14 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

2 3

Upgrade Caracal --> Epoxy obstacles
by Eugen Block 24 Sep '25

24 Sep '25

Good morning, I just wanted to share some of our experience from this week when we upgraded our production cloud from Caracal to Epoxy (and also Ubuntu 22.04 --> 24.04). It wasn't as easy as previous upgrades, there were a couple of obstacles on the way. # RabbitMQ The main issue was rabbitmq, I'd say. Since the rabbitmq version was 3.9 on Jammy, we had to get to 3.12 for Noble. I had tried a lot of things in my test environment, in the end I decided to reduce the number of rabbitmq hosts to one, upgraded it with podman from 3.9 to 3.10 to 3.11 to enable the feature flags required for 3.12 (all still under Jammy). Then I could upgrade the OS to Noble so the package based rabbitmq-server would start successfully. # Database The recent thread about older password hashes (https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…) was helpful, we set the same passwords for those system users that came up in the MySQL query, and we didn't notice any issue. But there were two issues we had to deal with regarding the database: During nova-manage db sync we got these errors: 2025-09-22 14:08:19 2 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), fou nd type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB'). 2025-09-22 14:08:19 2 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255). This had not popped up during my tests, but our production database is around 10 years old when we started with our OpenStack journey, my test environment is quite new. What fixed it was: mysql_upgrade check-if-upgrade-is-needed Then there's another thing I have to fully grasp yet, but we now have a ".local" directory in /var/lib/mysql: find /var/lib/mysql/.local/share/containers/storage/ /var/lib/mysql/.local/share/containers/storage/ /var/lib/mysql/.local/share/containers/storage/libpod /var/lib/mysql/.local/share/containers/storage/db.sql Which mysql_upgrade didn't like. I assume that podman created that, but I'm not sure why (I only created a rabbitmq pod). Anyway, I removed that directory so I could proceed with the db sync commands to complete the upgrade process. In my test cloud I removed podman, then restarted galera and the directory is gone, so that's what I'm gonna do in the production as well. # opentack-client Then there are these messages during every openstack command: Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages' Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages' I'll need to check if they impact any of our scripts, we don't use zaqar, but removing that package would also remove heat, so that won't be possible. # nova-compute: I waited for version 19.2.1 of librbd1 to be released, otherwise nova-compute would fail: qemu-block-extra : Depends: librbd1 (>= 19.2.1-0ubuntu0.24.04.1) but 19.2.0-0ubuntu0.24.04.2 is to be installed # neutron-metadata-agent There's a bug for functioning agents showing as dead, applying the mentioned patches fixes it: https://bugs.launchpad.net/neutron/+bug/2112492 # Pacemaker We use pacemaker for high availability and during the dist-upgrade I had to run 'apt --fix broken', then remove resource-agents and resource-agents-base to get through the dist-upgrade, then install pacemaker and crmsh again afterwards. Not sure if anyone will read this far, but I wanted to get that off my chest. ;-) So obviously, most of the issues weren't really OpenStack issues, but I wanted to share this experience anyway. If you read this far, thanks! ;-) Regards, Eugen

3 2

[kolla] Weekly meeting cancelled today (24th Sep 2025)
by Michał Nasiadka 24 Sep '25

24 Sep '25

Hello, Apologies for bad timing, but the weekly meeting is cancelled today due to lack of key participants. See you next week! Michal

1 0

I can not get user list with project in openstack version epoxy
by cheung 24 Sep '25

24 Sep '25

Dears: I can get user list with project in openstack version caracal. # openstack project list -f value 7fb2be327b4c4f9e83db04a4b5cb922b service c1bcb532eca44aadabb028b1ee715412 admin de4012f880e7408f9fb21b67930c21ac project_it # openstack user list -f value 84b39214b13148d68bf88242aa6abdf9 admin 62f7113645c24199a9946cd16911ca64 glance b21da7173fe3400f91d3eb73f392474d nova 95a6ff7b9bba42f9aadf3b07a0811330 placement 4ac268a8e75b4a4288f067e28ad14425 neutron 46fa578d54624e06ae2e0b3761a67de7 cinder 1c1646579b5042a6bb463e8f46b7762f test001 57689f8310e744889dce33d6b32f4524 test002 # openstack user list -f value --project project_it 1c1646579b5042a6bb463e8f46b7762f test001 57689f8310e744889dce33d6b32f4524 test002 # openstack user list -f value --project service 4ac268a8e75b4a4288f067e28ad14425 neutron 46fa578d54624e06ae2e0b3761a67de7 cinder b21da7173fe3400f91d3eb73f392474d nova 62f7113645c24199a9946cd16911ca64 glance 95a6ff7b9bba42f9aadf3b07a0811330 placement # dpkg -l| grep keyston ii keystone 2:25.0.0-0ubuntu1 all OpenStack identity service - Daemons ii keystone-common 2:25.0.0-0ubuntu1 all OpenStack identity service - Common files ii python3-keystone 2:25.0.0-0ubuntu1 all OpenStack identity service - Python 3 library ii python3-keystoneauth1 5.6.0-0ubuntu1.1 all authentication library for OpenStack Identity - Python 3.x ii python3-keystoneclient 1:5.4.0-0ubuntu1 all client library for the OpenStack Keystone API - Python 3.x ii python3-keystonemiddleware 10.6.0-0ubuntu1 all Middleware for OpenStack Identity (Keystone) - Python 3.x But I can not get user list in epoxy version. # openstack project list -f value 0087125cb3564da39a5e87dc61aece78 project_it 8b6a848d2d7841118567f1dad2eeb9f5 admin dcaa69462fd3447ca75133a8ce4b95c3 service # openstack user list -f value 27bd419eb7bc4da58f9e246232a12720 placement 7c5a457f461440e5a24a368f827ecafc admin 846cca4cb4394d1ab4e8d95dffb63ce2 glance 92122f012abb4db38a0d3ca287c04122 nova ad039b4d1f4f4c42a59e69e56eef4f16 neutron ebde13fec7584f5180becb88afa0e1c7 cinder f4cf0b16f52e407d994e2aa729936580 test002 f9ee38a0a4cb412dbd293a83b5b32dc5 test001 # openstack user list -f value --project project_it Either group or user should be specified # openstack user list -f value --project service Either group or user should be specified What do I miss? I need to read which one document? 本電子郵件及其所有附件所含之資訊均屬機密，僅供指定之收件人使用，未經寄件人同意不得揭露、複製或散布本電子郵件。若您並非指定之收件人，請勿使用、保存或揭露本電子郵件之任何部分，並請立即通知寄件人並完全刪除本電子郵件。網路通訊可能含有病毒，收件人應自行確認本郵件是否安全，若因此造成損害，寄件人恕不負責。 The information contained in this communication and attachment is confidential and is intended only for the use of the recipient to which this communication is addressed. Any disclosure, copying or distribution of this communication without the sender's consents is strictly prohibited. If you are not the intended recipient, please notify the sender and delete this communication entirely without using, retaining, or disclosing any of its contents. Internet communications cannot be guaranteed to be virus-free. The recipient is responsible for ensuring that this communication is virus free and the sender accepts no liability for any damages caused by virus transmitted by this communication.

2 3

[ops][largescale-sig][scientific-sig] Ops Radio Hour on September 26, 2025
by Ildiko Vancsa 24 Sep '25

24 Sep '25

Hi OpenStack community, I'm reaching out to you with a friendly reminder that the next OpenStack Ops Radio Hour[1] is scheduled to happen on __September 26th, 2025 at 1300UTC.__ The call series is a casual setting for operators, engineers, and contributors to come together, share knowledge, and collaborate on the complexities of managing and deploying OpenStack environments. Topics the next gathering will dive into: - (SWIFT) Compliance processes If you have further topics you would like to dive into during the call, please add them to this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour A few reminders: - The group on the call agreed to try out Matrix as a chat platform for more synchronous communication outside calls + You can access the Matrix room here: https://matrix.to/#/#openstack-ops:opendev.org<https://matrix.to/#/%23openstack-ops:opendev.org> + Further details are here: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - If you would like to read and catch up on notes from previous calls, you can find them in this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour See you on the call on September 26th! Bets Regards, Ildikó [1] https://wiki.openstack.org/wiki/Operations/Meetups

1 1

[watcher] watcher and affinity policy
by Nguyễn Hữu Khôi 24 Sep '25

24 Sep '25

Hi, Would you like to correct me if I understand that watcher will not consider server groups when it does jobs? If I am right, do we have any solutions to this problem? Thank you. Regards Nguyen Huu Khoi

3 3

[tc][all] OpenInfra Summit 2025 Open Source Pavilion Opportunity
by Clark Boylan 23 Sep '25

23 Sep '25

The OpenInfra Summit 2025 will host an Open Source Pavilion booth space where open source projects can book an hour or two to showcase their work, engage with attendees, and share updates on their latest developments at the Summit. If you are interested in taking advantage of this opportunity for your project please fill out this form [0] and the event planners will get back to you with scheduling details. This seems like a good opportunity for both the broader OpenStack community to engage with attendees, and also for specific OpenStack projects, working groups, and SIGs. Feel free to followup with any questions you might have here on this thread, and I'll do my best to get answers. I suspect that you can also get the process started by filling out the form, and the organizers can followup with you to answer questions as well. [0] https://openinfrafoundation.formstack.com/forms/open_source_pavillion Clark

2 1

[designate] DNS records stuck in PENDING state but records created succesfully
by Jadon Naas 23 Sep '25

23 Sep '25

Hello! I am using Designate on 2023.2 with a PowerDNS backend. I have a recordset with three A records created through Designate. The A records are stuck in "PENDING" state with action "CREATE" or "UPDATE". The records were successfully created in PowerDNS, and changes are pushed to PowerDNS. However, the record status never finishes updating. Deleting and recreating the recordset results in the same situation. I do not see any notable errors in the logs for Designate or for PowerDNS. I have checked that the Desigate configuration for the nameservers and for the targets match the correct IP addresses for my PowerDNS server and for my Designate service hosts. Would folks have any suggestions for what I could do to determine what is preventing Designate from updating these records or to manually update the status of the records? Thank you! Jadon Naas

2 1

[publiccloud-sig] Public Cloud SIG meeting d.d. 23 September
by Kees Meijs | Nefos 23 Sep '25

23 Sep '25

Hello Stackers, Tomorrow (that's 23 September) the bi-weekly Public Cloud SIG meeting will be held again. The publicly available video meeting will be conducted from 07:00 hrs UTC (09:00 hrs CEST) and on here: https://meetpad.opendev.org/publiccloud-sig Additional information to be found on https://wiki.openstack.org/wiki/PublicCloudSIG See you tomorrow! Cheers, Kees __ Kees Meijs BICT Nefos Cloud & IT <https://nefos.com/contact> Nefos IT bv Burgemeester Mollaan 34a 5582 CK Waalre - NL kvk 66494931 +31 (0)88 2088 188 <tel:+31882088188> nefos.com <https://nefos.com/contact> The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organization. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended. General terms and conditions ("The NLdigital Terms") apply to all our products and services.

1 1

Compatebilty
by nahoms＠altacomputec.com 23 Sep '25

23 Sep '25

The customer currently uses 15 Dell PowerEdge R650 servers for their OpenStack private cloud and wants to expand. Is there any compatibility issue if we propose Lenovo servers along with the existing Dell ones?

2 1