Good morning, I just wanted to share some of our experience from this week when we upgraded our production cloud from Caracal to Epoxy (and also Ubuntu 22.04 --> 24.04). It wasn't as easy as previous upgrades, there were a couple of obstacles on the way. # RabbitMQ The main issue was rabbitmq, I'd say. Since the rabbitmq version was 3.9 on Jammy, we had to get to 3.12 for Noble. I had tried a lot of things in my test environment, in the end I decided to reduce the number of rabbitmq hosts to one, upgraded it with podman from 3.9 to 3.10 to 3.11 to enable the feature flags required for 3.12 (all still under Jammy). Then I could upgrade the OS to Noble so the package based rabbitmq-server would start successfully. # Database The recent thread about older password hashes (https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....) was helpful, we set the same passwords for those system users that came up in the MySQL query, and we didn't notice any issue. But there were two issues we had to deal with regarding the database: During nova-manage db sync we got these errors: 2025-09-22 14:08:19 2 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), fou nd type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB'). 2025-09-22 14:08:19 2 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255). This had not popped up during my tests, but our production database is around 10 years old when we started with our OpenStack journey, my test environment is quite new. What fixed it was: mysql_upgrade check-if-upgrade-is-needed Then there's another thing I have to fully grasp yet, but we now have a ".local" directory in /var/lib/mysql: find /var/lib/mysql/.local/share/containers/storage/ /var/lib/mysql/.local/share/containers/storage/ /var/lib/mysql/.local/share/containers/storage/libpod /var/lib/mysql/.local/share/containers/storage/db.sql Which mysql_upgrade didn't like. I assume that podman created that, but I'm not sure why (I only created a rabbitmq pod). Anyway, I removed that directory so I could proceed with the db sync commands to complete the upgrade process. In my test cloud I removed podman, then restarted galera and the directory is gone, so that's what I'm gonna do in the production as well. # opentack-client Then there are these messages during every openstack command: Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages' Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages' I'll need to check if they impact any of our scripts, we don't use zaqar, but removing that package would also remove heat, so that won't be possible. # nova-compute: I waited for version 19.2.1 of librbd1 to be released, otherwise nova-compute would fail: qemu-block-extra : Depends: librbd1 (>= 19.2.1-0ubuntu0.24.04.1) but 19.2.0-0ubuntu0.24.04.2 is to be installed # neutron-metadata-agent There's a bug for functioning agents showing as dead, applying the mentioned patches fixes it: https://bugs.launchpad.net/neutron/+bug/2112492 # Pacemaker We use pacemaker for high availability and during the dist-upgrade I had to run 'apt --fix broken', then remove resource-agents and resource-agents-base to get through the dist-upgrade, then install pacemaker and crmsh again afterwards. Not sure if anyone will read this far, but I wanted to get that off my chest. ;-) So obviously, most of the issues weren't really OpenStack issues, but I wanted to share this experience anyway. If you read this far, thanks! ;-) Regards, Eugen