[nova] nova-manage fails when upgrading to Antelope
Hello everyone, I´m trying to upgrade to Antelope and have a problem when executing the db migrations. `nova-manage db sync` fails when upgrading from Zed to Antelope (same for Yoga --> Antelope) - Nova: 27.4.0 - Database: PostgreSQL 16 - Operating System: Ubuntu 22.04 - OpenStack Zed --> Antelope Full Log. /var/log/nova-manage.log: ``` 2025-03-04 11:42:14.963 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Context impl PostgresqlImpl. 2025-03-04 11:42:14.964 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Will assume transactional DDL. 2025-03-04 11:42:14.967 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Running upgrade ccb0fa1a2252 -> 960aac0e09ea, de-duplicate_indexes_in_instances__console_auth_tokens 2025-03-04 11:42:14.970 2519974 WARNING oslo_db.sqlalchemy.exc_filters [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] DBAPIError exception wrapped.: psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid constraint consoles_instance_uuid_fkey on table consoles depends on index uuid constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid constraint migrations_instance_uuid_fkey on table migrations depends on index uuid constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters self.dialect.do_execute( 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint consoles_instance_uuid_fkey on table consoles depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint migrations_instance_uuid_fkey on table migrations depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters ``` Any suggestions? Thanks!
I hoped that someone would respond to this thread. I don’t have a good idea, unfortunately. But since we’re quite close to upgrading to Zed (and then Antelope a bit later), I’d be interested if this is actually a bug or something just went wrong in your environment. But it definitely sounds wrong to remove a uuid. Until now, I haven’t seen reports about failing upgrades Z —> A. Do you have the option to set up a VM to test the db upgrade? That’s what I did a few years ago when we reinstalled the entire cloud to skip a couple of versions and then just import the DB dump from the VM. Zitat von Roby G <pi3.14@tuta.io>:
Hello everyone,
I´m trying to upgrade to Antelope and have a problem when executing the db migrations. `nova-manage db sync` fails when upgrading from Zed to Antelope (same for Yoga --> Antelope)
- Nova: 27.4.0 - Database: PostgreSQL 16 - Operating System: Ubuntu 22.04 - OpenStack Zed --> Antelope
Full Log. /var/log/nova-manage.log: ``` 2025-03-04 11:42:14.963 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Context impl PostgresqlImpl. 2025-03-04 11:42:14.964 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Will assume transactional DDL. 2025-03-04 11:42:14.967 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Running upgrade ccb0fa1a2252 -> 960aac0e09ea, de-duplicate_indexes_in_instances__console_auth_tokens 2025-03-04 11:42:14.970 2519974 WARNING oslo_db.sqlalchemy.exc_filters [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] DBAPIError exception wrapped.: psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid constraint consoles_instance_uuid_fkey on table consoles depends on index uuid constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid constraint migrations_instance_uuid_fkey on table migrations depends on index uuid constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters self.dialect.do_execute( 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint consoles_instance_uuid_fkey on table consoles depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint migrations_instance_uuid_fkey on table migrations depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters ```
Any suggestions?
Thanks!
I´m trying to upgrade to Antelope and have a problem when executing the db migrations. `nova-manage db sync` fails when upgrading from Zed to Antelope (same for Yoga --> Antelope)
- Nova: 27.4.0 - Database: PostgreSQL 16
I suspect this is your problem here. While you should be able to use postgres, we don't test it and things like this are exactly the sort of problem you're likely to run into. See the following: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.h... --Dan
Just a quick note: I have upgraded my test cluster twice from Zed to Antelope, no issues (at least not wrt to the database). I'm planning to upgrade our production cloud tomorrow. Zitat von Dan Smith <dms@danplanet.com>:
I´m trying to upgrade to Antelope and have a problem when executing the db migrations. `nova-manage db sync` fails when upgrading from Zed to Antelope (same for Yoga --> Antelope)
- Nova: 27.4.0 - Database: PostgreSQL 16
I suspect this is your problem here. While you should be able to use postgres, we don't test it and things like this are exactly the sort of problem you're likely to run into. See the following:
https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.h...
--Dan
participants (3)
-
Dan Smith
-
Eugen Block
-
Roby G