Open Stack

Hello,

On 10/28/20 12:01, melanie witt wrote:
 >> The main idea of the row deletions is to delete "orphan" records 
which are records tied to an instance's lifecycle when that instance no 
longer exists. Going forward, nova will delete these records itself at 
instance deletion time but did not in the past because of bugs, and any 
records generated before a bug was fixed will become orphaned once the 
associated instance is deleted.

I've done the following in this order:

nova-manage api_db sync

nova-manage db sync

(to bring the DBs up to the version I'm upgrading to (Train)

nova-manage db archive_deleted_rows --verbose --until-complete

Archiving..............................................................complete
+--------------------------+-------------------------+
| Table | Number of Rows Archived |
+--------------------------+-------------------------+
| block_device_mapping | 4805 |
| instance_actions | 7813 |
| instance_actions_events | 8556 |
| instance_extra | 2658 |
| instance_faults | 1325 |
| instance_group_member | 0 |
| instance_id_mappings | 118 |
| instance_info_caches | 2656 |
| instance_mappings | 2656 |
| instance_metadata | 845 |
| instance_system_metadata | 24437 |
| instances | 2656 |
| migrations | 121 |
| request_specs | 2656 |
| virtual_interfaces | 2767 |
+--------------------------+-------------------------+

nova-manage db purge --all --verbose

DB: Deleted 4805 rows from shadow_block_device_mapping based on 
timestamp column (n/a)
DB: Deleted 7813 rows from shadow_instance_actions based on timestamp 
column (n/a)
DB: Deleted 8556 rows from shadow_instance_actions_events based on 
timestamp column (n/a)
DB: Deleted 2658 rows from shadow_instance_extra based on timestamp 
column (n/a)
DB: Deleted 1325 rows from shadow_instance_faults based on timestamp 
column (n/a)
DB: Deleted 118 rows from shadow_instance_id_mappings based on timestamp 
column (n/a)
DB: Deleted 2656 rows from shadow_instance_info_caches based on 
timestamp column (n/a)
DB: Deleted 845 rows from shadow_instance_metadata based on timestamp 
column (n/a)
DB: Deleted 24437 rows from shadow_instance_system_metadata based on 
timestamp column (n/a)
DB: Deleted 2656 rows from shadow_instances based on timestamp column (n/a)
DB: Deleted 121 rows from shadow_migrations based on timestamp column (n/a)
DB: Deleted 2767 rows from shadow_virtual_interfaces based on timestamp 
column (n/a)
[root at server01 placement]# nova-manage db null_instance_uuid_scan
There were no records found where instance_uuid was NULL.
[root at server01 placement]# nova-manage db online_data_migrations
Running batches of 50 until complete
4 rows matched query migrate_empty_ratio, 4 migrated
46 rows matched query fill_virtual_interface_list, 0 migrated
46 rows matched query populate_user_id, 44 migrated
49 rows matched query fill_virtual_interface_list, 0 migrated
50 rows matched query populate_user_id, 50 migrated
48 rows matched query fill_virtual_interface_list, 0 migrated
50 rows matched query populate_user_id, 47 migrated
50 rows matched query fill_virtual_interface_list, 0 migrated
50 rows matched query populate_user_id, 45 migrated
50 rows matched query fill_virtual_interface_list, 0 migrated
50 rows matched query populate_user_id, 44 migrated
50 rows matched query fill_virtual_interface_list, 0 migrated
50 rows matched query populate_user_id, 44 migrated
19 rows matched query fill_virtual_interface_list, 0 migrated
50 rows matched query populate_user_id, 44 migrated
49 rows matched query populate_user_id, 42 migrated
7 rows matched query populate_user_id, 0 migrated
+---------------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+---------------------------------------------+--------------+-----------+
| create_incomplete_consumers | 0 | 0 |
| delete_build_requests_with_no_instance_uuid | 0 | 0 |
| fill_virtual_interface_list | 312 | 0 |
| migrate_empty_ratio | 4 | 4 |
| migrate_keypairs_to_api_db | 0 | 0 |
| migrate_quota_classes_to_api_db | 0 | 0 |
| migrate_quota_limits_to_api_db | 0 | 0 |
| migration_migrate_to_uuid | 0 | 0 |
| populate_missing_availability_zones | 0 | 0 |
| populate_queued_for_delete | 0 | 0 |
| populate_user_id | 402 | 360 |
| populate_uuids | 0 | 0 |
| service_uuids_online_data_migration | 0 | 0 |
+---------------------------------------------+--------------+-----------+

just to clean up the nova and nova_api databases before running your 
suggested SQL query:

MariaDB [(none)]> delete from nova_api.consumers where 
nova_api.consumers.uuid not in (select 
nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);
Query OK, 2656 rows affected (0.080 sec)

Excellent! 2656 is quite a few.

Then I run mysql-migrate-db.sh:

# /usr/share/placement/mysql-migrate-db.sh --migrate
Nova database contains data, placement database does not. Okay to 
proceed with migration
Dumping from NOVA_API to migrate-db.wh1tDGAz/from-nova.sql
Loading to PLACEMENT from migrate-db.wh1tDGAz/from-nova.sql
/usr/lib/python2.7/site-packages/pymysql/cursors.py:170: Warning: (1280, 
u"Name 'alembic_version_pkc' ignored for PRIMARY key.")
result = self._query(query)

Hmm, a little worrying but let's continue:

# placement-status upgrade check
+----------------------------------------------------------------------+
| Upgrade Check Results |
+----------------------------------------------------------------------+
| Check: Missing Root Provider IDs |
| Result: Success |
| Details: None |
+----------------------------------------------------------------------+
| Check: Incomplete Consumers |
| Result: Warning |
| Details: There are 6 incomplete consumers table records for existing |
| allocations. Run the "placement-manage db |
| online_data_migrations" command. |
+----------------------------------------------------------------------+

Only 6, and it's now a positive number! Much progress.

# placement-manage db online_data_migrations
Running batches of 50 until complete
8 rows matched query create_incomplete_consumers, 8 migrated
+-----------------------------+-------------+-----------+
| Migration | Total Found | Completed |
+-----------------------------+-------------+-----------+
| set_root_provider_ids | 0 | 0 |
| create_incomplete_consumers | 8 | 8 |
+-----------------------------+-------------+-----------+

Hmm, 8 incomplete consumers created...

# placement-status upgrade check
+-----------------------------------------------------------------------+
| Upgrade Check Results |
+-----------------------------------------------------------------------+
| Check: Missing Root Provider IDs |
| Result: Success |
| Details: None |
+-----------------------------------------------------------------------+
| Check: Incomplete Consumers |
| Result: Warning |
| Details: There are -2 incomplete consumers table records for existing |
| allocations. Run the "placement-manage db |
| online_data_migrations" command. |
+-----------------------------------------------------------------------+

argh! again a negative number! But at least it's only 2, which is well 
within the realm of manual fixes.

Thank you very much Melanie for pointing me at what has been a very 
frustrating bug.

Seth