Open Stack

On 10/23/20 02:43, Seth Tunstall wrote:
> Hello,
> 
> As another person has mentioned (Chanu Romain), I am attempting an upgrade from Queens to Train. I have successfully upgraded Keystone and Glance, but while trying to separate Placement's databases out from Nova, I run into the the same issue with placement-status upgrade check:
> 
> # placement-status upgrade check
> +-----------------------------------------------------------------+
> | Upgrade Check Results                                           |
> +-----------------------------------------------------------------+
> | Check: Missing Root Provider IDs                                |
> | Result: Success                                                 |
> | Details: None                                                   |
> +-----------------------------------------------------------------+
> | Check: Incomplete Consumers                                     |
> | Result: Warning                                                 |
> | Details: There are -3511 incomplete consumers table records for |
> |   existing allocations. Run the "placement-manage db            |
> |   online_data_migrations" command.                              |
> +-----------------------------------------------------------------+
> 
> Likewise, running placement-manage db online_data_migrations does nothing.

Hm, in both cases I see that the status check is reporting a _negative_ number of consumers, which is unexpected. Looking at the code, this value is calculated by subtracting the number of 'consumers' [1] table records from the number of 'allocations' table consumers. The fact that this is negative means there are more 'consumers' table records with unique project/user pairs than there are 'allocations' table records with unique project/user pairs.

This implies to me that there are orphaned 'consumers' table records in your databases. We had a bug around this in the past [2] where 'consumers' table records were not being deleted when they no longer had any 'allocations' table records associated with them. That bug was fixed in the Rocky release but the fix does not clean up orphaned 'consumers' records that were orphaned before the fix landed. Is your original deployment from before Rocky and has been upgraded?

If so, then if you have orphaned 'consumers' records, you will need to do a one-time manual database clean up as described in this comment [3]. You can check whether you have any orphaned records by a SQL query like this:

select * from nova_api.consumers where nova_api.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);

If you do, after you clean them up manually, you should stop seeing the placement-status upgrade check warning.

Note that it is not so harmful to have orphaned 'consumers' in database in that it won't hurt the operation of nova other than taking up database space and possibly impacting database and nova performance as a result. But it's a good idea to clean them up. And >= Rocky 'consumers' are properly deleted by placement so orphans cannot happen from then on.

Hope this helps,
-melanie

[1] 'consumers' are project/user combinations of instance owners
[2] https://bugs.launchpad.net/nova/+bug/1780799
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1726256#c24