Open Stack

Hello,

I tried your procedure it didnt remove the warning from "placement-status upgrade check".

As you said I did:
delete from nova_api.consumers where nova_api.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);

it removed only 2458 rows, far far away from my -15753 consumers.

I restored my backup and tried what you said here
https://bugzilla.redhat.com/show_bug.cgi?id=1726256#c24

update instance_id_mappings set deleted = id where uuid in (select uuid from instances where deleted != 0);
nova-manage db archive_deleted_rows --until-complete

delete from instance_id_mappings where uuid not in (select uuid from instances);
(removed 70 rows)

then I tried again:
delete from nova_api.consumers where nova_api.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);
(deleted around 15900 rows)

Sadly, placement-status upgrade check still returns  (There are -15753 incomplete consumers...)

I restored my backup.

It does not fix the bug but should i still delete the rows like I did? Do you have any other lead to fix it? Did I miss something?

Best regards,
Romain

________________________________________
From: melanie witt <melwittt at gmail.com>
Sent: Friday, October 23, 2020 5:17 PM
To: Seth Tunstall; openstack-discuss at lists.openstack.org
Cc: CHANU ROMAIN
Subject: Re: [placement] Train upgrade warning

On 10/23/20 02:43, Seth Tunstall wrote:
> Hello,
>
> As another person has mentioned (Chanu Romain), I am attempting an upgrade from Queens to Train. I have successfully upgraded Keystone and Glance, but while trying to separate Placement's databases out from Nova, I run into the the same issue with placement-status upgrade check:
>
> # placement-status upgrade check
> +-----------------------------------------------------------------+
> | Upgrade Check Results                                           |
> +-----------------------------------------------------------------+
> | Check: Missing Root Provider IDs                                |
> | Result: Success                                                 |
> | Details: None                                                   |
> +-----------------------------------------------------------------+
> | Check: Incomplete Consumers                                     |
> | Result: Warning                                                 |
> | Details: There are -3511 incomplete consumers table records for |
> |   existing allocations. Run the "placement-manage db            |
> |   online_data_migrations" command.                              |
> +-----------------------------------------------------------------+
>
> Likewise, running placement-manage db online_data_migrations does nothing.

Hm, in both cases I see that the status check is reporting a _negative_ number of consumers, which is unexpected. Looking at the code, this value is calculated by subtracting the number of 'consumers' [1] table records from the number of 'allocations' table consumers. The fact that this is negative means there are more 'consumers' table records with unique project/user pairs than there are 'allocations' table records with unique project/user pairs.

This implies to me that there are orphaned 'consumers' table records in your databases. We had a bug around this in the past [2] where 'consumers' table records were not being deleted when they no longer had any 'allocations' table records associated with them. That bug was fixed in the Rocky release but the fix does not clean up orphaned 'consumers' records that were orphaned before the fix landed. Is your original deployment from before Rocky and has been upgraded?

If so, then if you have orphaned 'consumers' records, you will need to do a one-time manual database clean up as described in this comment [3]. You can check whether you have any orphaned records by a SQL query like this:

select * from nova_api.consumers where nova_api.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);

If you do, after you clean them up manually, you should stop seeing the placement-status upgrade check warning.

Note that it is not so harmful to have orphaned 'consumers' in database in that it won't hurt the operation of nova other than taking up database space and possibly impacting database and nova performance as a result. But it's a good idea to clean them up. And >= Rocky 'consumers' are properly deleted by placement so orphans cannot happen from then on.

Hope this helps,
-melanie

[1] 'consumers' are project/user combinations of instance owners
[2] https://bugs.launchpad.net/nova/+bug/1780799
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1726256#c24