Open Stack

On 10/25/20 03:10, CHANU ROMAIN wrote:
> Hello,
> 
> I tried your procedure it didnt remove the warning from "placement-status upgrade check".
> 
> As you said I did:
> delete from nova_api.consumers where nova_api.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);
> 
> it removed only 2458 rows, far far away from my -15753 consumers.
> 
> I restored my backup and tried what you said here
> https://bugzilla.redhat.com/show_bug.cgi?id=1726256#c24
> 
> update instance_id_mappings set deleted = id where uuid in (select uuid from instances where deleted != 0);
> nova-manage db archive_deleted_rows --until-complete
> 
> delete from instance_id_mappings where uuid not in (select uuid from instances);
> (removed 70 rows)
> 
> then I tried again:
> delete from nova_api.consumers where nova_api.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);
> (deleted around 15900 rows)
> 
> Sadly, placement-status upgrade check still returns  (There are -15753 incomplete consumers...)

> I restored my backup.
> 
> It does not fix the bug but should i still delete the rows like I did? Do you have any other lead to fix it? Did I miss something?

OK, if you did all of that and the value "-15753 incomplete consumers" 
has not changed at all, this sounds like your 'placement-status upgrade 
check' command is running on a different database than 'nova_api'. Did 
you already split out the placement database into a different database?

If so, then you would have to do the cleanups with, for example: 'delete 
from placement.consumers where placement.consumers.uuid not in (select 
nova_api.instance_mappings.instance_uuid from 
nova_api.instance_mappings);" if you have split the placement data into 
a new database called 'placement'. The queries written in that bug were 
done as though nova and placement are still sharing the 'nova_api' 
database. The 'consumers' table is owned by placement. The rest of the 
mentioned tables belong to nova.

The main idea of the row deletions is to delete "orphan" records which 
are records tied to an instance's lifecycle when that instance no longer 
exists. Going forward, nova will delete these records itself at instance 
deletion time but did not in the past because of bugs, and any records 
generated before a bug was fixed will become orphaned once the 
associated instance is deleted.

-melanie

> ________________________________________
> From: melanie witt <melwittt at gmail.com>
> Sent: Friday, October 23, 2020 5:17 PM
> To: Seth Tunstall; openstack-discuss at lists.openstack.org
> Cc: CHANU ROMAIN
> Subject: Re: [placement] Train upgrade warning
> 
> On 10/23/20 02:43, Seth Tunstall wrote:
>> Hello,
>>
>> As another person has mentioned (Chanu Romain), I am attempting an upgrade from Queens to Train. I have successfully upgraded Keystone and Glance, but while trying to separate Placement's databases out from Nova, I run into the the same issue with placement-status upgrade check:
>>
>> # placement-status upgrade check
>> +-----------------------------------------------------------------+
>> | Upgrade Check Results                                           |
>> +-----------------------------------------------------------------+
>> | Check: Missing Root Provider IDs                                |
>> | Result: Success                                                 |
>> | Details: None                                                   |
>> +-----------------------------------------------------------------+
>> | Check: Incomplete Consumers                                     |
>> | Result: Warning                                                 |
>> | Details: There are -3511 incomplete consumers table records for |
>> |   existing allocations. Run the "placement-manage db            |
>> |   online_data_migrations" command.                              |
>> +-----------------------------------------------------------------+
>>
>> Likewise, running placement-manage db online_data_migrations does nothing.
> 
> Hm, in both cases I see that the status check is reporting a _negative_ number of consumers, which is unexpected. Looking at the code, this value is calculated by subtracting the number of 'consumers' [1] table records from the number of 'allocations' table consumers. The fact that this is negative means there are more 'consumers' table records with unique project/user pairs than there are 'allocations' table records with unique project/user pairs.
> 
> This implies to me that there are orphaned 'consumers' table records in your databases. We had a bug around this in the past [2] where 'consumers' table records were not being deleted when they no longer had any 'allocations' table records associated with them. That bug was fixed in the Rocky release but the fix does not clean up orphaned 'consumers' records that were orphaned before the fix landed. Is your original deployment from before Rocky and has been upgraded?
> 
> If so, then if you have orphaned 'consumers' records, you will need to do a one-time manual database clean up as described in this comment [3]. You can check whether you have any orphaned records by a SQL query like this:
> 
> select * from nova_api.consumers where nova_api.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings);
> 
> If you do, after you clean them up manually, you should stop seeing the placement-status upgrade check warning.
> 
> Note that it is not so harmful to have orphaned 'consumers' in database in that it won't hurt the operation of nova other than taking up database space and possibly impacting database and nova performance as a result. But it's a good idea to clean them up. And >= Rocky 'consumers' are properly deleted by placement so orphans cannot happen from then on.
> 
> Hope this helps,
> -melanie
> 
> [1] 'consumers' are project/user combinations of instance owners
> [2] https://bugs.launchpad.net/nova/+bug/1780799
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1726256#c24
>