Open Stack

On 10/30/20 01:37, Seth Tunstall wrote:
> Hello,
> 
> On 10/28/20 12:01, melanie witt wrote:
>  >> The main idea of the row deletions is to delete "orphan" records 
> which are records tied to an instance's lifecycle when that instance no 
> longer exists. Going forward, nova will delete these records itself at 
> instance deletion time but did not in the past because of bugs, and any 
> records generated before a bug was fixed will become orphaned once the 
> associated instance is deleted.
> 
> I've done the following in this order:
> 
> nova-manage api_db sync
> 
> nova-manage db sync
> 
> (to bring the DBs up to the version I'm upgrading to (Train)
> 
> nova-manage db archive_deleted_rows --verbose --until-complete

The thing I notice here ^ is that you didn't (but should) use 
--all-cells to also clean up based on the nova_cell0 database (where 
instances that failed scheduling go). If you've ever had an instance go 
into ERROR state for failing the scheduling step and you deleted it, its 
nova_api.instance_mappings record would be a candidate for being 
archived (removed).

<snip>

> # placement-status upgrade check
> +-----------------------------------------------------------------------+
> | Upgrade Check Results |
> +-----------------------------------------------------------------------+
> | Check: Missing Root Provider IDs |
> | Result: Success |
> | Details: None |
> +-----------------------------------------------------------------------+
> | Check: Incomplete Consumers |
> | Result: Warning |
> | Details: There are -2 incomplete consumers table records for existing |
> | allocations. Run the "placement-manage db |
> | online_data_migrations" command. |
> +-----------------------------------------------------------------------+
> 
> argh! again a negative number! But at least it's only 2, which is well 
> within the realm of manual fixes.

The only theory I have for how this occurred is you have 2 consumers 
that are orphaned due to missing the nova_cell0 during database 
archiving ... Like if you have a couple of deleted instances in 
nova_cell0 and thus still have nova_api.instance_mappings and without 
--all-cells those instance_mappings didn't get removed and so affected 
the manual cleanup query you ran (presence of instance_mappings 
prevented deletion of 2 orphaned consumers).

If that's not it, then I'm afraid I don't have any other ideas at the 
moment.

-melanie