On 10/30/20 01:37, Seth Tunstall wrote:
Hello,
On 10/28/20 12:01, melanie witt wrote:
The main idea of the row deletions is to delete "orphan" records
which are records tied to an instance's lifecycle when that instance no longer exists. Going forward, nova will delete these records itself at instance deletion time but did not in the past because of bugs, and any records generated before a bug was fixed will become orphaned once the associated instance is deleted.
I've done the following in this order:
nova-manage api_db sync
nova-manage db sync
(to bring the DBs up to the version I'm upgrading to (Train)
nova-manage db archive_deleted_rows --verbose --until-complete
The thing I notice here ^ is that you didn't (but should) use --all-cells to also clean up based on the nova_cell0 database (where instances that failed scheduling go). If you've ever had an instance go into ERROR state for failing the scheduling step and you deleted it, its nova_api.instance_mappings record would be a candidate for being archived (removed).
<snip>
# placement-status upgrade check +-----------------------------------------------------------------------+ | Upgrade Check Results | +-----------------------------------------------------------------------+ | Check: Missing Root Provider IDs | | Result: Success | | Details: None | +-----------------------------------------------------------------------+ | Check: Incomplete Consumers | | Result: Warning | | Details: There are -2 incomplete consumers table records for existing | | allocations. Run the "placement-manage db | | online_data_migrations" command. | +-----------------------------------------------------------------------+
argh! again a negative number! But at least it's only 2, which is well within the realm of manual fixes.
The only theory I have for how this occurred is you have 2 consumers that are orphaned due to missing the nova_cell0 during database archiving ... Like if you have a couple of deleted instances in nova_cell0 and thus still have nova_api.instance_mappings and without --all-cells those instance_mappings didn't get removed and so affected the manual cleanup query you ran (presence of instance_mappings prevented deletion of 2 orphaned consumers).
If that's not it, then I'm afraid I don't have any other ideas at the moment.
-melanie