Nova Ussuri: DB online migration cannot migrate (some) populate_user_id
Hi,
We upgraded one of our clouds to Ussuri (we are late!) a couple of months ago. After the upgrade 'nova-manage db online_data_migration' was run. fill_virtual_interface_list and populate_user_id where non zero and after running the command fill_virtual_interface_list was 0 but populate_user_id stayed 24. Looking in nova-manage.log, I found the follwoing message (twice every time the command is run):
Some instance mappings were not migratable. This may be transient due to in-flight instance builds, or could be due to stale data that will be cleaned up after running "nova-manage db archive_deleted_rows --purge".
I ran the suggest command but the result is that fill_virtual_interface_list is non zero again (15) and there is no change to populate_user_i. If I run again online_data_migration, it clears again fill_virtual_interface_list... and so on.
After 2 months the situation remains the same, no increase in populate_user_id but no possibility to fix them. We were wondering if it could have been caused by the fact that we had not yet upgraded all the compute nodes when we first ran the online_data_migration. We are planning to upgrade to Victoria and we'd like to fix the situation before as we understand that online data migration not done can be blocking for the next upgrade.
We don't have any clue on the reason this happened. It is a test cloud with a very low usage. On our production cloud we have a somewhat similar problem but populate_user_id=1 and running the purge doesn't increase fill_virtual_interface_list.
Thanks in advance for any hint! Best regards,
Michel
On 07/20/23 22:45, Michel Jouvin wrote:
Hi,
We upgraded one of our clouds to Ussuri (we are late!) a couple of months ago. After the upgrade 'nova-manage db online_data_migration' was run. fill_virtual_interface_list and populate_user_id where non zero and after running the command fill_virtual_interface_list was 0 but populate_user_id stayed 24. Looking in nova-manage.log, I found the follwoing message (twice every time the command is run):
Some instance mappings were not migratable. This may be transient due to in-flight instance builds, or could be due to stale data that will be cleaned up after running "nova-manage db archive_deleted_rows --purge".
I ran the suggest command but the result is that fill_virtual_interface_list is non zero again (15) and there is no change to populate_user_i. If I run again online_data_migration, it clears again fill_virtual_interface_list... and so on.
After 2 months the situation remains the same, no increase in populate_user_id but no possibility to fix them. We were wondering if it could have been caused by the fact that we had not yet upgraded all the compute nodes when we first ran the online_data_migration. We are planning to upgrade to Victoria and we'd like to fix the situation before as we understand that online data migration not done can be blocking for the next upgrade.
We don't have any clue on the reason this happened. It is a test cloud with a very low usage. On our production cloud we have a somewhat similar problem but populate_user_id=1 and running the purge doesn't increase fill_virtual_interface_list.
I'm not sure whether it's related but your post reminded me of this past bug:
https://bugs.launchpad.net/nova/+bug/1824435
so it might be worth checking whether you have a duplicate 'security_groups' table record in your database. If so, there is a workaround described in comment #28 on that bug report.
-melwitt
participants (2)
-
melanie witt
-
Michel Jouvin