[nova][placement] Incomplete Consumers return negative value after upgrade
Hi,
after an upgrade from Stein to Train, I hit an issue with negative value during upgrade check for placement:
# placement-status upgrade check+------------------------------------------------------------------+| Upgrade Check Results |+------------------------------------------------------------------+| Check: Missing Root Provider IDs || Result: Success || Details: None |+------------------------------------------------------------------+| Check: Incomplete Consumers || Result: Warning || Details: There are -20136 incomplete consumers table records for || existing allocations. Run the "placement-manage db || online_data_migrations" command. |+------------------------------------------------------------------+
Seems that negative value is a result that I get such values from tables consumer and allocations:
mysql> select count(id), consumer_id from allocations group by consumer_id;...1035 rows in set (0.00 sec)
mysql> select count(*) from consumers;+----------+| count(*) |+----------+| 21171 |+----------+1 row in set (0.04 sec)
Unfortunately such warning cannot be solved by execution of suggested command( placement-manage db online_data_migrations) as it seems it adds records to consumers table - not to allocations, which looks like to be a problem here. I was following recommendations from this discussion: http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018536.... but unfortunately it doesn't solve the issue(even not changing a negative value). I'm just wondering if I skipped something important and you can suggest some (obvious?) solution.
Thank you in advance for your time and help. Best regards, Jan
On Thu May 05 2022 02:31:16 GMT-0700 (Pacific Daylight Time), Jan Wasilewski finarffin@gmail.com wrote:
Hi,
after an upgrade from Stein to Train, I hit an issue with negative value during upgrade check for placement:
# placement-status upgrade check +------------------------------------------------------------------+ | Upgrade Check Results | +------------------------------------------------------------------+ | Check: Missing Root Provider IDs | | Result: Success | | Details: None | +------------------------------------------------------------------+ | Check: Incomplete Consumers | | Result: Warning | | Details: There are -20136 incomplete consumers table records for | | existing allocations. Run the "placement-manage db | | online_data_migrations" command. | +------------------------------------------------------------------+
Seems that negative value is a result that I get such values from tables consumer and allocations:
mysql> select count(id), consumer_id from allocations group by consumer_id; ... 1035 rows in set (0.00 sec)
mysql> select count(*) from consumers; +----------+ | count(*) | +----------+ | 21171 | +----------+ 1 row in set (0.04 sec)
Unfortunately such warning cannot be solved by execution of suggested command( |placement-manage db online_data_migrations|) as it seems it adds records to consumers table - not to allocations, which looks like to be a problem here. I was following recommendations from this discussion: http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018536.... http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018536.html but unfortunately it doesn't solve the issue(even not changing a negative value). I'm just wondering if I skipped something important and you can suggest some (obvious?) solution.
I think you need the following patch which changes the logic of the upgrade check to ignore orphaned consumers which are not relevant to the upgrade:
https://review.opendev.org/c/openstack/placement/+/840704
The above originally landed in Xena but was overlooked for backport to older versions. I have proposed backports just now. If you can apply this patch for 'placement-status' and the check passes afterward, you are good.
You may still have orphaned consumers table records in the placement database but they don't hurt anything. If you want/need to clean them up, it has to be done manually, something like (disclaimer I did not test this):
delete from placement.consumers where placement.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings where nova_api.instance_mappings.queued_for_delete = true);
HTH, -melanie
On Thu May 05 2022 09:58:10 GMT-0700 (Pacific Daylight Time), melanie witt melwittt@gmail.com wrote:
You may still have orphaned consumers table records in the placement database but they don't hurt anything. If you want/need to clean them up, it has to be done manually, something like (disclaimer I did not test this):
delete from placement.consumers where placement.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings where nova_api.instance_mappings.queued_for_delete = true);
Sorry I got the where condition backwards, it should be:
delete from placement.consumers where placement.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings where nova_api.instance_mappings.queued_for_delete != true);
And you might have s/placement/nova_api/ if you did not break out the separate placement database.
-melanie
Hello Melanie,
Thank you for your reply. It seems that everything is OK right now, after applying a patch I can see:
# placement-status upgrade check+----------------------------------+| Upgrade Check Results |+----------------------------------+| Check: Missing Root Provider IDs || Result: Success || Details: None |+----------------------------------+| Check: Incomplete Consumers || Result: Success || Details: None |+----------------------------------+
I've seen that zuul added -1 to review for this check, however I will apply my internal patch for it, additionally that you mentioned that such orphaned records shouldn't hurt me. Once again - thank you for your help.
Best regards, Jan
czw., 5 maj 2022 o 19:05 melanie witt melwittt@gmail.com napisał(a):
On Thu May 05 2022 09:58:10 GMT-0700 (Pacific Daylight Time), melanie witt melwittt@gmail.com wrote:
You may still have orphaned consumers table records in the placement database but they don't hurt anything. If you want/need to clean them up, it has to be done manually, something like (disclaimer I did not test this):
delete from placement.consumers where placement.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings where nova_api.instance_mappings.queued_for_delete = true);
Sorry I got the where condition backwards, it should be:
delete from placement.consumers where placement.consumers.uuid not in (select nova_api.instance_mappings.instance_uuid from nova_api.instance_mappings where nova_api.instance_mappings.queued_for_delete != true);
And you might have s/placement/nova_api/ if you did not break out the separate placement database.
-melanie
participants (2)
-
Jan Wasilewski
-
melanie witt