On Sat, 2024-04-13 at 12:52 +0000, Michael Sherman wrote:
Hi all,
We’re running into an issue, where for two sites with 150-250 ironic nodes on a single conductor and nova-compute instance, we’ve started to get “no hosts available” errors from nova scheduler.
We’re using the blazar-nova filter to match on hosts in specifically tagged aggregates. After adding some debug logs, I found that the “host_state” object passed to the filter seems to have out-of-date aggregate information.
so first thing to be aware of is that host aggrates are not support with ironic virt driver. until the the caracal release the ironic virt driver uses a hash ring to balance comptue nodes between compute services with amoung other things broke host aggrates. From a nova project perspective using hostaggrates with ironic compute services is unsupported.
in caracal they might now work when ironic sharding is used. host aggrates are used to map compute servivices not compute nodes to an aggrate so when using shards you can map a given shard to a host aggrated by mapping the compute service for that shard to an aggrate.
Specifically, if I query the system with “openstack aggregate show …” or “openstack allocation candidate list”, I see the correct aggregate for the nodes in question, but the contents of “host_state” reflect a previous state.
This “staleness” does not seem to correct itself over time, but is resolved by restarting the nova-scheduler process (actually restarting the kolla docker container, but the same effect). However the issues return over the course of a couple hours.
this is likely caused by a combination of the hash ring and the host cache. again your current toplogy is unsupproted as we do not offically supprot using host aggrates with ironic nodes. with that said you could try disabling the cacheing by setting https://docs.openstack.org/nova/latest/configuration/config.html#filter_sche... on the scheduler and all compute services.
that may or may not work depending on why but my guess is its becauses the compute nodes that are "stale" have been reblance by the hash ring.
the other way to work around this might eb to ensure you do not use peer list and have exactly one compute service per conductor group. again im not sure if that will work because your trying to use a feature (host aggarates) that is not supported by the ironic virt dirver but it might mitigate the incompatiblity in older releases.
We haven’t increased the number of nodes, or otherwise changed the hardware, so I’m not sure what could have triggered this issue.
Any advice on further debugging steps would be greatly appreciated. Thank you!
-- Michael Sherman Infrastructure Lead – Chameleon Computer Science, University of Chicago MCS, Argonne National Lab