On Sat, 2024-04-13 at 12:52 +0000, Michael Sherman wrote:
> Hi all,
>
> We’re running into an issue, where for two sites with 150-250 ironic nodes on a single conductor and nova-compute
> instance, we’ve started to get “no hosts available” errors from nova scheduler.
>
> We’re using the blazar-nova filter to match on hosts in specifically tagged aggregates. After adding some debug logs,
> I found that the “host_state” object passed to the filter seems to have out-of-date aggregate information.
so first thing to be aware of is that host aggrates are not support with ironic virt driver.
until the the caracal release the ironic virt driver uses a hash ring to balance comptue nodes between compute
services with amoung other things broke host aggrates. From a nova project perspective using
hostaggrates with ironic compute services is unsupported.
in caracal they might now work when ironic sharding is used.
host aggrates are used to map compute servivices not compute nodes to an aggrate
so when using shards you can map a given shard to a host aggrated by mapping the compute service for that shard to an
aggrate.
>
> Specifically, if I query the system with “openstack aggregate show …” or “openstack allocation candidate list”, I see
> the correct aggregate for the nodes in question, but the contents of “host_state” reflect a previous state.
>
> This “staleness” does not seem to correct itself over time, but is resolved by restarting the nova-scheduler process
> (actually restarting the kolla docker container, but the same effect). However the issues return over the course of a
> couple hours.
this is likely caused by a combination of the hash ring and the host cache.
again your current toplogy is unsupproted as we do not offically supprot using host aggrates with ironic nodes.
with that said you could try disabling the cacheing by setting
https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes=false on
the scheduler and all compute services.
that may or may not work depending on why but my guess is its becauses the compute nodes that are "stale" have been
reblance by the hash ring.
the other way to work around this might eb to ensure you do not use peer list and have exactly one compute service per
conductor group. again im not sure if that will work because your trying to use a feature (host aggarates) that is not
supported by the ironic virt dirver but it might mitigate the incompatiblity in older releases.
>
> We haven’t increased the number of nodes, or otherwise changed the hardware, so I’m not sure what could have triggered
> this issue.
>
> Any advice on further debugging steps would be greatly appreciated. Thank you!
>
> --
> Michael Sherman
> Infrastructure Lead – Chameleon
> Computer Science, University of Chicago
> MCS, Argonne National Lab