Thanks Arne and Julia with the great suggestions on scaling ironic nodes.

We are currently trying to root cause an issue (it has occured twice) where a large number of nodes
(but not all the nodes) suddenly migrate from one IC to another.

E.g.
69 nodes moved from sc-ironic04 and sc-ironic05 to sc-ironic06 from 21:07 to 21:10 on nov. 23rd.

[root@sc-ironic06 nova]# grep "moving from" /var/log/nova/nova-compute.log-20191124

2019-11-23 21:07:46.606 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode 1cb9ef2e-aa7d-4e25-8878-14669a3ead7a moving from sc-ironic05.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com

2019-11-23 21:08:17.518 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode 56e58642-12ac-4455-bc95-2a328198f845 moving from sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com

2019-11-23 21:08:35.843 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode e0b9b94c-2ea3-4324-a85f-645d572e370b moving from sc-ironic05.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com

2019-11-23 21:08:42.264 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode 1c7d461c-2de7-4d9a-beff-dcb490c7b2e4 moving from sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com

2019-11-23 21:08:43.819 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode 73ed8bd4-23c2-46bc-b748-e6f5ab6fa932 moving from sc-ironic05.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com

2019-11-23 21:08:45.651 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode 51da1570-5666-4a21-a46f-4b7510d28415 moving from sc-ironic05.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com

2019-11-23 21:08:46.905 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode 38b41797-4b97-405b-bbd5-fccc61d237c3 moving from sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com

2019-11-23 21:08:49.065 210241 INFO nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode c5c89749-a11c-4eb8-b159-e8d47ecfcbb9 moving from sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com


Restarting nova-compute and ironic-conductor services on the IC seems to have fixed the issue but we are still in the root cause analysis phase 
and seem to have hit a wall narrowing this down.  Any suggestions are welcome.

Thanks,
Fred.


On Wednesday, October 30, 2019, 02:02:42 PM PDT, Arne Wiebalck <arne.wiebalck@cern.ch> wrote:


Hi Fred,

To confirm what Julia said:

We currently have ~3700 physical nodes in Ironic, managed by 3 controllers
(16GB VMs running httpd, conductor, and inspector). We recently moved to
l