<html><head></head><body><div class="ydp334821aayahoo-style-wrap" style="font-family:times new roman, new york, times, serif;font-size:16px;"><div></div>
<div dir="ltr" data-setdir="false">Thanks Arne and Julia with the great suggestions on scaling ironic nodes.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">We are currently trying to root cause an issue (it has occured twice) where a large number of nodes</div><div dir="ltr" data-setdir="false">(but not all the nodes) suddenly migrate from one IC to another.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false">E.g.</div><div>69 nodes moved from sc-ironic04 and sc-ironic05 to
sc-ironic06 from 21:07 to 21:10 on nov. 23<sup>rd</sup>.</div>
<p class="ydp9344aa81MsoNormal">[root@sc-ironic06 nova]# grep "moving from" /var/log/nova/nova-compute.log-20191124<br></p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:07:46.606 210241 INFO
nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - -
-] ComputeNode 1cb9ef2e-aa7d-4e25-8878-14669a3ead7a moving from
sc-ironic05.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com</p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:08:17.518 210241 INFO
nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - -
-] ComputeNode 56e58642-12ac-4455-bc95-2a328198f845 moving from
sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com</p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:08:35.843 210241 INFO
nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - -
-] ComputeNode e0b9b94c-2ea3-4324-a85f-645d572e370b moving from
sc-ironic05.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com</p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:08:42.264 210241 INFO
nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - -
-] ComputeNode 1c7d461c-2de7-4d9a-beff-dcb490c7b2e4 moving from
sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com</p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:08:43.819 210241 INFO nova.compute.resource_tracker
[req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode
73ed8bd4-23c2-46bc-b748-e6f5ab6fa932 moving from sc-ironic05.nvc.nvidia.com to
sc-ironic06.nvc.nvidia.com</p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:08:45.651 210241 INFO nova.compute.resource_tracker
[req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - - -] ComputeNode
51da1570-5666-4a21-a46f-4b7510d28415 moving from sc-ironic05.nvc.nvidia.com to
sc-ironic06.nvc.nvidia.com</p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:08:46.905 210241 INFO
nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - -
-] ComputeNode 38b41797-4b97-405b-bbd5-fccc61d237c3 moving from
sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com</p>
<p class="ydp9344aa81MsoNormal">2019-11-23 21:08:49.065 210241 INFO
nova.compute.resource_tracker [req-96baf341-0ecb-4dec-a204-32c2f77f3f64 - - - -
-] ComputeNode c5c89749-a11c-4eb8-b159-e8d47ecfcbb9 moving from
sc-ironic04.nvc.nvidia.com to sc-ironic06.nvc.nvidia.com</p></div><br></div><div dir="ltr" data-setdir="false">Restarting nova-compute and ironic-conductor services on the IC seems to have fixed the issue but we are still in the root cause analysis phase </div><div dir="ltr" data-setdir="false">and seem to have hit a wall narrowing this down. Any suggestions are welcome.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thanks,</div><div dir="ltr" data-setdir="false">Fred.</div><div dir="ltr" data-setdir="false"><br></div><div><br></div>
</div><div id="ydp95c13e97yahoo_quoted_5240077867" class="ydp95c13e97yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Wednesday, October 30, 2019, 02:02:42 PM PDT, Arne Wiebalck <arne.wiebalck@cern.ch> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr">Hi Fred,<br clear="none"><br clear="none">To confirm what Julia said:<br clear="none"><br clear="none">We currently have ~3700 physical nodes in Ironic, managed by 3 controllers<br clear="none">(16GB VMs running httpd, conductor, and inspector). We recently moved to<br clear="none">l<div class="ydp95c13e97yqt9208168125" id="ydp95c13e97yqtfd40896"><br clear="none"></div></div></div>
</div>
</div></body></html>