<html><head></head><body><div class="yahoo-style-wrap" style="font-family:times new roman, new york, times, serif;font-size:16px;"><div dir="ltr" data-setdir="false">Hi folks,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">We are seeing some weird issues with multiple compute nodes and would appreciate your thoughts.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Background:</div><div dir="ltr" data-setdir="false">We are on stable Queens.</div><div dir="ltr" data-setdir="false">As part of an upgrade to accomodate 3X more servers, we decided to add three more compute nodes</div><div dir="ltr" data-setdir="false">+ three more ICs for a total of 6 compute nodes and 6 ICs.</div><div dir="ltr" data-setdir="false">As soon as we added these in preparation for the 3X increase in servers I am seeing weird</div><div dir="ltr" data-setdir="false">behaviour. </div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">A general question to everyone:</div><div dir="ltr" data-setdir="false">How many of you run your baremetal clouds with 5+ computes and ICs?</div><div dir="ltr" data-setdir="false">Are things stable with the setup ?</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Logs and Analysis:</div><div dir="ltr" data-setdir="false">all compute and conductor services are up and running.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">1) Baremetal node <span><span style="color: rgb(0, 0, 0); font-family: "times new roman", "new york", times, serif; font-size: 16px;"> c1bda753-d46c-4379-8d07-7787c2a4a7f2 mapped to sc-ironic08</span></span></div><div dir="ltr" data-setdir="false"><div><div>root@stg-cl1-dev-001:~# openstack hypervisor show c1bda753-d46c-4379-8d07-7787c2a4a7f2 | grep ironic</div><div> |<br></div><div>| service_host | sc-ironic08.nvc.nvidia.com</div><div><br></div><div dir="ltr" data-setdir="false">2)Mac address is <span><span style="color: rgb(0, 0, 0); font-family: "times new roman", "new york", times, serif; font-size: 16px;">6c:b3:11:4f:8a:c0</span></span></div><div dir="ltr" data-setdir="false"><div><div>root@stg-cl1-dev-001:~# openstack baremetal port list --node c1bda753-d46c-4379-8d07-7787c2a4a7f2</div><div>+--------------------------------------+-------------------+</div><div>| UUID | Address |</div><div>+--------------------------------------+-------------------+</div><div>| a517fb41-f977-438d-8c0d-21046e2918d9 | 6c:b3:11:4f:8a:c0 |</div><div>+--------------------------------------+-------------------+</div><div><br></div></div><br></div><div dir="ltr" data-setdir="false"><br></div><div><br></div><div dir="ltr" data-setdir="false">2)Provisioning starts:</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">ironic06 receives the VIF update: WHY ?</div><div dir="ltr" data-setdir="false"><div><div>2020-04-21 15:05:47.509 71431 INFO ironic.conductor.manager VIF 657fea31-3218-4f10-b6ad-8b6a0fa7bab8 successfully attached to node c1bda753-d46c-4379-8d07-7787c2a4a7f2</div><div><br></div></div>ironic08 (correct one) also receives updates.</div><div dir="ltr" data-setdir="false"><div><div>[root@sc-ironic08 master_images]# tail -f /var/log/ironic/ironic-conductor.log | grep c1bda753-d46c-4379-8d07-7787c2a4a7f2</div><div>2020-04-21 15:08:04.943 27542 INFO ironic.conductor.task_manager [req-259b0175-65bc-4707-8c88-a65189a29954 - - - - -] Node c1bda753-d46c-4379-8d07-7787c2a4a7f2 moved to provision state "deploying" from state "wait call-back"; target provision state is "active"</div><div><br></div></div><br></div><div dir="ltr" data-setdir="false">For now we have backed down to 3 and are stable again but I would really like to overprovision our computes and conductors if possible.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Please let me know your thoughts and if anything rings a bell.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">thanks,</div><div dir="ltr" data-setdir="false">Fred.</div><div dir="ltr" data-setdir="false"><br></div><div><br></div></div><br></div></div></body></html>