<div dir="ltr"><div dir="ltr">Hi,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 22, 2020 at 5:59 PM <a href="mailto:fsbiz@yahoo.com">fsbiz@yahoo.com</a> <<a href="mailto:fsbiz@yahoo.com">fsbiz@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:16px"><div></div>
<div dir="ltr">Hi Arne,</div><div dir="ltr"><br></div><div dir="ltr">Thanks for responding. </div><div dir="ltr">Yes, it is definitely an issue with the hash ring.</div><div dir="ltr"><br></div><div dir="ltr">With Queens:</div><div dir="ltr">With 3 NCs and 3 ICs we are relatively stable.</div><div dir="ltr"><br></div><div dir="ltr">With 6 NCs/6ICs, it becomes pretty much unusable. There seems to be a race condition where 2 NCs</div><div dir="ltr">are competing with each other to get hold of the provisioning. </div></div></div></blockquote><div><br></div><div>I don't think the number of nova-compute instances is related to this problem. NC and IC are balanced independently, every request from Nova is re-balanced again in ironic-api.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:16px"><div dir="ltr"> A few transitions are</div><div dir="ltr">handled by one IC, then when another NC takes over some transitions are handled by its IC.</div><div dir="ltr">So we end up in scenarios where the image download happens one one IC but due to the competing</div><div dir="ltr">NCs another IC is entrusted with doing the ISCSI transfer down to the node. And the provision </div><div dir="ltr">fails because the image cannot be found.</div></div></div></blockquote><div><br></div><div>This is certainly concerning. When adding IC, do you wait for nodes to rebalance (it may take minutes)?</div><div><br></div><div>Dmitry<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:16px"><div dir="ltr"><br></div><div dir="ltr">Appreciate your quick response.</div><div dir="ltr"><br></div><div dir="ltr">Regards,</div><div dir="ltr">Fred.</div><div><br></div>
</div><div id="gmail-m_222189157651801389ydp1bda8e09yahoo_quoted_8047352530">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Wednesday, April 22, 2020, 12:41:44 AM PDT, Arne Wiebalck <<a href="mailto:arne.wiebalck@cern.ch" target="_blank">arne.wiebalck@cern.ch</a>> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr">Hi Fred,<br clear="none"><br clear="none">For quite a while we ran with 3 ICs and 1 NC to manage ~5000 nodes.<br clear="none"><br clear="none">Since this brings some scaling issues with resource tracking, we have<br clear="none">started to split things into conductor groups. Currently, we are at 6<br clear="none">ICs and 3 NCs, but the plan is to have 10 ICs with 10 NCs managing<br clear="none">groups of ~500 nodes.<br clear="none"><br clear="none">The ICs and the NCs will basically be mapped 1:1, rather than having<br clear="none">all NCs see all ICs. The reason is that in the past we saw issues with<br clear="none">the hash ring when the nodes were visible to all NCs, e.g. multiple<br clear="none">NCs were claiming overlapping set of nodes ... having multiple ICs per<br clear="none">group is not an issue, though.<br clear="none"><br clear="none">We are currently still on Stein, but it could well be that you hit this<br clear="none">issue as when you add more NCs, the nodes will be reshuffled.<br clear="none"><br clear="none">Cheers,<br clear="none"> Arne<br clear="none"><div id="gmail-m_222189157651801389ydp1bda8e09yqtfd73636"><br clear="none">On 22.04.20 00:32, <a shape="rect" href="mailto:fsbiz@yahoo.com" rel="nofollow" target="_blank">fsbiz@yahoo.com</a> wrote:<br clear="none">> Hi folks,<br clear="none">> <br clear="none">> We are seeing some weird issues with multiple compute nodes and would <br clear="none">> appreciate your thoughts.<br clear="none">> <br clear="none">> Background:<br clear="none">> We are on stable Queens.<br clear="none">> As part of an upgrade to accomodate 3X more servers, we decided to add <br clear="none">> three more compute nodes<br clear="none">> + three more ICs for a total of 6 compute nodes and 6 ICs.<br clear="none">> As soon as we added these in preparation for the 3X increase in servers <br clear="none">> I am seeing weird<br clear="none">> behaviour.<br clear="none">> <br clear="none">> A general question to everyone:<br clear="none">> How many of you run your baremetal clouds with 5+ computes and ICs?<br clear="none">> Are things stable with the setup ?<br clear="none">> <br clear="none">> Logs and Analysis:<br clear="none">> all compute and conductor services are up and running.<br clear="none">> <br clear="none">> 1) Baremetal node c1bda753-d46c-4379-8d07-7787c2a4a7f2 mapped to <br clear="none">> sc-ironic08<br clear="none">> <a shape="rect" href="mailto:root@stg-cl1-dev-001" rel="nofollow" target="_blank">root@stg-cl1-dev-001</a>:~# openstack hypervisor show <br clear="none">> c1bda753-d46c-4379-8d07-7787c2a4a7f2 | grep ironic<br clear="none">> |<br clear="none">> | service_host | <a href="http://sc-ironic08.nvc.nvidia.com" target="_blank">sc-ironic08.nvc.nvidia.com</a><br clear="none">> <br clear="none">> 2)Mac address is 6c:b3:11:4f:8a:c0<br clear="none">> <a shape="rect" href="mailto:root@stg-cl1-dev-001" rel="nofollow" target="_blank">root@stg-cl1-dev-001</a>:~# openstack baremetal port list --node <br clear="none">> c1bda753-d46c-4379-8d07-7787c2a4a7f2<br clear="none">> +--------------------------------------+-------------------+<br clear="none">> | UUID | Address |<br clear="none">> +--------------------------------------+-------------------+<br clear="none">> | a517fb41-f977-438d-8c0d-21046e2918d9 | 6c:b3:11:4f:8a:c0 |<br clear="none">> +--------------------------------------+-------------------+<br clear="none">> <br clear="none">> <br clear="none">> <br clear="none">> <br clear="none">> 2)Provisioning starts:<br clear="none">> <br clear="none">> ironic06 receives the VIF update: WHY ?<br clear="none">> 2020-04-21 15:05:47.509 71431 INFO ironic.conductor.manager VIF <br clear="none">> 657fea31-3218-4f10-b6ad-8b6a0fa7bab8 successfully attached to node <br clear="none">> c1bda753-d46c-4379-8d07-7787c2a4a7f2<br clear="none">> <br clear="none">> ironic08 (correct one) also receives updates.<br clear="none">> [<a shape="rect" href="mailto:root@sc-ironic08" rel="nofollow" target="_blank">root@sc-ironic08</a> master_images]# tail -f <br clear="none">> /var/log/ironic/ironic-conductor.log | grep <br clear="none">> c1bda753-d46c-4379-8d07-7787c2a4a7f2<br clear="none">> 2020-04-21 15:08:04.943 27542 INFO ironic.conductor.task_manager <br clear="none">> [req-259b0175-65bc-4707-8c88-a65189a29954 - - - - -] Node <br clear="none">> c1bda753-d46c-4379-8d07-7787c2a4a7f2 moved to provision state <br clear="none">> "deploying" from state "wait call-back"; target provision state is "active"<br clear="none">> <br clear="none">> <br clear="none">> For now we have backed down to 3 and are stable again but I would really <br clear="none">> like to overprovision our computes and conductors if possible.<br clear="none">> <br clear="none">> Please let me know your thoughts and if anything rings a bell.<br clear="none">> <br clear="none">> thanks,<br clear="none">> Fred.<br clear="none">> <br clear="none">> <br clear="none">> <br clear="none"><br clear="none"></div></div></div>
</div>
</div></div></blockquote></div></div>