<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hey OpenStackers,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I work on a cloud that allows users to reserve and provision bare metal instances with Ironic. We recently performed a long-overdue upgrade of our core components, all the way from Ocata up through Rocky. During this, we noticed that instance build requests
were taking 4-5x (!!) as long as before. We have two deployments, one with ~150 bare metal nodes, and another with ~300. These are each managed by one nova-compute process running the Ironic driver.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
After investigation, the root cause appeared to be contention between the update_resources periodic task and the instance claim step. There is one semaphore "compute_resources" that is used to control every access within the resource_tracker. In our case, what
was happening was the update_resources job, which runs every minute by default, was constantly queuing up accesses to this semaphore, because each hypervisor is updated independently, in series. This meant that, for us, each Ironic node was being processed
and was holding the semaphore during its update (which took about 2-5 seconds in practice.) Multiply this by 150 and our update task was running constantly. Because an instance claim also needs to access this semaphore, this led to instances getting stuck
in the "Build" state, after scheduling, for tens of minutes on average. There seemed to be some probabilistic effect here, which I hypothesize is related to the locking mechanism not using a "fair" lock (first-come, first-served) by default.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Our fix was to drastically increase the interval this task runs at--from every 1 minute to every 12 hours. We only provision bare metal, so my rationale was that the periodic full resource sync was less important and mostly helpful for fixing weird things where
somehow Placement's state got out of sync with Nova's somehow.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I'm wondering, after all this, if it makes sense to rethink this one-semaphore thing, and instead create a per-hypervisor semaphore when doing the resource syncing. I can't think of a reason why the entire set of hypervisors needs to be considered as a whole
when doing this task, but I could very well be missing something.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b>TL;DR</b>: if you have one nova-compute process managing lots of Ironic hypervisors, consider tweaking the update_resources_interval to a higher value, especially if you're seeing instances stuck in the Build state for a while.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Cheers,<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
<span style="background-color:white"><font size="1" face="Helvetica"><span style="font-size:12px; background-color:white; font-weight:normal; text-transform:none"><font size="2" face="Calibri,Helvetica,sans-serif"><span dir="ltr" id="divtagdefaultwrapper" style="font-size:10pt"><font size="3"><span id="divtagdefaultwrapper" style="font-size:12pt"></span></font></span></font></span></font></span></p>
<font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3">
<div style="margin-top:0; margin-bottom:0"><font size="2"><span style="font-size:10pt"><br>
</span></font></div>
</font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3">
<div style="margin-top:0; margin-bottom:0"><font size="2"><span style="font-size:12pt">Jason Anderson</span></font></div>
</font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3">
<div style="margin-top:0; margin-bottom:0"><font size="2"><span style="font-size:12pt"></span></font></div>
</font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3"></font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3">
<div style="margin-top:0; margin-bottom:0"><font size="2"><span style="font-size:10pt"><br>
</span></font></div>
<div style="margin-top:0; margin-bottom:0"><font size="2"><span style="font-size:10pt">Cloud Computing Software Developer</span></font></div>
</font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3">
<div style="margin-top:0; margin-bottom:0"></div>
</font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3"></font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3">
<div style="margin-top:0; margin-bottom:0"><font size="2"><span style="font-size:9pt"><b><span id="ms-rterangepaste-start"></span><span class="st">Consortium for Advanced Science and Engineering</span><span id="ms-rterangepaste-end"></span>, The University
of Chicago</b></span></font></div>
</font></font></font><font size="1" face="Helvetica"><font size="2" face="Calibri,Helvetica,sans-serif"><font size="3">
<div style="margin-top:0; margin-bottom:0"><font size="2"><span style="font-size:9pt"><b>Mathematics & Computer Science Division, Argonne National Laboratory</b></span></font></div>
</font></font></font><br>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px">
</p>
</div>
</div>
</body>
</html>