<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"> The feedback in the Nova – Cyborg interaction spec [1] is to move the call for creating/binding accelerator requests (ARQs) from the conductor (just before the call to build_and_run_instance, [2]) to the compute manager (just before
spawn, without holding the build sempahore [3]). The point where the results of the bind are needed is in the virt driver [4] – that is not changing. The reason for the move is to enable Cyborg to notify Nova [5] instead of Nova virt driver polling Cyborg,
thus making the interaction similar to other services like Neutron.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The binding involves device preparation by Cyborg, which may take some time (ballpark: milliseconds to few seconds to perhaps 10s of seconds – of course devices vary a lot). We want to overlap as much of this as possible with other tasks,
by starting the binding as early as possible and making it asynchronous, so that bulk VM creation rate etc. are not affected. These considerations are probably specific to Cyborg, so trying to make it uniform with other projects deserve a closer look before
we commit to it.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Moving the binding from [2] to [3] reduces this overlap. I did some measurements of the time window from [2] to [3]: it was consistently between 20 and 50 milliseconds, whether I launched 1 VM at a time, 2 at a time, etc. This seems acceptable.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But this was just in a two-node deployment. Are there situations where this window could get much larger (thus reducing the overlap)? Such as in larger deployments, or issues with RabbitMQ messaging, etc. Are there larger considerations
of performance or scaling for this approach?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks in advance.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://review.opendev.org/#/c/603955/">https://review.opendev.org/#/c/603955/</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1501">
https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1501</a><o:p></o:p></p>
<p class="MsoNormal">[3] <a href="https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1882">
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1882</a> <o:p>
</o:p></p>
<p class="MsoNormal">[4] <a href="https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3215">
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3215</a>
<o:p></o:p></p>
<p class="MsoNormal">[5] <a href="https://wiki.openstack.org/wiki/Nova/ExternalEventAPI">
https://wiki.openstack.org/wiki/Nova/ExternalEventAPI</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Sundar<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>