<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Times New Roman","serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you for your comments and questions, Boris.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">We will test it asap and get back to you.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">--</span><span lang="EN" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Shane<o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Boris Pavlovic [mailto:boris@pavlovic.me]
<br>
<b>Sent:</b> Wednesday, July 31, 2013 7:00 PM<br>
<b>To:</b> OpenStack Development Mailing List<br>
<b>Subject:</b> Re: [openstack-dev] A simple way to improve nova scheduler<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Shane, <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks for implementing this one new approach. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Yes, I agree that it solves problems with "JOIN". <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">But now I am worry about new problem db.compute_node_update() that changes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">every time field with "TEXT" type which means that this should work really slow. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">So I have some question about testing time, did you test just joins or <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">joins with parallel N/60 updates/sec of compute_node_update() calls? <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Also we will need Russell confirmation to merge such a big change right before release. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Russell what do you think? <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">From what I know since we don't have a clear solution for this issue community agreed that it would be discussed on the coming summit.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Best regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Boris Pavlovic<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Mirantis Inc. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Wed, Jul 31, 2013 at 9:36 AM, Wang, Shane <<a href="mailto:shane.wang@intel.com" target="_blank">shane.wang@intel.com</a>> wrote:<o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">I have a patchset ready for your review
</span><span lang="EN-US" style="font-size:11.0pt"><a href="https://review.openstack.org/#/c/38802/" target="_blank">https://review.openstack.org/#/c/38802/</a></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">This patchset is to remove table compute_node_stats and add one more column “stats”
in table compute_nodes as JSON dict. With that, compute_node_get_all() doesn’t need to join another table when nova schedulers call it.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">My team has done some preliminary tests. The performance could be reduced to ~1.32 seconds
from ~16.89 seconds, where we suppose there are 10K compute nodes and each node has 20 stats records in compute_node_stats.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you for your review, and what do you think?</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">--</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Shane</span><span lang="EN-US"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Joshua
Harlow [mailto:<a href="mailto:harlowja@yahoo-inc.com" target="_blank">harlowja@yahoo-inc.com</a>]
<br>
<b>Sent:</b> Thursday, July 25, 2013 5:36 AM<br>
<b>To:</b> OpenStack Development Mailing List; Boris Pavlovic</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
<b>Subject:</b> Re: [openstack-dev] A simple way to improve nova scheduler<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">As far as the send only when you have to. That reminds me of this piece of work that could be resurrected
that slowed down the periodic updates when nothing was changing.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""><a href="https://review.openstack.org/#/c/26291" target="_blank">https://review.openstack.org/#/c/26291</a>/</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">Could be brought back, the concept still feels useful imho. But maybe not to others :-P</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:
</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Boris Pavlovic <<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a>><br>
<b>Reply-To: </b>OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<b>Date: </b>Wednesday, July 24, 2013 12:12 PM<br>
<b>To: </b>OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [openstack-dev] A simple way to improve nova scheduler</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">Hi Mike,</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">On Wed, Jul 24, 2013 at 1:01 AM, Mike Wilson <<a href="mailto:geekinutah@gmail.com" target="_blank">geekinutah@gmail.com</a>>
wrote:</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">Again I can only speak for qpid, but it's not really a big load on the qpidd server itself. I think
the issue is that the updates come in serially into each scheduler that you have running. We don't process those quickly enough for it to do any good, which is why the lookup from db. You can see this for yourself using the fake hypervisor, launch yourself
a bunch of simulated nova-compute, launch a nova-scheduler on the same host and even with 1k or so you will notice the latency between the update being sent and the update actually meaning anything for the scheduler.
</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">I think a few points that have been brought up could mitigate this quite a bit. My personal view is
the following:</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">-Only update when you have to (ie. 10k nodes all sending update every periodic interval is heavy, only
send when you have to)</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">-Don't fanout to schedulers, update a single scheduler which in turn updates a shared store that is
fast such as memcache</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">I guess that effectively is what you are proposing with the added twist of the shared store.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">Absolutely agree with this. Especially with using memcached (or redis) as common storage for all schedulers. </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">Best regards,</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">Boris Pavlovic</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">---</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif"">Mirantis Inc. </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:8.5pt;font-family:"Calibri","sans-serif""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>