<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Adrian,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If the reason to avoid leader election is because it is complicated and error prone, this argument may not be true. Leader election is complicated in a pure
distributed system in which there is no centralized storage. However, Magnum has a centralized database, so it is possible to implement a very simple leader election algorithm. For example, we can let each conductor register itself in the DB and elect the
first registered conductor to be the leader. How about that?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hongbin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Adrian Otto [mailto:adrian.otto@rackspace.com]
<br>
<b>Sent:</b> August-03-15 12:49 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [Magnum]horizontal scalability<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Aug 2, 2015, at 7:40 PM, <span lang="ZH-CN" style="font-family:SimSun">
王华</span> <<a href="mailto:wanghua.humble@gmail.com">wanghua.humble@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi all, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_450">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">As discussed in the Vancouver Summit, we are going to drop the bay lock implementation. Instead, each conductor will call Heat concurrently and rely on heat for concurrency control.
However, I think we need an approach for state convergence from heat to magnum. Either periodic task [1] or heat notification [2] looks like a candidate.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_457">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">[1] <a href="https://blueprints.launchpad.net/magnum/+spec/add-periodic-task"><span style="color:#0033AA;text-decoration:none">https://blueprints.launchpad.net/magnum/+spec/add-periodic-task</span></a><br>
[2] <a href="http://lists.openstack.org/pipermail/openstack-dev/2015-March/058898.html"><span style="color:#0033AA;text-decoration:none">http://lists.openstack.org/pipermail/openstack-dev/2015-March/058898.html</span></a><br>
--hongbin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_457">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">If we use periodic task to sync state from heat to magnum, I think we should make periodic task a independent process and magnum-conductor only operates heat stack.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_457">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">How to make periodic task high available</span><span lang="ZH-CN" style="font-size:9.0pt;font-family:SimSun;color:#333333">？</span><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_457">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">1.We can run several periodic tasks.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_457">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">2.Or we can use a leader selection mechanism to make only a periodic task running<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_457">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">and other periodic tasks waiting.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:14.4pt;line-height:13.5pt;max-width:60em" id="yui_3_10_3_1_1438566042119_457">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">Shall we make periodic task a independent process? How to make periodic task high available? <o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal">Good question. The traditional solution for handling this in a distributed system is to hold a leader election. The elected leader is responsible for dispatching the job to a queue that one available worker will pick up and run. However,
that may not actually be needed in our case. Consider the question: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What harm will come if all master nodes in the cluster perform the same periodic task?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One drawback is that more resources will be consumed than necessary (efficiency). As long as updates to the Magnum database are transactional, having concurrent updates to the same bay is actually not something we expect would result in
data corruption. Worst case the same update would be processed multiple times. The advantage of using this approach is that we would not need to implement any form of leader selection. This would keep our implementation simpler, and less error prone.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We could still supervise each periodic task process so if they end up crashing on each node, they would be restarted. This is simple to do from a patent process that calls os.wait() on the child task.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thoughts?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Adrian<o:p></o:p></p>
</div>
</div>
</body>
</html>