<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
font-size:10.5pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">>Nova now can detect host unreachable. But it fails to make out host isolation, host dead and nova compute service down. When host unreachable
is reported, users have to find out the exact state by himself and then take the appropriate measure to recover. Therefore we’d like to improve the host detection for nova.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I guess this depends on the service group driver that you use. For example if you use the DB driver, then there is a thread running on the compute manager that periodically updates the “alive”
status – which included both a liveness check (to the extent that the thread is still running) of the compute manager and that it can contact the DB. If the compute manager is using conductor then it also includes implicitly a check that the compute manager
can talk to MQ (a nice side effect of conductor – as before a node could be “Up” because it could talk to the DB but not able to process any messages)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">So to me the DB driver kind of already covers “send network heartbeat to the central agent and writes timestamp in shared storage periodically” – so maybe this is more of a specific ServiceGroup
Driver issue rather than a generic ServiceGroup change ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Phil<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" align="left" style="text-align:left"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Jiangying (Jenny) [mailto:jenny.jiangying@huawei.com]
<br>
<b>Sent:</b> 28 April 2014 13:31<br>
<b>To:</b> openstack-dev@lists.openstack.org<br>
<b>Subject:</b> [openstack-dev] [nova][service group]improve host state detection<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" align="left" style="text-align:left"><o:p> </o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">Nova now can detect host unreachable. But it fails to make out host isolation, host dead and nova compute service down. When host unreachable
is reported, users have to find out the exact state by himself and then take the appropriate measure to recover. Therefore we’d like to improve the host detection for nova.<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">Currently the service group API factors out the host detection and makes it a set of abstract internal APIs with a pluggable backend implementation.
The backend we designed is as follows:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">A detection central agent is introduced. When a member joins into the service group, the member host starts to send network heartbeat to the
central agent and writes timestamp in shared storage periodically. When the central agent stops receiving the network heartbeats from a member, it pings the member and checks the storage heartbeat before declaring the host to have failed.
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">----------------------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">network heartbeat|network ping|storage heartbeat| state | reason<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">------------------------|-----------------|------------------------|---------------------------|------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"> OK | - | - | Running | -<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"> Not OK | Not OK | Not OK | Dead | hardware failure/abnormal host shut down<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"> Not OK | OK | Not OK | Service unreachable| service process crashed<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"> Not OK | Not OK | OK | Isolated | network unreachable<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">----------------------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">Based on the state recognition table, nova can discern the exact host state and assign the reasons.<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">Thoughts?<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span style="font-size:11.0pt;color:black;mso-fareast-language:ZH-CN">Jenny<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>