<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi Louie,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> Thank you for patch and Sorry for the delay response.</div><div class="gmail_default"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I prefer option 2. </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">From Masakari point of view, this is an instance event. Because, even if some thing</span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">wrong inside the VM, Masakari only can try to fix it by restart, rebuilt, migrate... etc the VM.</span></div><div class="gmail_default">Which are the same recovery work flow for instance failures. Therefore, I prefer option 2</div><div class="gmail_default">rather than option1.</div><div class="gmail_default"> Currently, we are discussing how to implement recovery method customization feature [0] in</div><div class="gmail_default">Masakari. With this feature, you may able to call external workflows for certain failure events.</div><div class="gmail_default">For this feature, different failure models required distinguishable events and option 3 will not</div><div class="gmail_default">be appropriate.</div><div class="gmail_default"><br></div><div class="gmail_default">[0] <a href="https://review.openstack.org/#/c/458023/">https://review.openstack.org/#/c/458023/</a></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_extra"><div class="gmail_extra"></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">> 1. define a new type of event for Intrusive Instance monitoring or</span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">> 2. add a new event within the INSTANCE_EVENTS as we may eventually integrate with instance monitoring or</span><br></div><div class="gmail_extra"><div class="gmail_extra"><div class="gmail_extra">>3.simply reuse the LIFECYCLE/STOPPED_FAILED event ( which is what we are implementing for now.)</div><div class="gmail_extra"><br></div></div><div><div class="gmail_signature"><div dir="ltr"><div>---<font color="#333333" face="Times New Roman"><span style="font-size:14px;line-height:21.6px"> R</span></font><span style="color:rgb(51,51,51);font-family:"Times New Roman";font-size:14px;line-height:21.6px">egards,</span></div><div>Sampath</div><div><br></div></div></div></div>
<br><div class="gmail_quote">On Fri, Feb 16, 2018 at 12:05 AM, Kwan, Louie <span dir="ltr"><<a href="mailto:Louie.Kwan@windriver.com" target="_blank">Louie.Kwan@windriver.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="white" lang="EN-US">
<div class="gmail-m_3142668853765571881WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">We submitted the first implementation patch for the following blueprint
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><a href="https://blueprints.launchpad.net/openstack/?searchtext=intrusive-instance-monitoring" target="_blank">https://blueprints.launchpad.<wbr>net/openstack/?searchtext=<wbr>intrusive-instance-monitoring</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">i.e. <a href="https://review.openstack.org/#/c/534958/" target="_blank">https://review.openstack.org/#<wbr>/c/534958/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">The second patch will be pushed within a week time or so.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">One item we would like to seek clarification among the community is about how we should integrate the notification within the masakari engine.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">One option is to reuse what has been defined at masakari/engine/instance_<wbr>events.py.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">e.g.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> def masakari_notifier(self, domain_uuid):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> if self.getJournalObject(domain_<wbr>uuid).getSentNotification():<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> LOG.debug('notifier.send_<wbr>notification Skipped:' + domain_uuid)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> else:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> hostname = socket.gethostname()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> noticeType = ec.EventConstants.TYPE_VM<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> current_time = timeutils.utcnow()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> event = {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'notification': {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'type': noticeType,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'hostname': hostname,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'generated_time': current_time,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'payload': {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'event': 'LIFECYCLE',<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'instance_uuid': domain_uuid,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'vir_domain_event': 'STOPPED_FAILED'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> LOG.debug(str(event))<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> self.notifier.send_<wbr>notification(CONF.callback.<wbr>retry_max,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> <wbr> CONF.callback.retry_interval,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> <wbr> event)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> self.getJournalObject(domain_<wbr>uuid).setSentNotification(<wbr>True)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"></span></p><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"></div>Should we <u></u>
<u></u><p></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="gmail-m_3142668853765571881MsoListParagraph"><u></u><span style="font-size:11pt;color:rgb(31,73,125)"><span>1.<span style="font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;color:rgb(31,73,125)">define a new type of event for Intrusive Instance monitoring or
<u></u><u></u></span></p>
<p class="gmail-m_3142668853765571881MsoListParagraph"><u></u><span style="font-size:11pt;color:rgb(31,73,125)"><span>2.<span style="font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;color:rgb(31,73,125)">add a new event within the INSTANCE_EVENTS as we may eventually integrate with instance monitoring or<u></u><u></u></span></p>
<p class="gmail-m_3142668853765571881MsoListParagraph"><u></u><span style="font-size:11pt;color:rgb(31,73,125)"><span>3.<span style="font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:"Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;color:rgb(31,73,125)">simply reuse the LIFECYCLE/STOPPED_FAILED event ( which is what we are implementing for now.)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">One of our reference test case is to detect application meltdown within VM which QEMU may not aware the failure. The recovery should pretty much be the same as LIFECYCLE/STOPPED_FAILED event.
What do you think?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">Thanks.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">Louie<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">Ntoe:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">Here is what we got from masakari/engine/instance_<wbr>events.py<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">These are the events which needs to be processed by masakari in case of<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">instance recovery failure.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">"""<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">INSTANCE_EVENTS = {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> # Add more events and vir_domain_events here.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'LIFECYCLE': ['STOPPED_FAILED'],<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)"> 'IO_ERROR': ['IO_ERROR_REPORT']<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31,73,125)">}<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
</div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div></div>