<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Yes I am good with writing spec for this in masakari-spec.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Do you use gerrit for this git ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Do you have a template for your specs ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Greg.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">Sam P <sam47priya@gmail.com><br>
<b>Reply-To: </b>"openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org><br>
<b>Date: </b>Thursday, May 18, 2017 at 1:51 PM<br>
<b>To: </b>"openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org><br>
<b>Subject: </b>Re: [openstack-dev] [vitrage] [nova] [HA] [masakari] VM Heartbeat / Healthcheck Monitoring<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi Greg,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thank you Adam for followup.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This is new feature for masakari-monitors and think Masakari can<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">accommodate this feature in masakari-monitors.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">From the implementation prospective, it is not that hard to do.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">However, as you can see in our Boston presentation, Masakari will<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">replace its monitoring parts ( which is masakari-monitors) with,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">nova-host-alerter, **-process-alerter, and **-instance-alerter. (**<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">part is not defined yet..:p)...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Therefore, I would like to save this specifications, and make sure we<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">will not miss anything in the transformation..<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Does is make sense to write simple spec for this in masakari-spec [1]?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">So we can discuss about the requirements how to implement it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://github.com/openstack/masakari-specs">https://github.com/openstack/masakari-specs</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--- Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Sampath<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On Thu, May 18, 2017 at 2:29 AM, Adam Spiers <<a href="mailto:aspiers@suse.com">aspiers@suse.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-right:0cm" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal">I don't see any reason why masakari couldn't handle that, but you'd<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">have to ask Sampath and the masakari team whether they would consider<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">that in scope for their roadmap.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Waines, Greg <<a href="mailto:Greg.Waines@windriver.com">Greg.Waines@windriver.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-right:0cm" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sure. I can propose a new user story.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And then are you thinking of including this user story in the scope of<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">what masakari would be looking at ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Greg.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From: Adam Spiers <<a href="mailto:aspiers@suse.com">aspiers@suse.com</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Reply-To: "<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Date: Wednesday, May 17, 2017 at 10:08 AM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">To: "<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Subject: Re: [openstack-dev] [vitrage] [nova] [HA] VM Heartbeat /<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Healthcheck Monitoring<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks for the clarification Greg. This sounds like it has the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">potential to be a very useful capability. May I suggest that you<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">propose a new user story for it, along similar lines to this existing<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">one?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/ha_vm.html">http://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/ha_vm.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Waines, Greg <<a href="mailto:Greg.Waines@windriver.com">Greg.Waines@windriver.com</a><<a href="mailto:Greg.Waines@windriver.com%3e">mailto:Greg.Waines@windriver.com></a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">wrote:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Yes that’s correct.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">VM Heartbeating / Health-check Monitoring would introduce intrusive /<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">white-box type monitoring of VMs / Instances.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I realize this is somewhat in the gray-zone of what a cloud should be<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">monitoring or not,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">but I believe it provides an alternative for Applications deployed in VMs<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">that do not have an external monitoring/management entity like a VNF Manager<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">in the MANO architecture.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">And even for VMs with VNF Managers, it provides a highly reliable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">alternate monitoring path that does not rely on Tenant Networking.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You’re correct, that VM HB/HC Monitoring would leverage<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://wiki.libvirt.org/page/Qemu_guest_agent">https://wiki.libvirt.org/page/Qemu_guest_agent</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">that would require the agent to be installed in the images for talking<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">back to the compute host.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">( there are other examples of similar approaches in openstack ... the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">murano-agent for installation, the swift-agent for object store management )<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Although here, in the case of VM HB/HC Monitoring, via the QEMU Guest<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Agent, the messaging path is internal thru a QEMU virtual serial device.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">i.e. a very simple interface with very few dependencies ... it’s up and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">available very early in VM lifecycle and virtually always up.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Wrt failure modes / use-cases<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">· a VM’s response to a Heartbeat Challenge Request can be as<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">simple as just ACK-ing,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">this alone allows for detection of:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o a failed or hung QEMU/KVM instance, or<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o a failed or hung VM’s OS, or<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o a failure of the VM’s OS to schedule the QEMU Guest Agent daemon, or<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o a failure of the VM to route basic IO via linux sockets.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">· I have had feedback that this is similar to the virtual hardware<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">watchdog of QEMU/KVM (<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://libvirt.org/formatdomain.html#elementsWatchdog">https://libvirt.org/formatdomain.html#elementsWatchdog</a> )<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">· However, the VM Heartbeat / Health-check Monitoring<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o provides a higher-level (i.e. application-level) heartbeating<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">§ i.e. if the Heartbeat requests are being answered by the Application<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">running within the VM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o provides more than just heartbeating, as the Application can use it to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">trigger a variety of audits,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o provides a mechanism for the Application within the VM to report a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Health Status / Info back to the Host / Cloud,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o provides notification of the Heartbeat / Health-check status to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">higher-level cloud entities thru Vitrage<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">§ e.g. VM-Heartbeat-Monitor - to - Vitrage - (EventAlarm) - Aodh - ...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- VNF-Manager<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">- (StateChange) - Nova - ... - VNF Manager<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Greg.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From: Adam Spiers <<a href="mailto:aspiers@suse.com">aspiers@suse.com</a><<a href="mailto:aspiers@suse.com%3e">mailto:aspiers@suse.com></a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Reply-To:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">"<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><<a href="mailto:openstack-dev@lists.openstack.org%3e">mailto:openstack-dev@lists.openstack.org></a>"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><<a href="mailto:openstack-dev@lists.openstack.org%3e">mailto:openstack-dev@lists.openstack.org></a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Date: Tuesday, May 16, 2017 at 7:29 PM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">To:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">"<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><<a href="mailto:openstack-dev@lists.openstack.org%3e">mailto:openstack-dev@lists.openstack.org></a>"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><<a href="mailto:openstack-dev@lists.openstack.org%3e">mailto:openstack-dev@lists.openstack.org></a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Subject: Re: [openstack-dev] [vitrage] [nova] [HA] VM Heartbeat /<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Healthcheck Monitoring<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Waines, Greg<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><<a href="mailto:Greg.Waines@windriver.com">Greg.Waines@windriver.com</a><<a href="mailto:Greg.Waines@windriver.com%3e%3cmailto:Greg.Waines@windriver.com%3e%3cmailto:Greg.Waines@windriver.com%3e%3e">mailto:Greg.Waines@windriver.com><mailto:Greg.Waines@windriver.com><mailto:Greg.Waines@windriver.com%3e></a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">wrote:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">thanks for the pointers Sam.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I took a quick look.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I agree that the VM Heartbeat / Health-check looks like a good fit into<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Masakari.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Currently your instance monitoring looks like it is strictly black-box<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">type monitoring thru libvirt events.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Is that correct ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">i.e. you do not do any intrusive type monitoring of the instance thru the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">QUEMU Guest Agent facility<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> correct ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That is correct:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/instancemonitor/instance.py">https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/instancemonitor/instance.py</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think this is what VM Heartbeat / Health-check would add to Masaraki.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Let me know if you agree.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">OK, so you are looking for something slightly different I guess, based<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">on this QEMU guest agent?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <a href="https://wiki.libvirt.org/page/Qemu_guest_agent">https://wiki.libvirt.org/page/Qemu_guest_agent</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That would require the agent to be installed in the images, which is<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">extra work but I imagine quite easily justifiable in some scenarios.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">What failure modes do you have in mind for covering with this<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">approach - things like the guest kernel freezing, for instance?<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">__________________________________________________________________________<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">OpenStack Development Mailing List (not for usage questions)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">__________________________________________________________________________<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">OpenStack Development Mailing List (not for usage questions)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>