<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Jay,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think we should agree on a general direction before asking for a spec. It is bad to have contributors spend time working on something that might not be accepted.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hongbin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Jay Lau [mailto:jay.lau.513@gmail.com]
<br>
<b>Sent:</b> December-20-15 6:17 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [openstack][magnum] Quota for Magnum Resources<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Thanks Adrian and Tim, I saw that @Vilobh already uploaded a patch
<a href="https://review.openstack.org/#/c/259201/">https://review.openstack.org/#/c/259201/</a> here, perhaps we can first have a spec and discuss there. ;-)
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Mon, Dec 21, 2015 at 2:44 AM, Tim Bell <<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Given the lower level quotas in Heat, Neutron, Nova etc., the error feedback is very
important. A Magnum “cannot create” message requires a lot of debugging whereas a “Floating IP quota exceeded” gives a clear root cause.</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Whether we quota Magnum resources or not, some error scenarios and appropriate testing+documentation
would be a great help for operators.</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Tim</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span lang="EN-GB"><o:p></o:p></span></p>
<div style="border:none;border-left:solid windowtext 1.5pt;padding:0cm 0cm 0cm 4.0pt;border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color blue">
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:-moz-use-text-color -moz-use-text-color">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">
Adrian Otto [mailto:<a href="mailto:adrian.otto@rackspace.com" target="_blank">adrian.otto@rackspace.com</a>]
<br>
<b>Sent:</b> 20 December 2015 18:50<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>></span><span lang="EN-GB"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-GB"><br>
<b>Subject:</b> Re: [openstack-dev] [openstack][magnum] Quota for Magnum Resources<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">This sounds like a source-of-truth concern. From my perspective the solution is not to create redundant quotas. Simply quota the Magnum resources. Lower level
limits *could* be queried by magnum prior to acting to CRUD the lower level resources. In the case we could check the maximum allowed number of (or access rate of) whatever lower level resource before requesting it, and raising an understandable error. I see
that as an enhancement rather than a must-have. In all honesty that feature is probably more complicated than it's worth in terms of value.<br>
<br>
-- <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">Adrian<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB"><br>
On Dec 20, 2015, at 6:36 AM, Jay Lau <<a href="mailto:jay.lau.513@gmail.com" target="_blank">jay.lau.513@gmail.com</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">I also have the same concern with Lee, as Magnum depend on HEAT and HEAT need call nova, cinder, neutron to create the Bay resources. But both Nova and Cinder
has its own quota policy, if we define quota again in Magnum, then how to handle the conflict? Another point is that limiting the Bay by quota seems a bit coarse-grainded as different bays may have different configuration and resource request. Comments? Thanks.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">On Thu, Dec 17, 2015 at 4:10 AM, Lee Calcote <<a href="mailto:leecalcote@gmail.com" target="_blank">leecalcote@gmail.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color rgb(204,204,204)">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">Food for thought - there is a cost to FIPs (in the case of public IP addresses), security groups (to a lesser extent, but in terms of the computation of many
hundreds of them), etc. Administrators may wish to enforce quotas on a variety of resources that are direct costs or indirect costs (e.g. # of bays, where a bay consists of a number of multi-VM / multi-host pods and services, which consume CPU, mem, etc.).
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">If Magnum quotas are brought forward, they should govern (enforce quota) on Magnum-specific constructs only, correct? Resources created by Magnum COEs should
be governed by existing quota policies governing said resources (e.g. Nova and vCPUs).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="color:#888888"> </span><span lang="EN-GB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="color:#888888">Lee</span><span lang="EN-GB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"> <o:p></o:p></span></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">On Dec 16, 2015, at 1:56 PM, Tim Bell <<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>> wrote:<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt;text-align:start;word-spacing:0px">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">-----Original Message-----<br>
From: Clint Byrum [<a href="mailto:clint@fewbar.com" target="_blank">mailto:clint@fewbar.com</a>]<br>
Sent: 15 December 2015 22:40<br>
To: openstack-dev <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
Subject: Re: [openstack-dev] [openstack][magnum] Quota for Magnum<br>
Resources<br>
<br>
Hi! Can I offer a counter point?<br>
<br>
Quotas are for _real_ resources.</span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif""><br>
The CERN container specialist agrees with you ... it would be good to<br>
reflect on the needs given that ironic, neutron and nova are policing the<br>
resource usage. Quotas in the past have been used for things like key pairs<br>
which are not really real.<br>
<br>
</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">Memory, CPU, disk, bandwidth. These are all _closely_ tied to things that</span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">cost</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">real money and cannot be conjured from thin air. As such, the user being<br>
able to allocate 1 billion or 2 containers is not limited by Magnum, but</span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">by real</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">things that they must pay for. If they have enough Nova quota to allocate</span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">1</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">billion tiny pods, why would Magnum stop them? Who actually benefits from<br>
that limitation?<br>
<br>
So I suggest that you not add any detailed, complicated quota system to<br>
Magnum. If there are real limitations to the implementation that Magnum<br>
has chosen, such as we had in Heat (the entire stack must fit in memory),<br>
then make that the limit. Otherwise, let their vcpu, disk, bandwidth, and<br>
memory quotas be the limit, and enjoy the profit margins that having an<br>
unbound force multiplier like Magnum in your cloud gives you and your<br>
users!<br>
<br>
Excerpts from Vilobh Meshram's message of 2015-12-14 16:58:54 -0800:</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">Hi All,<br>
<br>
Currently, it is possible to create unlimited number of resource like<br>
bay/pod/service/. In Magnum, there should be a limitation for user or<br>
project to create Magnum resource, and the limitation should be<br>
configurable[1].<br>
<br>
I proposed following design :-<br>
<br>
1. Introduce new table magnum.quotas<br>
+------------+--------------+------+-----+---------+----------------+<br>
<br>
| Field | Type | Null | Key | Default | Extra |<br>
<br>
+------------+--------------+------+-----+---------+----------------+<br>
<br>
| id | int(11) | NO | PRI | NULL | auto_increment |<br>
<br>
| created_at | datetime | YES | | NULL | |<br>
<br>
| updated_at | datetime | YES | | NULL | |<br>
<br>
| deleted_at | datetime | YES | | NULL | |<br>
<br>
| project_id | varchar(255) | YES | MUL | NULL | |<br>
<br>
| resource | varchar(255) | NO | | NULL | |<br>
<br>
| hard_limit | int(11) | YES | | NULL | |<br>
<br>
| deleted | int(11) | YES | | NULL | |<br>
<br>
+------------+--------------+------+-----+---------+----------------+<br>
<br>
resource can be Bay, Pod, Containers, etc.<br>
<br>
<br>
2. API controller for quota will be created to make sure basic CLI<br>
commands work.<br>
<br>
quota-show, quota-delete, quota-create, quota-update<br>
<br>
3. When the admin specifies a quota of X number of resources to be<br>
created the code should abide by that. For example if hard limit for Bay</span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">is 5</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">(i.e.</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">a project can have maximum 5 Bay's) if a user in a project tries to<br>
exceed that hardlimit it won't be allowed. Similarly goes for other</span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">resources.</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif""><br>
4. Please note the quota validation only works for resources created<br>
via Magnum. Could not think of a way that Magnum to know if a COE<br>
specific utilities created a resource in background. One way could be<br>
to see the difference between whats stored in magnum.quotas and the<br>
information of the actual resources created for a particular bay in</span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">k8s/COE.</span><span lang="EN-GB"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif""><br>
5. Introduce a config variable to set quotas values.<br>
<br>
If everyone agrees will start the changes by introducing quota<br>
restrictions on Bay creation.<br>
<br>
Thoughts ??<br>
<br>
<br>
-Vilobh<br>
<br>
[1] <a href="https://blueprints.launchpad.net/magnum/+spec/resource-quota" target="_blank">
https://blueprints.launchpad.net/magnum/+spec/resource-quota</a></span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif""><br>
________________________________________________________________<br>
__________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: OpenStack-dev-<br>
<a href="mailto:request@lists.openstack.org" target="_blank">request@lists.openstack.org</a>?subject:unsubscribe<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><span lang="EN-GB"><o:p></o:p></span></p>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: </span><span lang="EN-GB"><a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">OpenStack-dev-request@lists.openstack.org</span></a></span><span lang="EN-GB" style="font-size:10.5pt;font-family:"Calibri","sans-serif"">?subject:unsubscribe<br>
</span><span lang="EN-GB"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif"">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"> <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="EN-GB">Thanks,<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">Jay Lau (Guangya Liu)<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Thanks,<o:p></o:p></p>
</div>
<p class="MsoNormal">Jay Lau (Guangya Liu)<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>