<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">The thing laggy is, currently resource tracker will update the usage information whenever resource changes, not only in periodic tasks. If you
really want to get the current result with periodic update, you have to do some in-memory management and you even need sync between different scheduler controller, as stated in </span><span lang="EN-US"><a href="http://lists.openstack.org/pipermail/openstack-dev/2013-June/010490.html">http://lists.openstack.org/pipermail/openstack-dev/2013-June/010490.html</a>
. <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">-jyh</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Boris Pavlovic [mailto:boris@pavlovic.me]
<br>
<b>Sent:</b> Monday, July 22, 2013 5:17 AM<br>
<b>To:</b> OpenStack Development Mailing List<br>
<b>Subject:</b> Re: [openstack-dev] A simple way to improve nova scheduler<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">Joe, <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">>> </span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif"">Speaking of Chris Beherns "</span><span lang="EN-US" style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">Relying on
anything but the DB for current memory free, etc, is just too laggy… so we need to stick with it, IMO."
</span><span lang="EN-US"><a href="http://lists.openstack.org/pipermail/openstack-dev/2013-June/010485.html" target="_blank"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">http://lists.openstack.org/pipermail/openstack-dev/2013-June/010485.html</span></a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">It doesn't scale, use tons of resources, works slow and is hard to extend.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Also the mechanism of getting free and used memory is done by virt layer. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">And only thing that could be laggy is rpc (but it is used also by compute node update) <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif"">>> * How do you bring a new scheduler up in an existing deployment and make it get the full state of the system?</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif"">You should wait for a one periodic task time. And you will get full information about all compute nodes. </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif"">>> * Broadcasting RPC updates from compute nodes to the scheduler means every scheduler has to process the same RPC message. And if a deployment hits the point
where the number of compute updates is consuming 99 percent of the scheduler's time just adding another scheduler won't fix anything as it will get bombarded too.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">If we are speaking about numbers. You are able to see our doc, where they are counted. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">If we have 10k nodes it will make only 150rpc calls/sec (which means nothing for cpu). By the way we way we will remove 150 calls/s from conductor. One more thing currently in 10nodes deployment I think we will spend
almost all time fro waiting DB (compute_nodes_get_all()). And also when we are calling this method in this moment we should process all data for 60 sec. (So in this case in numbers we are doing on scheduler side 60*request_pro_sec of our approach. Which means
if we get more then 1 request pro sec we will do more CPU load.)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif"">>> Also OpenStack is already deeply invested in using the central DB model for the state of the 'world' and while I am not against changing that, I think we should
evaluate that switch in a larger context.<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Step by step. As first step we could just remove compute_node_get_all method. Which will make our openstack much scalable and fast. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">By the way see one more time answers on your comments in doc. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Best regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Boris Pavlovic<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Mirantis Inc. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Sat, Jul 20, 2013 at 3:14 AM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>> wrote:<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Fri, Jul 19, 2013 at 3:13 PM, Sandy Walsh <<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a>> wrote:<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
<br>
On 07/19/2013 05:36 PM, Boris Pavlovic wrote:<br>
> Sandy,<br>
><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">> I don't think that we have such problems here.<br>
> Because scheduler doesn't pool compute_nodes.<br>
> The situation is another compute_nodes notify scheduler about their<br>
> state. (instead of updating their state in DB)<br>
><br>
> So for example if scheduler send request to compute_node, compute_node<br>
> is able to run rpc call to schedulers immediately (not after 60sec).<br>
><br>
> So there is almost no races.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">There are races that occur between the eventlet request threads. This is<br>
why the scheduler has been switched to single threaded and we can only<br>
run one scheduler.<br>
<br>
This problem may have been eliminated with the work that Chris Behrens<br>
and Brian Elliott were doing, but I'm not sure.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Speaking of Chris Beherns "</span><span lang="EN-US" style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">Relying on anything but the DB for current memory free, etc, is just too laggy… so we need to stick
with it, IMO." </span><span lang="EN-US"><a href="http://lists.openstack.org/pipermail/openstack-dev/2013-June/010485.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2013-June/010485.html</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Although there is some elegance to the proposal here I have some concerns.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">If just using RPC broadcasts from compute to schedulers to keep track of things, we get two issues: <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">* How do you bring a new scheduler up in an existing deployment and make it get the full state of the system?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">* Broadcasting RPC updates from compute nodes to the scheduler means every scheduler has to process the same RPC message. And if a deployment hits the point where the number of compute updates is consuming 99 percent
of the scheduler's time just adding another scheduler won't fix anything as it will get bombarded too.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Also OpenStack is already deeply invested in using the central DB model for the state of the 'world' and while I am not against changing that, I think we should evaluate that switch in a larger context.<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US"><br>
But certainly, the old approach of having the compute node broadcast<br>
status every N seconds is not suitable and was eliminated a long time ago.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
><br>
><br>
> Best regards,<br>
> Boris Pavlovic<br>
><br>
> Mirantis Inc.<br>
><br>
><br>
><br>
> On Sat, Jul 20, 2013 at 12:23 AM, Sandy Walsh <<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">> <mailto:<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a>>> wrote:<br>
><br>
><br>
><br>
> On 07/19/2013 05:01 PM, Boris Pavlovic wrote:<br>
> > Sandy,<br>
> ><br>
> > Hm I don't know that algorithm. But our approach doesn't have<br>
> > exponential exchange.<br>
> > I don't think that in 10k nodes cloud we will have a problems with 150<br>
> > RPC call/sec. Even in 100k we will have only 1.5k RPC call/sec.<br>
> > More then (compute nodes update their state in DB through conductor<br>
> > which produce the same count of RPC calls).<br>
> ><br>
> > So I don't see any explosion here.<br>
><br>
> Sorry, I was commenting on Soren's suggestion from way back (essentially<br>
> listening on a separate exchange for each unique flavor ... so no<br>
> scheduler was needed at all). It was a great idea, but fell apart rather<br>
> quickly.<br>
><br>
> The existing approach the scheduler takes is expensive (asking the db<br>
> for state of all hosts) and polling the compute nodes might be do-able,<br>
> but you're still going to have latency problems waiting for the<br>
> responses (the states are invalid nearly immediately, especially if a<br>
> fill-first scheduling algorithm is used). We ran into this problem<br>
> before in an earlier scheduler implementation. The round-tripping kills.<br>
><br>
> We have a lot of really great information on Host state in the form of<br>
> notifications right now. I think having a service (or notification<br>
> driver) listening for these and keeping an the HostState incrementally<br>
> updated (and reported back to all of the schedulers via the fanout<br>
> queue) would be a better approach.<br>
><br>
> -S<br>
><br>
><br>
> ><br>
> > Best regards,<br>
> > Boris Pavlovic<br>
> ><br>
> > Mirantis Inc.<br>
> ><br>
> ><br>
> > On Fri, Jul 19, 2013 at 11:47 PM, Sandy Walsh<br>
> <<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a> <mailto:<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a>><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">> > <mailto:<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">> <mailto:<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a>>>> wrote:<br>
> ><br>
> ><br>
> ><br>
> > On 07/19/2013 04:25 PM, Brian Schott wrote:<br>
> > > I think Soren suggested this way back in Cactus to use MQ<br>
> for compute<br>
> > > node state rather than database and it was a good idea then.<br>
> ><br>
> > The problem with that approach was the number of queues went<br>
> exponential<br>
> > as soon as you went beyond simple flavors. Add Capabilities or<br>
> other<br>
> > criteria and you get an explosion of exchanges to listen to.<br>
> ><br>
> ><br>
> ><br>
> > > On Jul 19, 2013, at 10:52 AM, Boris Pavlovic<br>
> <<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a> <mailto:<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a>><br>
> > <mailto:<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a> <mailto:<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a>>><br>
> > > <mailto:<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a> <mailto:<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a>><br>
> <mailto:<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a> <mailto:<a href="mailto:boris@pavlovic.me" target="_blank">boris@pavlovic.me</a>>>>> wrote:<br>
> > ><br>
> > >> Hi all,<br>
> > >><br>
> > >><br>
> > >> In Mirantis Alexey Ovtchinnikov and me are working on nova<br>
> scheduler<br>
> > >> improvements.<br>
> > >><br>
> > >> As far as we can see the problem, now scheduler has two<br>
> major issues:<br>
> > >><br>
> > >> 1) Scalability. Factors that contribute to bad scalability<br>
> are these:<br>
> > >> *) Each compute node every periodic task interval (60 sec<br>
> by default)<br>
> > >> updates resources state in DB.<br>
> > >> *) On every boot request scheduler has to fetch information<br>
> about all<br>
> > >> compute nodes from DB.<br>
> > >><br>
> > >> 2) Flexibility. Flexibility perishes due to problems with:<br>
> > >> *) Addiing new complex resources (such as big lists of complex<br>
> > objects<br>
> > >> e.g. required by PCI Passthrough<br>
> > >><br>
> ><br>
> <a href="https://review.openstack.org/#/c/34644/5/nova/db/sqlalchemy/models.py" target="_blank">
https://review.openstack.org/#/c/34644/5/nova/db/sqlalchemy/models.py</a>)<br>
> > >> *) Using different sources of data in Scheduler for example<br>
> from<br>
> > >> cinder or ceilometer.<br>
> > >> (as required by Volume Affinity Filter<br>
> > >> <a href="https://review.openstack.org/#/c/29343/" target="_blank">
https://review.openstack.org/#/c/29343/</a>)<br>
> > >><br>
> > >><br>
> > >> We found a simple way to mitigate this issues by avoiding<br>
> of DB usage<br>
> > >> for host state storage.<br>
> > >><br>
> > >> A more detailed discussion of the problem state and one of<br>
> a possible<br>
> > >> solution can be found here:<br>
> > >><br>
> > >><br>
> ><br>
> <a href="https://docs.google.com/document/d/1_DRv7it_mwalEZzLy5WO92TJcummpmWL4NWsWf0UWiQ/edit" target="_blank">
https://docs.google.com/document/d/1_DRv7it_mwalEZzLy5WO92TJcummpmWL4NWsWf0UWiQ/edit#</a><br>
> > >><br>
> > >><br>
> > >> Best regards,<br>
> > >> Boris Pavlovic<br>
> > >><br>
> > >> Mirantis Inc.<br>
> > >><br>
> > >> _______________________________________________<br>
> > >> OpenStack-dev mailing list<br>
> > >> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">
OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>><br>
> > <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>>><br>
> > >> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>><br>
> > <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>>>><br>
> > >><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > ><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > OpenStack-dev mailing list<br>
> > > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">
OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>><br>
> > <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>>><br>
> > ><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > ><br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>><br>
> > <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>>><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>