<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi, Kevin and Joshua,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">As my understanding, Tooz only addresses the issue of agent status management, but how to solve the concurrent dynamic load impact on large scale
( for example 100k managed nodes with the dynamic load like security goup rule update, routers_updated, etc )<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">And one more question is, if we have 100k managed nodes, how to do the partition? Or all nodes will be managed by one Tooz service, like Zookeeper?
Can Zookeeper manage 100k nodes status?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Best Regards<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Chaoyi Huang ( Joe Huang )<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Kevin Benton [mailto:blak111@gmail.com]
<br>
<b>Sent:</b> Monday, April 13, 2015 3:52 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [neutron] Neutron scaling datapoints?<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">></span><span lang="EN-US" style="font-size:7.5pt">Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its
own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being that its backend specific and tooz supports varying backends.</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:7.5pt">Very cool. Is the backend completely transparent so a deployer could choose a service they are comfortable maintaining, or will that change the properties WRT to resiliency of state on node restarts,
partitions, etc?</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">The Nova implementation of Tooz seemed pretty straight-forward, although it looked like it had pluggable drivers for service management already. Before I dig into it much further I'll file a spec on the Neutron side to
see if I can get some other cores onboard to do the review work if I push a change to tooz.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <<a href="mailto:harlowja@outlook.com" target="_blank">harlowja@outlook.com</a>> wrote:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Kevin Benton wrote:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">So IIUC tooz would be handling the liveness detection for the agents.<br>
That would be nice to get ride of that logic in Neutron and just<br>
register callbacks for rescheduling the dead.<br>
<br>
Where does it store that state, does it persist timestamps to the DB<br>
like Neutron does? If so, how would that scale better? If not, who does<br>
a given node ask to know if an agent is online or offline when making a<br>
scheduling decision?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being
that its backend specific and tooz supports varying backends.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><br>
However, before (what I assume is) the large code change to implement<br>
tooz, I would like to quantify that the heartbeats are actually a<br>
bottleneck. When I was doing some profiling of them on the master branch<br>
a few months ago, processing a heartbeat took an order of magnitude less<br>
time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A<br>
few query optimizations might buy us a lot more headroom before we have<br>
to fall back to large refactors.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
Sure, always good to avoid prematurely optimizing things...<br>
<br>
Although this is relevant for u I think anyway:<br>
<br>
<a href="https://review.openstack.org/#/c/138607/" target="_blank">https://review.openstack.org/#/c/138607/</a> (same thing/nearly same in nova)...<br>
<br>
<a href="https://review.openstack.org/#/c/172502/" target="_blank">https://review.openstack.org/#/c/172502/</a> (a WIP implementation of the latter).<br>
<br>
[1] <a href="https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes" target="_blank">
https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><br>
Kevin Benton wrote:<br>
<br>
<br>
One of the most common is the heartbeat from each agent. However, I<br>
don't think we can't eliminate them because they are used to determine<br>
if the agents are still alive for scheduling purposes. Did you have<br>
something else in mind to determine if an agent is alive?<br>
<br>
<br>
Put each agent in a tooz[1] group; have each agent periodically<br>
heartbeat[2], have whoever needs to schedule read the active members of<br>
that group (or use [3] to get notified via a callback), profit...<br>
<br>
Pick from your favorite (supporting) driver at:<br>
<br>
<a href="http://docs.openstack.org/__developer/tooz/compatibility.__html" target="_blank">http://docs.openstack.org/__developer/tooz/compatibility.__html</a><br>
<<a href="http://docs.openstack.org/developer/tooz/compatibility.html" target="_blank">http://docs.openstack.org/developer/tooz/compatibility.html</a>><br>
<br>
[1]<br>
<a href="http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping" target="_blank">http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping</a><br>
<<a href="http://docs.openstack.org/developer/tooz/compatibility.html#grouping" target="_blank">http://docs.openstack.org/developer/tooz/compatibility.html#grouping</a>><br>
[2]<br>
<a href="https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315" target="_blank">https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315</a><br>
<<a href="https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315" target="_blank">https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315</a>><br>
[3]<br>
<a href="http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes" target="_blank">http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes</a><br>
<<a href="http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes" target="_blank">http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes</a>><br>
<br>
<br>
______________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
OpenStack-dev-request@lists.__<a href="http://openstack.org?subject:__unsubscribe" target="_blank">openstack.org?subject:__unsubscribe</a><br>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<a href="http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev" target="_blank">http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Kevin Benton<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>