<div dir="ltr"><div><font face="georgia, serif">Hi Robin,</font></div><div><font face="georgia, serif"><br></font></div><font face="georgia, serif">The idea sounds good to me too. I am working on refactoring <font color="#000000">ServiceGroup code</font>. Tooz has a nice compatibility matrix which can be found here [2] which you might find useful.</font><div><font face="georgia, serif"><br></font></div><div><font face="georgia, serif">-Vilobh</font></div><div><font face="georgia, serif"><br></font></div><div><font face="georgia, serif">[1] Servicegroup code refactoring <span style="color:rgb(31,73,125)">: </span><a href="https://review.openstack.org/#/c/172502/" target="_blank">https://review.openstack.org/#/c/172502/</a></font></div><div><font face="georgia, serif">[2] Tooz compatibility matrix : <a href="http://docs.openstack.org/developer/tooz/compatibility.html">http://docs.openstack.org/developer/tooz/compatibility.html</a></font></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 14, 2015 at 6:07 AM, Wangbibo <span dir="ltr"><<a href="mailto:wangbibo@huawei.com" target="_blank">wangbibo@huawei.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="ZH-CN" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Kevin and Joshua,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks for the review. Glad to see that oslo puts distributed coordination into its scope now. Per out of date info [1] (oslo doesn’t do it, while each project
should do it separately ), specific backend (zk/memcached) manipulating is included in spec[2], as nova ServiceGroup did. Now we have tooz, then that part should be moved out of AgentGroup and let tooz take it over. Neutron AgentGroup spec needs an update,
as what nova ServiceGroup refactor is doing. [3]<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Per spec[3], tooz doesn’t intend to eliminate or replace ServiceGroup completely. They are integrated and co-work to provide nova ServiceGroup functionalities.
That may answer the question from Kevin and Kyle, about relationship between AgentGroup and tooz. Let’s jump into [3][4]:<u></u><u></u></span></p>
<p style="margin-left:18pt">
<u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1)<span style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Service Group still exists;<u></u><u></u></span></p>
<p style="margin-left:18pt">
<u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2)<span style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Add Tooz driver for ServiceGroup, to take over zk/redis/… backend;<u></u><u></u></span></p>
<p style="margin-left:18pt">
<u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>3)<span style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Db-based ServiceGroup driver is retained. Db driver was introduced for backward compatibility (with db-based liveness monitoring which existed for a
long time before ServiceGroup was added). Since this driver uses tables and a data model that is intrinsically tied to the internal of nova, tooz cannot take it over.
<u></u><u></u></span></p>
<p style="margin-left:18pt">
<u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>4)<span style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Zk/memcached ServiceGroup drivers are temporarily retained, but will be deprecated in future;
<u></u><u></u></span></p>
<p style="margin-left:18pt">
<u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>5)<span style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Eventually, there would be two ServiceGroup drivers: db driver & tooz driver;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Actually, things are the same for neutron, except that we don’t need to consider zk/memcached driver deprecation. I would like to refine current spec and propose
a ”Agent Group and using tooz” spec, following the outlines above. What do you think, Kevin and Joshua? Thanks.
</span><span lang="EN-US" style="font-family:Wingdings;color:rgb(31,73,125)">J</span><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Best,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Robin<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">[1]
<a href="https://wiki.openstack.org/wiki/NovaZooKeeperHeartbeat" target="_blank">https://wiki.openstack.org/wiki/NovaZooKeeperHeartbeat</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">[2]
</span><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a href="https://review.openstack.org/#/c/168921/" target="_blank">https://review.openstack.org/#/c/168921/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">[3]
<a href="https://review.openstack.org/#/c/138607/11/specs/liberty/approved/service-group-using-tooz.rst" target="_blank">
https://review.openstack.org/#/c/138607/11/specs/liberty/approved/service-group-using-tooz.rst</a></span><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">[4] ServiceGroup refactor code:
<a href="https://review.openstack.org/#/c/172502/" target="_blank">https://review.openstack.org/#/c/172502/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10pt">发件人<span lang="EN-US">:</span></span></b><span lang="EN-US" style="font-size:10pt"> Wangbibo [mailto:<a href="mailto:wangbibo@huawei.com" target="_blank">wangbibo@huawei.com</a>]
<br>
</span><b><span style="font-size:10pt">发送时间<span lang="EN-US">:</span></span></b><span lang="EN-US" style="font-size:10pt"> 2015</span><span style="font-size:10pt">年<span lang="EN-US">4</span>月<span lang="EN-US">13</span>日<span lang="EN-US"> 16:52<br>
</span><span class=""><b>收件人<span lang="EN-US">:</span></b><span lang="EN-US"> OpenStack Development Mailing List (not for usage questions)<br>
</span></span><b>主题<span lang="EN-US">:</span></b><span lang="EN-US"> [openstack-dev] </span>
答复<span lang="EN-US">: [neutron] Neutron scaling datapoints?<u></u><u></u></span></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Kevin,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Totally agree with you that heartbeat from each agent is something that we cannot eliminate currently. Agent status depends on it, and further scheduler and HA
depends on agent status.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">I proposed a Liberty spec for introducing open framework/pluggable agent status drivers.[1][2] It allows us to use some other 3<sup>rd</sup> party backend to monitor
agent status, such as zookeeper, memcached. Meanwhile, it guarantees backward compatibility so that users could still use db-based status monitoring mechanism as their default choice.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Base on that, we may do further optimization on issues Attila and you mentioned. Thanks.
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">[1] BP -
<a href="https://blueprints.launchpad.net/neutron/+spec/agent-group-and-status-drivers" target="_blank">
https://blueprints.launchpad.net/neutron/+spec/agent-group-and-status-drivers</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">[2] Liberty Spec proposed -
<a href="https://review.openstack.org/#/c/168921/" target="_blank">https://review.openstack.org/#/c/168921/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Best,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Calibri,sans-serif;color:rgb(31,73,125)">Robin<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10pt">发件人<span lang="EN-US">:</span></span></b><span lang="EN-US" style="font-size:10pt"> Kevin Benton [<a href="mailto:blak111@gmail.com" target="_blank">mailto:blak111@gmail.com</a>]
<br>
</span><b><span style="font-size:10pt">发送时间<span lang="EN-US">:</span></span></b><span lang="EN-US" style="font-size:10pt"> 2015</span><span style="font-size:10pt">年<span lang="EN-US">4</span>月<span lang="EN-US">11</span>日<span lang="EN-US"> 12:35<br>
</span><b>收件人<span lang="EN-US">:</span></b><span lang="EN-US"> OpenStack Development Mailing List (not for usage questions)<br>
</span><b>主题<span lang="EN-US">:</span></b><span lang="EN-US"> Re: [openstack-dev] [neutron] Neutron scaling datapoints?<u></u><u></u></span></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">Which periodic updates did you have in mind to eliminate? One of the few remaining ones I can think of is sync_routers but it would be great if you can enumerate the ones you observed because eliminating overhead in agents
is something I've been working on as well.<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">One of the most common is the heartbeat from each agent. However, I don't think we can't eliminate them because they are used to determine if the agents are still alive for scheduling purposes. Did you have something
else in mind to determine if an agent is alive?<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas <<a href="mailto:afazekas@redhat.com" target="_blank">afazekas@redhat.com</a>> wrote:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I'm 99.9% sure, for scaling above 100k managed node,<br>
we do not really need to split the openstack to multiple smaller openstack,<br>
or use significant number of extra controller machine.<br>
<br>
The problem is openstack using the right tools SQL/AMQP/(zk),<br>
but in a wrong way.<br>
<br>
For example.:<br>
Periodic updates can be avoided almost in all cases<br>
<br>
The new data can be pushed to the agent just when it needed.<br>
The agent can know when the AMQP connection become unreliable (queue or connection loose),<br>
and needs to do full sync.<br>
<a href="https://bugs.launchpad.net/neutron/+bug/1438159" target="_blank">https://bugs.launchpad.net/neutron/+bug/1438159</a><br>
<br>
Also the agents when gets some notification, they start asking for details via the<br>
AMQP -> SQL. Why they do not know it already or get it with the notification ?<br>
<br>
<br>
<span>----- Original Message -----</span><br>
<span>> From: "Neil Jerram" <<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>></span><u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">> To: "OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
> Sent: Thursday, April 9, 2015 5:01:45 PM<br>
> Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?<br>
><br>
> Hi Joe,<br>
><br>
> Many thanks for your reply!<br>
><br>
> On 09/04/15 03:34, joehuang wrote:<br>
> > Hi, Neil,<br>
> ><br>
> > From theoretic, Neutron is like a "broadcast" domain, for example,<br>
> > enforcement of DVR and security group has to touch each regarding host<br>
> > where there is VM of this project resides. Even using SDN controller, the<br>
> > "touch" to regarding host is inevitable. If there are plenty of physical<br>
> > hosts, for example, 10k, inside one Neutron, it's very hard to overcome<br>
> > the "broadcast storm" issue under concurrent operation, that's the<br>
> > bottleneck for scalability of Neutron.<br>
><br>
> I think I understand that in general terms - but can you be more<br>
> specific about the broadcast storm? Is there one particular message<br>
> exchange that involves broadcasting? Is it only from the server to<br>
> agents, or are there 'broadcasts' in other directions as well?<br>
><br>
> (I presume you are talking about control plane messages here, i.e.<br>
> between Neutron components. Is that right? Obviously there can also be<br>
> broadcast storm problems in the data plane - but I don't think that's<br>
> what you are talking about here.)<br>
><br>
> > We need layered architecture in Neutron to solve the "broadcast domain"<br>
> > bottleneck of scalability. The test report from OpenStack cascading shows<br>
> > that through layered architecture "Neutron cascading", Neutron can<br>
> > supports up to million level ports and 100k level physical hosts. You can<br>
> > find the report here:<br>
> > <a href="http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers" target="_blank">
http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers</a><br>
><br>
> Many thanks, I will take a look at this.<br>
><br>
> > "Neutron cascading" also brings extra benefit: One cascading Neutron can<br>
> > have many cascaded Neutrons, and different cascaded Neutron can leverage<br>
> > different SDN controller, maybe one is ODL, the other one is OpenContrail.<br>
> ><br>
> > ----------------Cascading Neutron-------------------<br>
> > / \<br>
> > --cascaded Neutron-- --cascaded Neutron-----<br>
> > | |<br>
> > ---------ODL------ ----OpenContrail--------<br>
> ><br>
> ><br>
> > And furthermore, if using Neutron cascading in multiple data centers, the<br>
> > DCI controller (Data center inter-connection controller) can also be used<br>
> > under cascading Neutron, to provide NaaS ( network as a service ) across<br>
> > data centers.<br>
> ><br>
> > ---------------------------Cascading Neutron--------------------------<br>
> > / | \<br>
> > --cascaded Neutron-- -DCI controller- --cascaded Neutron-----<br>
> > | | |<br>
> > ---------ODL------ | ----OpenContrail--------<br>
> > |<br>
> > --(Data center 1)-- --(DCI networking)-- --(Data center 2)--<br>
> ><br>
> > Is it possible for us to discuss this in OpenStack Vancouver summit?<br>
><br>
> Most certainly, yes. I will be there from mid Monday afternoon through<br>
> to end Friday. But it will be my first summit, so I have no idea yet as<br>
> to how I might run into you - please can you suggest!<br>
><br>
> > Best Regards<br>
> > Chaoyi Huang ( Joe Huang )<br>
><br>
> Regards,<br>
> Neil<br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><br>
<br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">-- <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Kevin Benton<u></u><u></u></span></p>
</div>
</div>
</div>
</div></div></div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>