Open Stack

Hi Robin,

The idea sounds good to me too.  I am working on refactoring ServiceGroup
code. Tooz has a nice compatibility matrix which can be found here [2]
which you might find useful.

-Vilobh

[1] Servicegroup code refactoring : https://review.openstack.org/#/c/172502/
[2] Tooz compatibility matrix :
http://docs.openstack.org/developer/tooz/compatibility.html


On Tue, Apr 14, 2015 at 6:07 AM, Wangbibo <wangbibo at huawei.com> wrote:

>  Hi Kevin and Joshua,
>
>
>
> Thanks for the review.  Glad to see that oslo puts distributed
> coordination into its scope now.  Per out of date info [1] (oslo doesn’t do
> it, while each project should do it separately ),  specific backend
> (zk/memcached) manipulating is included in spec[2], as nova ServiceGroup
> did.  Now we have tooz, then that part should be moved out of AgentGroup
> and let tooz take it over. Neutron AgentGroup spec needs an update, as what
> nova ServiceGroup refactor is doing. [3]
>
>
>
> Per spec[3], tooz doesn’t intend to eliminate or replace ServiceGroup
> completely. They are integrated and co-work to provide nova ServiceGroup
> functionalities. That may answer the question from Kevin and Kyle, about
> relationship between AgentGroup and tooz. Let’s jump into [3][4]:
>
> 1)      Service Group still exists;
>
> 2)      Add Tooz driver for ServiceGroup, to take over zk/redis/… backend;
>
> 3)      Db-based ServiceGroup driver is retained.  Db driver was
> introduced for backward compatibility (with db-based liveness monitoring
> which existed for a long time before ServiceGroup was added). Since this
> driver uses tables and a data model that is intrinsically tied to the
> internal of nova, tooz cannot take it over.
>
> 4)      Zk/memcached ServiceGroup drivers are temporarily retained, but
> will be deprecated in future;
>
> 5)      Eventually, there would be two ServiceGroup drivers: db driver &
> tooz driver;
>
>
>
> Actually, things are the same for neutron, except that we don’t need to
> consider zk/memcached driver deprecation. I would like to refine current
> spec and propose a ”Agent Group and using tooz” spec, following the
> outlines above. What do you think, Kevin and Joshua? Thanks. J
>
>
>
> Best,
>
> Robin
>
>
>
> [1] https://wiki.openstack.org/wiki/NovaZooKeeperHeartbeat
>
> [2] https://review.openstack.org/#/c/168921/
>
> [3]
> https://review.openstack.org/#/c/138607/11/specs/liberty/approved/service-group-using-tooz.rst
>
> [4] ServiceGroup refactor code: https://review.openstack.org/#/c/172502/
>
>
>
>
>
>
>
>
>
>
>
> *发件人:* Wangbibo [mailto:wangbibo at huawei.com]
> *发送时间:* 2015年4月13日 16:52
> *收件人:* OpenStack Development Mailing List (not for usage questions)
> *主题:* [openstack-dev] 答复: [neutron] Neutron scaling datapoints?
>
>
>
> Hi Kevin,
>
>
>
> Totally agree with you that heartbeat from each agent is something that we
> cannot eliminate currently. Agent status depends on it, and further
> scheduler and HA depends on agent status.
>
>
>
> I proposed a Liberty spec for introducing open framework/pluggable agent
> status drivers.[1][2]  It allows us to use some other 3rd party backend
> to monitor agent status, such as zookeeper, memcached. Meanwhile, it
> guarantees backward compatibility so that users could still use db-based
> status monitoring mechanism as their default choice.
>
>
>
> Base on that, we may do further optimization on issues Attila and you
> mentioned. Thanks.
>
>
>
> [1] BP  -
> https://blueprints.launchpad.net/neutron/+spec/agent-group-and-status-drivers
>
> [2] Liberty Spec proposed - https://review.openstack.org/#/c/168921/
>
>
>
> Best,
>
> Robin
>
>
>
>
>
>
>
>
>
> *发件人:* Kevin Benton [mailto:blak111 at gmail.com <blak111 at gmail.com>]
> *发送时间:* 2015年4月11日 12:35
> *收件人:* OpenStack Development Mailing List (not for usage questions)
> *主题:* Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
>
>
> Which periodic updates did you have in mind to eliminate? One of the few
> remaining ones I can think of is sync_routers but it would be great if you
> can enumerate the ones you observed because eliminating overhead in agents
> is something I've been working on as well.
>
>
>
> One of the most common is the heartbeat from each agent. However, I don't
> think we can't eliminate them because they are used to determine if the
> agents are still alive for scheduling purposes. Did you have something else
> in mind to determine if an agent is alive?
>
>
>
> On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas <afazekas at redhat.com>
> wrote:
>
> I'm 99.9% sure, for scaling above 100k managed node,
> we do not really need to split the openstack to multiple smaller openstack,
> or use significant number of extra controller machine.
>
> The problem is openstack using the right tools SQL/AMQP/(zk),
> but in a wrong way.
>
> For example.:
> Periodic updates can be avoided almost in all cases
>
> The new data can be pushed to the agent just when it needed.
> The agent can know when the AMQP connection become unreliable (queue or
> connection loose),
> and needs to do full sync.
> https://bugs.launchpad.net/neutron/+bug/1438159
>
> Also the agents when gets some notification, they start asking for details
> via the
> AMQP -> SQL. Why they do not know it already or get it with the
> notification ?
>
>
> ----- Original Message -----
> > From: "Neil Jerram" <Neil.Jerram at metaswitch.com>
>
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> > Sent: Thursday, April 9, 2015 5:01:45 PM
> > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
> >
> > Hi Joe,
> >
> > Many thanks for your reply!
> >
> > On 09/04/15 03:34, joehuang wrote:
> > > Hi, Neil,
> > >
> > >  From theoretic, Neutron is like a "broadcast" domain, for example,
> > >  enforcement of DVR and security group has to touch each regarding host
> > >  where there is VM of this project resides. Even using SDN controller,
> the
> > >  "touch" to regarding host is inevitable. If there are plenty of
> physical
> > >  hosts, for example, 10k, inside one Neutron, it's very hard to
> overcome
> > >  the "broadcast storm" issue under concurrent operation, that's the
> > >  bottleneck for scalability of Neutron.
> >
> > I think I understand that in general terms - but can you be more
> > specific about the broadcast storm?  Is there one particular message
> > exchange that involves broadcasting?  Is it only from the server to
> > agents, or are there 'broadcasts' in other directions as well?
> >
> > (I presume you are talking about control plane messages here, i.e.
> > between Neutron components.  Is that right?  Obviously there can also be
> > broadcast storm problems in the data plane - but I don't think that's
> > what you are talking about here.)
> >
> > > We need layered architecture in Neutron to solve the "broadcast domain"
> > > bottleneck of scalability. The test report from OpenStack cascading
> shows
> > > that through layered architecture "Neutron cascading", Neutron can
> > > supports up to million level ports and 100k level physical hosts. You
> can
> > > find the report here:
> > >
> http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
> >
> > Many thanks, I will take a look at this.
> >
> > > "Neutron cascading" also brings extra benefit: One cascading Neutron
> can
> > > have many cascaded Neutrons, and different cascaded Neutron can
> leverage
> > > different SDN controller, maybe one is ODL, the other one is
> OpenContrail.
> > >
> > > ----------------Cascading Neutron-------------------
> > >              /         \
> > > --cascaded Neutron--   --cascaded Neutron-----
> > >         |                  |
> > > ---------ODL------       ----OpenContrail--------
> > >
> > >
> > > And furthermore, if using Neutron cascading in multiple data centers,
> the
> > > DCI controller (Data center inter-connection controller) can also be
> used
> > > under cascading Neutron, to provide NaaS ( network as a service )
> across
> > > data centers.
> > >
> > > ---------------------------Cascading Neutron--------------------------
> > >              /            |          \
> > > --cascaded Neutron--  -DCI controller-  --cascaded Neutron-----
> > >         |                 |            |
> > > ---------ODL------           |         ----OpenContrail--------
> > >                           |
> > > --(Data center 1)--   --(DCI networking)--  --(Data center 2)--
> > >
> > > Is it possible for us to discuss this in OpenStack Vancouver summit?
> >
> > Most certainly, yes.  I will be there from mid Monday afternoon through
> > to end Friday.  But it will be my first summit, so I have no idea yet as
> > to how I might run into you - please can you suggest!
> >
> > > Best Regards
> > > Chaoyi Huang ( Joe Huang )
> >
> > Regards,
> >       Neil
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
> --
>
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150414/eaccfbdc/attachment.html>