[openstack-dev] 答复: [neutron] Neutron scaling datapoints?

Joshua Harlow harlowja at outlook.com
Tue Apr 14 15:33:14 UTC 2015


Daniel Comnea wrote:
> Joshua,
>
> those are old and have been fixed/ documented on Consul side.
> As for ZK, i have nothing against it, just wish you good luck running it
> in a multi cross-DC setup :)

Totally fair, although I start to question a cross-DC setup of things, 
and why that's needed in this (and/or any) architecture, but to each 
there own ;)

>
> Dani
>
> On Mon, Apr 13, 2015 at 11:37 PM, Joshua Harlow <harlowja at outlook.com
> <mailto:harlowja at outlook.com>> wrote:
>
>     Did the following get addressed?
>
>     https://aphyr.com/posts/316-__call-me-maybe-etcd-and-consul
>     <https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul>
>
>     Seems like quite a few things got raised in that post about etcd/consul.
>
>     Maybe they are fixed, idk...
>
>     https://aphyr.com/posts/291-__call-me-maybe-zookeeper
>     <https://aphyr.com/posts/291-call-me-maybe-zookeeper> though worked
>     as expected (and without issue)...
>
>     I quote:
>
>     '''
>     Recommendations
>
>     Use Zookeeper. It’s mature, well-designed, and battle-tested.
>     Because the consequences of its connection model and linearizability
>     properties are subtle, you should, wherever possible, take advantage
>     of tested recipes and client libraries like Curator, which do their
>     best to correctly handle the complex state transitions associated
>     with session and connection loss.
>     '''
>
>     Daniel Comnea wrote:
>
>         My $2 cents:
>
>         I like the 3rd party backend however instead of ZK wouldn't
>         Consul [1]
>         fit better due to lighter/ out of box multi DC awareness?
>
>         Dani
>
>         [1] Consul - https://www.consul.io/
>
>
>         On Mon, Apr 13, 2015 at 9:51 AM, Wangbibo <wangbibo at huawei.com
>         <mailto:wangbibo at huawei.com>
>         <mailto:wangbibo at huawei.com <mailto:wangbibo at huawei.com>>> wrote:
>
>              Hi Kevin,____
>
>              __ __
>
>              Totally agree with you that heartbeat from each agent is
>         something
>              that we cannot eliminate currently. Agent status depends on
>         it, and
>              further scheduler and HA depends on agent status.____
>
>              __ __
>
>              I proposed a Liberty spec for introducing open
>         framework/pluggable
>              agent status drivers.[1][2]  It allows us to use some other
>         3^rd
>              party backend to monitor agent status, such as zookeeper,
>         memcached.
>              Meanwhile, it guarantees backward compatibility so that
>         users could
>              still use db-based status monitoring mechanism as their default
>              choice.____
>
>              __ __
>
>              Base on that, we may do further optimization on issues
>         Attila and
>              you mentioned. Thanks. ____
>
>              __ __
>
>              [1] BP  -
>         https://blueprints.launchpad.__net/neutron/+spec/agent-group-__and-status-drivers____
>         <https://blueprints.launchpad.net/neutron/+spec/agent-group-and-status-drivers____>
>
>              [2] Liberty Spec proposed -
>         https://review.openstack.org/#__/c/168921/____
>         <https://review.openstack.org/#/c/168921/____>
>
>              __ __
>
>              Best,____
>
>              Robin____
>
>              __ __
>
>              __ __
>
>              __ __
>
>              __ __
>
>              *发件人:*Kevin Benton [mailto:blak111 at gmail.com
>         <mailto:blak111 at gmail.com>
>         <mailto:blak111 at gmail.com <mailto:blak111 at gmail.com>>]
>              *发送时间:*2015年4月11日12:35
>              *收件人:*OpenStack Development Mailing List (not for usage
>         questions)
>              *主题:*Re: [openstack-dev] [neutron] Neutron scaling
>         datapoints?____
>
>              __ __
>
>              Which periodic updates did you have in mind to eliminate?
>         One of the
>              few remaining ones I can think of is sync_routers but it
>         would be
>              great if you can enumerate the ones you observed because
>         eliminating
>              overhead in agents is something I've been working on as
>         well.____
>
>              __ __
>
>              One of the most common is the heartbeat from each agent.
>         However, I
>              don't think we can't eliminate them because they are used to
>              determine if the agents are still alive for scheduling
>         purposes. Did
>              you have something else in mind to determine if an agent is
>         alive?____
>
>              __ __
>
>              On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas
>         <afazekas at redhat.com <mailto:afazekas at redhat.com>
>         <mailto:afazekas at redhat.com <mailto:afazekas at redhat.com>>>
>         wrote:____
>
>              I'm 99.9% sure, for scaling above 100k managed node,
>              we do not really need to split the openstack to multiple
>         smaller
>              openstack,
>              or use significant number of extra controller machine.
>
>              The problem is openstack using the right tools SQL/AMQP/(zk),
>              but in a wrong way.
>
>              For example.:
>              Periodic updates can be avoided almost in all cases
>
>              The new data can be pushed to the agent just when it needed.
>              The agent can know when the AMQP connection become
>         unreliable (queue
>              or connection loose),
>              and needs to do full sync.
>         https://bugs.launchpad.net/__neutron/+bug/1438159
>         <https://bugs.launchpad.net/neutron/+bug/1438159>
>
>              Also the agents when gets some notification, they start
>         asking for
>              details via the
>              AMQP -> SQL. Why they do not know it already or get it with the
>              notification ?
>
>
>              ----- Original Message -----
>         >   From: "Neil Jerram" <Neil.Jerram at metaswitch.com
>         <mailto:Neil.Jerram at metaswitch.com>
>         <mailto:Neil.Jerram at __metaswitch.com
>         <mailto:Neil.Jerram at metaswitch.com>>>____
>
>         >   To: "OpenStack Development Mailing List (not for usage
>         questions)"
>         <openstack-dev at lists.__openstack.org
>         <mailto:openstack-dev at lists.openstack.org>
>         <mailto:openstack-dev at lists.__openstack.org
>         <mailto:openstack-dev at lists.openstack.org>>>
>
>          >  Sent: Thursday, April 9, 2015 5:01:45 PM
>          >  Subject: Re: [openstack-dev] [neutron] Neutron scaling
>         datapoints?
>          >
>          >  Hi Joe,
>          >
>          >  Many thanks for your reply!
>          >
>          >  On 09/04/15 03:34, joehuang wrote:
>          > > Hi, Neil,
>          > >
>          > >  From theoretic, Neutron is like a "broadcast" domain, for
>         example,
>          > >  enforcement of DVR and security group has to touch each
>              regarding host
>          > >  where there is VM of this project resides. Even using SDN
>              controller, the
>          > > "touch" to regarding host is inevitable. If there are plenty of
>              physical
>          > >  hosts, for example, 10k, inside one Neutron, it's very hard to
>              overcome
>          > >  the "broadcast storm" issue under concurrent operation,
>         that's the
>          > >  bottleneck for scalability of Neutron.
>          >
>          >  I think I understand that in general terms - but can you be more
>          >  specific about the broadcast storm?  Is there one particular
>         message
>          >  exchange that involves broadcasting?  Is it only from the
>         server to
>          >  agents, or are there 'broadcasts' in other directions as well?
>          >
>          >  (I presume you are talking about control plane messages
>         here, i.e.
>          >  between Neutron components.  Is that right?  Obviously there can
>              also be
>          >  broadcast storm problems in the data plane - but I don't
>         think that's
>          >  what you are talking about here.)
>          >
>          > > We need layered architecture in Neutron to solve the "broadcast
>              domain"
>          > > bottleneck of scalability. The test report from OpenStack
>              cascading shows
>          > > that through layered architecture "Neutron cascading",
>         Neutron can
>          > > supports up to million level ports and 100k level physical
>              hosts. You can
>          > > find the report here:
>          > >
>         http://www.slideshare.net/__JoeHuang7/test-report-for-__open-stack-cascading-solution-__to-support-1-million-v-ms-in-__100-data-centers
>         <http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers>
>          >
>          >  Many thanks, I will take a look at this.
>          >
>          > > "Neutron cascading" also brings extra benefit: One cascading
>              Neutron can
>          > > have many cascaded Neutrons, and different cascaded Neutron can
>              leverage
>          > > different SDN controller, maybe one is ODL, the other one is
>              OpenContrail.
>          > >
>          > > ----------------Cascading Neutron-------------------
>          > >              /         \
>          > > --cascaded Neutron--   --cascaded Neutron-----
>          > >         |                  |
>          > > ---------ODL------       ----OpenContrail--------
>          > >
>          > >
>          > > And furthermore, if using Neutron cascading in multiple data
>              centers, the
>          > > DCI controller (Data center inter-connection controller) can
>              also be used
>          > > under cascading Neutron, to provide NaaS ( network as a service
>              ) across
>          > > data centers.
>          > >
>          > > ---------------------------__Cascading
>              Neutron-----------------------__---
>          > >              /            |          \
>          > > --cascaded Neutron--  -DCI controller-  --cascaded Neutron-----
>          > >         |                 |            |
>          > > ---------ODL------           |         ----OpenContrail--------
>          > >                           |
>          > > --(Data center 1)--   --(DCI networking)--  --(Data center 2)--
>          > >
>          > > Is it possible for us to discuss this in OpenStack
>         Vancouver summit?
>          >
>          >  Most certainly, yes.  I will be there from mid Monday afternoon
>              through
>          >  to end Friday.  But it will be my first summit, so I have no
>         idea
>              yet as
>          >  to how I might run into you - please can you suggest!
>          >
>          > > Best Regards
>          > > Chaoyi Huang ( Joe Huang )
>          >
>          >  Regards,
>          >        Neil
>          >
>          >
>
>         ______________________________________________________________________________
>          >  OpenStack Development Mailing List (not for usage questions)
>          >  Unsubscribe:
>         OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         <http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>         >
>         http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>         >
>
>
>         ______________________________________________________________________________
>              OpenStack Development Mailing List (not for usage questions)
>              Unsubscribe:
>         OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         <http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>         http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev____
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev____>
>
>
>
>              ____
>
>              __ __
>
>              -- ____
>
>              Kevin Benton____
>
>
>
>         ______________________________________________________________________________
>              OpenStack Development Mailing List (not for usage questions)
>              Unsubscribe:
>         OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         <http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>         http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>         ______________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>     ______________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list