Open Stack

----- Original Message -----
> From: "Wangbibo" <wangbibo at huawei.com>
> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
> Sent: Monday, April 13, 2015 10:51:39 AM
> Subject: [openstack-dev] 答复:  [neutron] Neutron scaling datapoints?
> 
> 
> 
> Hi Kevin,
> 
> 
> 
> Totally agree with you that heartbeat from each agent is something that we
> cannot eliminate currently. Agent status depends on it, and further
> scheduler and HA depends on agent status.
> 
------------
Actually we could eliminate it regarding to the q-agt,
The q-agt can be monitored by the n-cpu
n-cpu also should change his status to dead when the q-agt dead.

So neutron could reuse the aliveness data from n-cpu aliveness.

Sooner or later I will suggest a direct
connection between n-cpu and q-agt anyway.

----------

Also it is possible to implement the is_up by a dummy message send:

- All agent has to have an auto-delete queue, which consumed only by the agent.

- The is_up can use the Default exchange and start publishing a message
  with `immediate` flag.
  If the broker does not refuses it, the target system is alive.

https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.publish

This method has the same issue as the current memcached driver,
each is_up is a tcp request/response which consumes too many
time and resources when you `list` 100k node.

-----------

Actually the recommended method is:

Have a service which is:
 - HA (3(+) node)
 - Really able to use multiple threads (not cpython)
 - Does not do a real state change when the service state did not changed
 - The availability is based on the tcp connection health, which is checked either by
   - Frequent TCP keep-alive packages managed by the kernel
   - By an Application level payload
 - The service state interested parties are notified about state changes only
   when state change happened

For ex.: Zookeeper with ephemeral znodes.

Have a second service:
 - Which is subscribed to the first one (it can use tooz)
 - Consulting the service state changes with the database (add is_alive field to the table)
 - You can run multiple instances for HA,
   they does the same DB change with small delay, no split brain issue,
   but you should not run more than 3 instances.

Benefits of this approach compered to other zk based approaches.:

- Does not have all API worker to keep all service data in memory !
- Does not have all worker to cross reference the DB data with something else
  (especially in list context)
- Selecting only the dead or alive nodes will be simple and efficient

Cons.:

- 0.001 DB UPDATE/sec expected at 100k node (nothing :))
- Additional service component, but actually it saves memory 

PS.:
The zk has one other advantage compared to mc or the current db driver:
 Faster state change detection and report. 
 
> 
> I proposed a Liberty spec for introducing open framework/pluggable agent
> status drivers.[1][2] It allows us to use some other 3 rd party backend to
> monitor agent status, such as zookeeper, memcached. Meanwhile, it guarantees
> backward compatibility so that users could still use db-based status
> monitoring mechanism as their default choice.
> 
> 
> 
> Base on that, we may do further optimization on issues Attila and you
> mentioned. Thanks.
> 
> 
> 
> [1] BP -
> https://blueprints.launchpad.net/neutron/+spec/agent-group-and-status-drivers
> 
> [2] Liberty Spec proposed - https://review.openstack.org/#/c/168921/
> 
> 
> 
> Best,
> 
> Robin
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 发件人 : Kevin Benton [mailto:blak111 at gmail.com]
> 发送时间 : 2015 年 4 月 11 日 12:35
> 收件人 : OpenStack Development Mailing List (not for usage questions)
> 主题 : Re: [openstack-dev] [neutron] Neutron scaling datapoints?
> 
> 
> 
> 
> 
> Which periodic updates did you have in mind to eliminate? One of the few
> remaining ones I can think of is sync_routers but it would be great if you
> can enumerate the ones you observed because eliminating overhead in agents
> is something I've been working on as well.
> 
> 
> 
> 
> 
> One of the most common is the heartbeat from each agent. However, I don't
> think we can't eliminate them because they are used to determine if the
> agents are still alive for scheduling purposes. Did you have something else
> in mind to determine if an agent is alive?
> 
> 
> 
> 
> 
> On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas < afazekas at redhat.com >
> wrote:
> 
> I'm 99.9% sure, for scaling above 100k managed node,
> we do not really need to split the openstack to multiple smaller openstack,
> or use significant number of extra controller machine.
> 
> The problem is openstack using the right tools SQL/AMQP/(zk),
> but in a wrong way.
> 
> For example.:
> Periodic updates can be avoided almost in all cases
> 
> The new data can be pushed to the agent just when it needed.
> The agent can know when the AMQP connection become unreliable (queue or
> connection loose),
> and needs to do full sync.
> https://bugs.launchpad.net/neutron/+bug/1438159
> 
> Also the agents when gets some notification, they start asking for details
> via the
> AMQP -> SQL. Why they do not know it already or get it with the notification
> ?
> 
> 
> ----- Original Message -----
> > From: "Neil Jerram" < Neil.Jerram at metaswitch.com >
> 
> 
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> > openstack-dev at lists.openstack.org >
> > Sent: Thursday, April 9, 2015 5:01:45 PM
> > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
> > 
> > Hi Joe,
> > 
> > Many thanks for your reply!
> > 
> > On 09/04/15 03:34, joehuang wrote:
> > > Hi, Neil,
> > > 
> > > From theoretic, Neutron is like a "broadcast" domain, for example,
> > > enforcement of DVR and security group has to touch each regarding host
> > > where there is VM of this project resides. Even using SDN controller, the
> > > "touch" to regarding host is inevitable. If there are plenty of physical
> > > hosts, for example, 10k, inside one Neutron, it's very hard to overcome
> > > the "broadcast storm" issue under concurrent operation, that's the
> > > bottleneck for scalability of Neutron.
> > 
> > I think I understand that in general terms - but can you be more
> > specific about the broadcast storm? Is there one particular message
> > exchange that involves broadcasting? Is it only from the server to
> > agents, or are there 'broadcasts' in other directions as well?
> > 
> > (I presume you are talking about control plane messages here, i.e.
> > between Neutron components. Is that right? Obviously there can also be
> > broadcast storm problems in the data plane - but I don't think that's
> > what you are talking about here.)
> > 
> > > We need layered architecture in Neutron to solve the "broadcast domain"
> > > bottleneck of scalability. The test report from OpenStack cascading shows
> > > that through layered architecture "Neutron cascading", Neutron can
> > > supports up to million level ports and 100k level physical hosts. You can
> > > find the report here:
> > > http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
> > 
> > Many thanks, I will take a look at this.
> > 
> > > "Neutron cascading" also brings extra benefit: One cascading Neutron can
> > > have many cascaded Neutrons, and different cascaded Neutron can leverage
> > > different SDN controller, maybe one is ODL, the other one is
> > > OpenContrail.
> > > 
> > > ----------------Cascading Neutron-------------------
> > > / \
> > > --cascaded Neutron-- --cascaded Neutron-----
> > > | | 
> > > ---------ODL------ ----OpenContrail--------
> > > 
> > > 
> > > And furthermore, if using Neutron cascading in multiple data centers, the
> > > DCI controller (Data center inter-connection controller) can also be used
> > > under cascading Neutron, to provide NaaS ( network as a service ) across
> > > data centers.
> > > 
> > > ---------------------------Cascading Neutron--------------------------
> > > / | \
> > > --cascaded Neutron-- -DCI controller- --cascaded Neutron-----
> > > | | | 
> > > ---------ODL------ | ----OpenContrail--------
> > > | 
> > > --(Data center 1)-- --(DCI networking)-- --(Data center 2)--
> > > 
> > > Is it possible for us to discuss this in OpenStack Vancouver summit?
> > 
> > Most certainly, yes. I will be there from mid Monday afternoon through
> > to end Friday. But it will be my first summit, so I have no idea yet as
> > to how I might run into you - please can you suggest!
> > 
> > > Best Regards
> > > Chaoyi Huang ( Joe Huang )
> > 
> > Regards,
> > Neil
> > 
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> 
> 
> 
> 
> 
> --
> 
> 
> Kevin Benton
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>