Open Stack

Neil Jerram wrote:
> Hi again Joe, (+ list)
>
> On 11/04/15 02:00, joehuang wrote:
>> Hi, Neil,
>>
>> See inline comments.
>>
>> Best Regards
>>
>> Chaoyi Huang
>>
>> ________________________________________
>> From: Neil Jerram [Neil.Jerram at metaswitch.com]
>> Sent: 09 April 2015 23:01
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>>
>> Hi Joe,
>>
>> Many thanks for your reply!
>>
>> On 09/04/15 03:34, joehuang wrote:
>>> Hi, Neil,
>>>
>>> From theoretic, Neutron is like a "broadcast" domain, for example,
>>> enforcement of DVR and security group has to touch each regarding
>>> host where there is VM of this project resides. Even using SDN
>>> controller, the "touch" to regarding host is inevitable. If there are
>>> plenty of physical hosts, for example, 10k, inside one Neutron, it's
>>> very hard to overcome the "broadcast storm" issue under concurrent
>>> operation, that's the bottleneck for scalability of Neutron.
>>
>> I think I understand that in general terms - but can you be more
>> specific about the broadcast storm? Is there one particular message
>> exchange that involves broadcasting? Is it only from the server to
>> agents, or are there 'broadcasts' in other directions as well?
>>
>> [[joehuang]] for example, L2 population, Security group rule update,
>> DVR route update. Both direction in different scenario.
>
> Thanks. In case it's helpful to see all the cases together, sync_routers
> (from the L3 agent) was also mentioned in other part of this thread.
> Plus of course the liveness reporting from all agents.
>
>> (I presume you are talking about control plane messages here, i.e.
>> between Neutron components. Is that right? Obviously there can also be
>> broadcast storm problems in the data plane - but I don't think that's
>> what you are talking about here.)
>>
>> [[joehuang]] Yes, controll plane here.
>
> Thanks for confirming that.
>
>>> We need layered architecture in Neutron to solve the "broadcast
>>> domain" bottleneck of scalability. The test report from OpenStack
>>> cascading shows that through layered architecture "Neutron
>>> cascading", Neutron can supports up to million level ports and 100k
>>> level physical hosts. You can find the report here:
>>> http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
>>>
>>
>> Many thanks, I will take a look at this.
>
> It was very interesting, thanks. And by following through your links I
> also learned more about Nova cells, and about how some people question
> whether we need any kind of partitioning at all, and should instead
> solve scaling/performance problems in other ways... It will be
> interesting to see how this plays out.
>
> I'd still like to see more information, though, about how far people
> have scaled OpenStack - and in particular Neutron - as it exists today.
> Surely having a consensus set of current limits is an important input
> into any discussion of future scaling work.

+2 to this...

Shooting for the moon (although nice in theory) is not so useful when 
you can't even get up a hill ;)

>
> For example, Kevin mentioned benchmarking where the Neutron server
> processed a liveness update in <50ms and a sync_routers in 300ms.
> Suppose, the liveness update time was 50ms (since I don't know in detail
> what that < means) and agents report liveness every 30s. Does that mean
> that a single Neutron server can only support 600 agents?
>
> I'm also especially interested in the DHCP agent, because in Calico we
> have one of those on every compute host. We've just run tests which
> appeared to be hitting trouble from just 50 compute hosts onwards, and
> apparently because of DHCP agent communications. We need to continue
> looking into that and report findings properly, but if anyone already
> has any insights, they would be much appreciated.
>
> Many thanks,
> Neil
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev