[openstack-dev] [all] Million level scalability test report from cascading

joehuang joehuang at huawei.com
Wed Apr 1 07:22:34 UTC 2015

Hi Rob

There is no API request failure under current hardware configuration. If you remove one physical server for NOVA API service or for scheduler service, then API request failure will happen under 1000 concurrency. Similar to Neutron. Before expansion to the current test environment hardware configuration, API request failure was observed.

For the system load during the steady-state, because the cascaded OpenStack simulator will make the VM status changed now and then ( and also for volume, port ), these status change will be periodically batch polled and synchronized to the cascading layer. The status change is more frequently than normal steady-state in one OpenStack instance. That's why you see the load is not very low even if there is no API request.

For VM operation failure rate, because the cascaded OpenStack is simulator, so it's not measured. The rate is co-related how the model is implemented in the cascaded simulator.


I don't know why I can't see my mail and your reply in the OpenStack-dev mail list, but I had thought that the mail has not been sent successfully, but someone else told me he saw the mail.

I only found your reply through the mail list achieve : http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg49335.html

Best Regards

Chaoyi Huang ( joehuang )

On 31 March 2015 at 22:05, joehuang <joehu... at huawei.com> wrote:

> Hi, all,


> During the last cross project meeting[1][2] for the next step of OpenStack

> cascading solution[3], the conclusion of the meeting is "OpenStack isn't

> ready for the project, and if he want's it ready sooner than later, joehuang

> needs to help make it ready by working on scaling being coded now", and the

> scaling is on the first priority for OpenStack community.


> We just finished the 1 million VMs semi-simulation test report[4] for

> OpenStack cascading solution, the most interesting findings during the test

> is, the cascading architecture can support million level ports in Neutron,

> and also million level VMs in Nova. And the test report also shows that

> OpenStack cascading solution can manage up to 100k physical hosts without

> challenge. Some scaling issues were found during the test and listed in the

> report.


> The conclusion of the report is:

> "According to the Phase I and Phase II test data analysis, due to the

> hardware resources limitation, the OpenStack cascading solution with current

> configuration can supports a maximum of 1 million virtual machines and is

> capable of handling 500 concurrent API request if L3 (DVR) mode is included

> or, 1000 concurrent API request if only L2 networking needed. It's up to

> deployment policy to use OpenStack cascading solution inside one site ( one

> data center) or multi-sites (multi-data centers), the maximal sites (data

> centers) supported are 100, i.e., 100 cascaded OpenStack instances."


> The test report is shared first, let's discuss the next step later.

Wow thats beautiful stuff.

The next time someone does a report like this, I'd like to suggest

some extra metrics to capture.

API failure rate: what % of API errors occur.

VM failure rate: what % of operations lead to a failed VM (e.g. not

deleted on delete, or not started on create, or didn't boot correctly)

block device failure rate similarly.

Looking in your results, I observe significant load in the

steady-state mode for most of the DB's. Thats a little worrying, if as

I assume steady-state means 'no new API calls being made'.



Robert Collins <rbtcoll... at hp.com>

Distinguished Technologist

HP Converged Cloud


OpenStack Development Mailing List (not for usage questions)

Unsubscribe: openstack-dev-requ... at lists.openstack.org?subject:unsubscribe


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150401/a8ae6610/attachment-0001.html>

More information about the OpenStack-dev mailing list