Open Stack

Hello Community,

I want to share Ironic liberty scale testing. The tests were performed on
200 hw nodes lab (see server configuration below).

Hardware configuration:
HP ProLiant DL380 Gen9
CPU: 2 x E5-2680 v3 @ 2.50GHz
RAM: 256.0 GB
DISK: 12 x EH0600JDYTL HP 600-GB 12G 15K 2.5 DP SAS HDD Raid 10
NIC: 8 x 10.0 Gbps

Environment configuration:
3 controllers + ceph (*api, *scheduller, 1 active nova-compute, *ceph)
2 ironic (ironic-conductor)

We used the following test scenario:

1. Enrol hardware servers to Ironic
2. Launch rally boot_and_list
<https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/nova/boot-and-list.json>,
boot_and_delete
<https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/nova/boot-and-delete.json>
nova scenarios.
3. On each hardware server create 120 VMs with 1.5 Gb of RAM, 2 CPU and 11
GB of disk.
4. Enroll VMs to Ironic with libvirt driver [7], if needed.
5. Launch rally scenarios.
6. Go to step 4.

In general we add 4k+ nodes to Ironic and boot 4k Nova instances.
Brief test results are shown in the tables below.

boot_server

Total Nova instances

Min (sec)

95%ile (sec)

Max (sec)

Success

200

190.644

273.491

284.578

100.00%

500

168.075

293.545

335.318

100.00%

1000

210.525

286.275

315.824

99.00%

1800

158.064

526.742

569.499

99.50%

3700

160.575

559.092

662.142

99.20%

list_servers

Total Nova instances

Min (sec)

95%ile (sec)

Max (sec)

Success

200

0.258

0.687

0.775

100.00%

500

0.324

0.996

1.075

100.00%

1000

0.28

1.017

1.177

100.00%

1800

0.291

1.366

1.927

100.00%

3700

0.135

0.854

3.654

100.00%

delete_server

Total Nova instances

Min (sec)

95%ile (sec)

Max (sec)

Success

1000

20.364

20.731

20.879

100.00%

2200

20.325

20.84

21.047

100.00%

3700

20.342

20.874

21.01

100.00%

Nova tuning:
*max_concurrent_builds*: (IntOpt) Maximum number of instance builds to run
concurrently. Default is 10. In case of Ironic this value can be safely
increased since provisioning is done on the separate servers. We used 50
and didn't saw any performance degradation.

*scheduler_host_subset_size:* (IntOpt) New instances will be scheduled on a
host chosen randomly from a subset of the N best hosts. This property
defines the subset size that a host is chosen from. A value of 1 chooses
the first host returned by the weighing functions. This value must be at
least 1. Any value less than 1 will be ignored, and 1 will be used instead.
Since resource claim are performed at the nova compute, there might be
situations when several instances are placed on single compute
(hypervisor). As result one of them will fail to build. By increasing
*scheduler_host_subset_size
*to number of hypervisors we minimize such errors.

Issues:

   1. Claim resources at nova compute leads to situations when scheduler
   places several instances to single hypervisor. In case of Ironic one of
   instances will always fail to build. This situation is especially
   pronounced when we request boot instances with amount near to total amount
   of ironic nodes (nova hypervisors). Some work is being done already [1],
   [2], [3]

   2. Nova resource tracker
   <https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L46>uses
   shared lock for compute when performing operations. In case of Ironic it
   leads to lock starvation with higher number of ironic nodes.

   3. Deleting single Nova instance initiates resource update for all
   hyporvisors in the cloud [4]. It makes booting/deleing operations super
   slow especially with issue 2. [5]



[1]
https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html
[2] https://review.openstack.org/204641/
<https://review.openstack.org/#/c/204641/>
[3] https://review.openstack.org/225546/
<https://review.openstack.org/#/c/225546/>
[4] https://launchpad.net/bugs/1552446
[5] https://review.openstack.org/287498/
<https://review.openstack.org/#/c/287498/>
[6] https://review.openstack.org/287485/
[7] https://review.openstack.org <https://review.openstack.org/#/c/294688/>
/294688/ <https://review.openstack.org/#/c/294688/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160411/9b5a7e41/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rally_reports.tar.gz
Type: application/x-gzip
Size: 473248 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160411/9b5a7e41/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cacti-results.tar.gz
Type: application/x-gzip
Size: 22484363 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160411/9b5a7e41/attachment-0003.bin>