<div dir="ltr"><div><div><div><div><div><div><div><div><font size="2">Hello Community,<br><br></font></div><font size="2">I want to share Ironic liberty scale testing. The tests were performed on 200 hw nodes lab (see server configuration below). <br><br></font><div><font size="2">Hardware configuration:<br>HP ProLiant DL380 Gen9<br>CPU: 2 x E5-2680 v3 @ 2.50GHz<br>RAM: 256.0 GB<br>DISK: 12 x EH0600JDYTL HP 600-GB 12G 15K 2.5 DP SAS HDD Raid 10<br>NIC: 8 x 10.0 Gbps<br><br></font></div><div><font size="2">Environment configuration:<br></font></div><div><font size="2">3 controllers + ceph (*api, *scheduller, 1 active nova-compute, *ceph)<br></font></div><font size="2">2 ironic (ironic-conductor)</font><font size="2"><br><br></font></div><font size="2">We used the following test scenario:<br><br></font></div><font size="2">1. Enrol hardware servers to Ironic <br></font></div><font size="2">2. Launch rally <a href="https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/nova/boot-and-list.json" target="_blank">boot_and_list</a>, <a href="https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/nova/boot-and-delete.json" target="_blank">boot_and_delete</a> nova scenarios.<br></font></div><font size="2">3. On each hardware server create 120 VMs with 1.5 Gb of RAM, 2 CPU and 11 GB of disk.<br></font></div><font size="2">4. Enroll VMs to Ironic with libvirt driver [7], if needed.<br></font></div><font size="2">5. Launch rally scenarios.<br></font></div><font size="2">6. Go to step 4.<br></font><div><div><div><div><div><div><div><font size="2"><br></font></div><div><font size="2">In general we add 4k+ nodes to Ironic and boot 4k Nova instances.<br></font></div><div><font size="2">Brief test results are shown in the tables below. <br></font></div><div><font size="2"><br></font></div><div><font size="2">boot_server<br></font></div><div><div dir="ltr" style="margin-left:0pt"><table style="border:medium none;border-collapse:collapse;width:624px"><tbody><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Total Nova instances</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Min (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">95%ile (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Max (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Success</font></p></td></tr><tr style="height:29px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">200</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">190.644</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">273.491</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">284.578</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">500</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">168.075</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">293.545</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">335.318</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1000</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">210.525</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">286.275</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">315.824</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">99.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1800</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">158.064</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">526.742</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">569.499</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">99.50%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">3700</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">160.575</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">559.092</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">662.142</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">99.20%</font></p></td></tr></tbody></table></div><font size="2"> <br></font></div><div><font size="2">list_servers<br></font><div dir="ltr" style="margin-left:0pt"><table style="border:medium none;border-collapse:collapse;width:624px"><colgroup><col width="*"><col width="*"><col width="*"><col width="*"><col width="*"></colgroup><tbody><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Total Nova instances</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Min (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">95%ile (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Max (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Success</font></p></td></tr><tr style="height:31px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">200</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.258</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.687</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.775</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">500</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.324</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.996</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1.075</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1000</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.28</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1.017</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1.177</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1800</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.291</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1.366</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1.927</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">3700</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.135</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">0.854</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">3.654</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr></tbody></table></div><font size="2"><br></font></div><div><font size="2">delete_server<br></font><div dir="ltr" style="margin-left:0pt"><table style="border:medium none;border-collapse:collapse;width:624px"><colgroup><col width="*"><col width="*"><col width="*"><col width="*"><col width="*"></colgroup><tbody><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Total Nova instances</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Min (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">95%ile (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Max (sec)</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">Success</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">1000</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">20.364</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">20.731</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">20.879</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">2200</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">20.325</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">20.84</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">21.047</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr><tr style="height:0px"><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">3700</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">20.342</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">20.874</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">21.01</font></p></td><td style="border-width:1px;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:7px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><font size="2">100.00%</font></p></td></tr></tbody></table></div></div><div><font size="2"><br></font></div><div><font size="2">Nova tuning:<br><b><i>max_concurrent_builds</i></b>: (IntOpt) Maximum number of instance builds to run concurrently. Default is 10. In case of Ironic this value can be safely increased since provisioning is done on the separate servers. We used 50 and didn't saw any performance degradation.<br><br><i><b>scheduler_host_subset_size:</b></i> (IntOpt) New instances will be scheduled on a host chosen randomly from a subset of the N best hosts. This property defines the subset size that a host is chosen from. A value of 1 chooses the first host returned by the weighing functions. This value must be at least 1. Any value less than 1 will be ignored, and 1 will be used instead. Since resource claim are performed at the nova compute, there might be situations when several instances are placed on single compute (hypervisor). As result one of them will fail to build. By increasing <i><b>scheduler_host_subset_size </b></i>to number of hypervisors we minimize such errors.<br></font></div><div><font size="2"><br></font></div><div><font size="2">Issues:<br></font><ol><li><font size="2">Claim resources at nova compute leads to situations when scheduler places several instances to single hypervisor. In case of Ironic one of instances will always fail to build. This situation is especially pronounced when we request boot instances with amount near to total amount of ironic nodes (nova hypervisors). Some work is being done already [1], [2], [3]<br><br></font></li><li><font size="2"><a href="https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L46">Nova resource tracker </a>uses shared lock for compute when performing operations. In case of Ironic it leads to lock starvation with higher number of ironic nodes.<br><br></font></li><li><font size="2">Deleting single Nova instance initiates resource update for all hyporvisors in the cloud [4]. It makes booting/deleing operations super slow especially with issue 2. [5]<br></font></li></ol><font size="2"></font></div><br><div><font size="2"><br>[1] <a href="https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html">https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html</a><br>[2] <a href="https://review.openstack.org/#/c/204641/">https://review.openstack.org/204641/</a><br>[3] <a href="https://review.openstack.org/#/c/225546/">https://review.openstack.org/225546/</a><br>[4] <a href="https://launchpad.net/bugs/1552446">https://launchpad.net/bugs/1552446</a><br>[5] </font><a href="https://review.openstack.org/#/c/287498/">https://review.openstack.org/287498/</a><font size="2"><br>[6] <a href="https://review.openstack.org/287485/">https://review.openstack.org/287485/</a><br>[7] <a href="https://review.openstack.org/#/c/294688/" target="_blank">https://review.openstack.org</a><a href="https://review.openstack.org/#/c/294688/" target="_blank">/294688/</a><br><br></font></div></div></div></div></div></div></div></div>