Hi all, 

I’ve recently upgraded Openstack from Mitaka to Ocata and now I’m facing an increased latency in instance creation.
Before the upgrade I was able to launch sequentially 400 VMs in 5 minutes and now I need at least 15 minutes. Attached the script I use to measure timings.

Here you can find the comparison between Mitaka and Ocata. I also noticed that in Ocata DHCP agent is slower and sometimes when I launch more than 30 VMs sequentially some of them doesn’t took an address.
Time of instance creation increase with the number of ports in the network. 

https://docs.google.com/spreadsheets/d/1eHjRxkw_5BmJ2p0rBJg9suGKmu_1YrSY05f50gKJxbE/edit?usp=sharing

I’ve enabled the osprofiler inside nova and neutron api and attached you can see that the most of the time is spent on wsgi component of neutron.
The time is spent on getting port list. I’ve also verified with cli listing only ports and I can confirm is a port list related issue.

I’m trying to profile the neutron-server with cProfile but I’ve some problems with multi-threading tracing.

Does anyone have faced this issue?  

Thank you!

Regards
Matteo