Open Stack

Hi all,

we have the following setup:
- Openstack Ocata deployed with Openstack Ansible (v15.1.7)
- 66 compute nodes, each having between 50 and 150 VMs, depending on their hardware configuration
- we don't use Ceilometer (so not adding extra load on RabbitMQ cluster)
- using Openvswitch HA with DVR
- all messaging are going through a 3 servers RabbitMQ cluster
- we now have 3 CCs hosting (initially had 2) hosting every other internal service

What happens is, when we create a large number of VMs (it's something we do on a daily basis, just to test different types of VMs and apps, around 300 VMs), there are some of them that don't get the network interface attached in a reasonable time.
After investigating, we can see that Neutron Openvswitch agent sees the port attached to the server, from an Openstack point of view, I can see the tap interface created in Openvswitch using both its logs and dmesg, but I can see nova attaching the interface after a huge amount of time. (I could see even 45 minutes delay)

Since I can't see any reasonable errors I could take care of, my last chance is this mailing list.
Only thing I can think of, is that maybe libvirt is not able to attach the interface in a reasonable amount of time. But still, 45 minutes is way too much.

At the moment:
vif_plugging_is_fatal = True
vif_plugging_timeout = 600 (modified from default 300s)

That's because we needed VMs with networking. Otherwise, if either with error, either with no network, it's the same thing for us.

Thanks,

--

Radu Popescu <radu.popescu at emag.ro<mailto:radu.popescu at emag.ro>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180516/5fb7814d/attachment.html>