[openstack-dev] [nova] VIF plugin issue _get_neutron_events

Ajay Kalambur (akalambu) akalambu at cisco.com
Sat Oct 8 23:10:18 UTC 2016


Basically the issue is seen in the following three lines of nova compute log. For that port even though it received the vif plugging event 2 mins before it waits for it and blocks and times out
Is there a race condition in the code that basically gets the events to wait for and the one where it registers for this callback
Any comments?

2016-10-09 07:35:24.869 6 DEBUG nova.compute.manager [req-afbc9c31-757a-478e-af8f-a4937e787a57 53a2bd71435c43e3953c37ffb9b38e21 56b04dfbb802430face997791412a62a - - -] [instance: 59b4f72b-5d1d-41ee-bc57-c57a8204383a] Received event network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 external_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:6509
2016-10-09 07:35:24.871 6 DEBUG nova.compute.manager [req-afbc9c31-757a-478e-af8f-a4937e787a57 53a2bd71435c43e3953c37ffb9b38e21 56b04dfbb802430face997791412a62a - - -] [instance: 59b4f72b-5d1d-41ee-bc57-c57a8204383a] No waiting events found dispatching network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 pop_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:545
2016-10-09 07:37:36.668 6 DEBUG nova.compute.manager [req-31f88ed2-ab3d-4b1d-a75b-b4e7b80b9202 28450924ed064558948f98eaec39fe6e 54e6da5850104d3682ef3500f3eb7a57 - - -] [instance: 59b4f72b-5d1d-41ee-bc57-c57a8204383a] Preparing to wait for external event network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 prepare_for_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:507

From: Ajay Kalambur <akalambu at cisco.com<mailto:akalambu at cisco.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Tuesday, October 4, 2016 at 10:56 AM
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: [openstack-dev] [nova] VIF plugin issue _get_neutron_events

Hi
When booting Vms with 3 interfaces and when port binding happens quick enough I sometimes see that the first VM booted on a compute node gets stuck waiting for VIF plugging event but looking at the neutron logs the notification is sent and also nova responded with a 200 OK.

When looking at the code in nova/virt/libvirt/driver.py we see in spawn() there is a call to _create_domain_and_network which calls _get_neutron_events

So in what scenarios can this notification happen to soon or is there a race condition somewhere in this code wherby even though neutron sends the notification and nova responds the code here blocks for 300 seconds since it missed the event.

Any clues on what maybe going on here. Its almost like the neutron notification happened too quickly while nova code was busy performing create_image

This happens only with the first VM on a compute node

Any ideas on how to fix this

Ajay

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161008/4b40282b/attachment.html>


More information about the OpenStack-dev mailing list