Open Stack

I recently witnessed a strange issue with libvirt when upgrading one of our clusters from Kilo to Liberty. I'm not really looking for a specific diagnosis here because of the large number of confounding factors and the relative ease of remediating it, but I'm interested to hear if anyone else has witnessed this particular problem.

Background is we had a number of Kilo-based clusters, all running Ubuntu 14.04.4 with OpenStack installed from the Ubuntu cloud archive. The upgrade process to Liberty involved upgrading the OpenStack components and their dependencies (including libvirt), then afterward upgrading all remaining packages via dist-upgrade (and staging a kernel upgrade from 3.13 to 4.4, to take effect on the next reboot). 7 clusters had all been upgraded successfully using this strategy.

One cluster, however, decided to get a bit weird. After the upgrade, 4 hypervisors showed that nova-compute was refusing to come up properly and was showing as enabled/down in nova service-list. Upon further investigation, nova-compute was starting up but was getting jammed on loading nwfilters. When I ran "virsh nwfilter-list", the command stalled indefinitely. Killing nova-compute and restarting libvirt-bin service allowed the command to work again, but it did not list any of the nova-instance-instance-* nwfilters. Once nova-compute was started, it tried to start loading the instance-specific filters and libvirt would wedge. I spent a while tinkering with the affected systems but could not find any way of correcting the issue other than rebooting the hypervisor, after which everything was fine.

Has anyone ever seen anything like this? libvirt was upgraded from 1.2.12 to 1.2.16. Hundreds of hypervisors had already received this exact same upgrade without showing this problem, and I have no idea how I could reproduce it. I'm interested to hear if anyone else has ever run into this and if they figured out what the root cause was, though I've already braced myself for tumbleweeds.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20170222/ba94930a/attachment.html>