[openstack-dev] [Nova] [Neutron] heal_instance_info_cache_interval - Can we kill it?

Assaf Muller amuller at redhat.com
Wed May 28 14:39:39 UTC 2014



----- Original Message -----
> Hi,
> 
> Sorry somehow I missed this email. I don't think you want to disable it,
> though we can definitely have it run less often. The issue with disabling it
> is if one of the notifications from neutron->nova never gets sent
> successfully to nova (neutron-server is restarted before the event is sent
> or some other internal failure). Nova will never update it's cache if the
> heal_instance_info_cache_interval is set to 0.

The thing is, this periodic healing doesn't imply correctness either.
In the case where you lose a notification and the compute node hosting
the VM is hosting a non-trivial amount of VMs it can take (With the default
of 60 seconds) dozens of minutes to update the cache, since you only
update a VM a minute. I could understand the use of a sanity check if it
was performed much more often, but as it is now it seems useless to me
since you can't really rely on it.

What I'm trying to say is that with the inefficiency of the implementation,
coupled with Neutron's default plugin inability to cope with a "large"
amount of API calls, I feel like the disadvantages outweigh the
advantages when it comes to the cache healing.

How would you feel about disabling it, optimizing the implementation
(For example by introducing a new networking_for_instance API verb to Neutron?)
then enabling it again?

> The neutron->nova events help
> to ensure that the nova info_cache is up to date sooner by having neutron
> inform nova whenever a port's data has changed (@Joe Gordon - this happens
> regardless of virt driver).
> 
> If you're using the libvirt virt driver the neutron->nova events will also be
> used to ensure that the networking is 'ready' before the instance is powered
> on.
> 
> Best,
> 
> Aaron
> 
> P.S: we're working on making the heal_network call to neutron a lot less
> expensive as well in the future.
> 
> 
> 
> 
> On Tue, May 27, 2014 at 7:25 PM, Joe Gordon < joe.gordon0 at gmail.com > wrote:
> 
> 
> 
> 
> 
> 
> On Wed, May 21, 2014 at 6:21 AM, Assaf Muller < amuller at redhat.com > wrote:
> 
> 
> Dear Nova aficionados,
> 
> Please make sure I understand this correctly:
> Each nova compute instance selects a single VM out of all of the VMs
> that it hosts, and every <heal_instance_info_cache_interval> seconds
> queries Neutron for all of its networking information, then updates
> Nova's DB.
> 
> If the information above is correct, then I fail to see how that
> is in anyway useful. For example, for a compute node hosting 20 VMs,
> it would take 20 minutes to update the last one. Seems unacceptable
> to me.
> 
> Considering Icehouse's Neutron to Nova notifications, my question
> is if we can change the default to 0 (Disable the feature), deprecate
> it, then delete it in the K cycle. Is there a good reason not to do this?
> 
> Based on the patch that introduced this function [0] you may be on to
> something, but AFAIK unfortunately the neutron to nova notifications only
> work in libvirt right now [1], so I don' think we can fully deprecate this
> periodic task. That being said turning it off by default may be an option.
> Have you tried disabling this feature and seeing what happens (in the gate
> and/or in production)?
> 

We've disabled it in a scale lab and didn't observe any black holes forming
or other catastrophes.

> 
> [0] https://review.openstack.org/#/c/4269/
> [1] https://wiki.openstack.org/wiki/ReleaseNotes/Icehouse
> 
> 
> 
> 
> Assaf Muller, Cloud Networking Engineer
> Red Hat
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 



More information about the OpenStack-dev mailing list