<div dir="ltr"><div><div><div>Thanks. I wrongfully assumed it was being run automatically, so with that out of my mind, it didn't take too long to figure out what was triggering that. I'm running the Datadog agent, which is the source. It generated enough noise in a week I ended up with a million rows in the nova.instance_fault table, and the memory footprint of nova-api got very large, all of which resulted in multi-minute responses to instance list queries.<br><br></div>I can open a bug report about the log messages. I think it may be in the nova/compute/manager.py code, which doesn't seem to gracefully know what to do if get_diagnostics is called on a instance that isn't running, and results in a lot of useless rows in the instance_fault table.<br><font face="monospace,monospace"> <br> @wrap_instance_fault<br> def get_diagnostics(self, context, instance):<br> """Retrieve diagnostics for an instance on this host."""<br> current_power_state = self._get_power_state(context, instance)<br> if current_power_state == power_state.RUNNING:<br> LOG.info(_LI("Retrieving diagnostics"), context=context,<br> instance=instance)<br> return self.driver.get_diagnostics(instance)<br> else:<br> raise exception.InstanceInvalidState(<br> attr='power_state',<br> instance_uuid=instance.uuid,<br> state=instance.power_state,<br> method='get_diagnostics')<br><br><br></font></div><font face="monospace,monospace">Thanks Jay!<br><br></font></div><font face="monospace,monospace">-Peter<br></font></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 7, 2017 at 12:50 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 07/07/2017 12:30 PM, Peter Doherty wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
If I'm interpreting this correctly, nova compute is calling get_diagnostics on all instances, including ones currently in a shutdown state. And then it throws an exception, and adds an entry into the instance_faults table in the database.<br>
<br>
nova-compute logs this message:<br>
<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/oslo_messaging/rpc/dispatc<wbr>her.py", line 142, in _dispatch_and_reply<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher executor_callback))<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/oslo_messaging/rpc/dispatc<wbr>her.py", line 186, in _dispatch<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher executor_callback)<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/oslo_messaging/rpc/dispatc<wbr>her.py", line 129, in _do_dispatch<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/nova/exception.py", line 89, in wrapped<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher payload)<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/oslo_utils/excutils.py", line 195, in __exit__<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/nova/exception.py", line 72, in wrapped<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/nova/compute/manager.py", line 378, in decorated_function<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/oslo_utils/excutils.py", line 195, in __exit__<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/nova/compute/manager.py", line 366, in decorated_function<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packa<wbr>ges/nova/compute/manager.py", line 4089, in get_diagnostics<br>
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher method='get_diagnostics')<br>
<br>
2017-07-07 16:30:10.017 23077 ERROR oslo_messaging.rpc.dispatcher InstanceInvalidState: Instance 6ab60005-ccbf-4bc2-95ac-7daf31<wbr>716754 in power_state 4. Cannot get_diagnostics while the instance is in this state.<br>
<br>
I don't think it should be trying to gather diags on shutdown instances, and if it did, it shouldn't just create a never-ending stream of errors.<br>
If anyone has any info on if this might be a bug that is fixed in the latest release, or if I can turn off this behavior, it would be appreciated.<br>
</blockquote>
<br></div></div>
get_diagnostics() doesn't run automatically. Something is triggering a call to get_diagnostics() for each instance on the box (the internal compute manager only has a get_diagnostics(instance) call that takes one instance at a time). Not sure what is triggering that...<br>
<br>
I agree with you that ERRORs shouldn't be spewed into the nova-compute logs like the above, though. That should be fixed. Would you mind submitting a bug for that on Launchpad, Peter?<br>
<br>
Thank you!<br>
-jay<br>
<br>
______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Peter Doherty<br>Systems Engineer, Systems Engineering<br>Brightcove Inc.<br>290 Congress St., 4th Floor, Boston, MA 02210</div>
</div>