Open Stack

On 07/07/2017 12:30 PM, Peter Doherty wrote:
> Hi,
> 
> If I'm interpreting this correctly, nova compute is calling 
> get_diagnostics on all instances, including ones currently in a shutdown 
> state.  And then it throws an exception, and adds an entry into the 
> instance_faults table in the database.
> 
> nova-compute logs this message:
> 
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher 
> Traceback (most recent call last):
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", 
> line 142, in _dispatch_and_reply
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> executor_callback))
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", 
> line 186, in _dispatch
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> executor_callback)
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", 
> line 129, in _do_dispatch
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> result = func(ctxt, **new_args)
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> payload)
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in 
> __exit__
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> six.reraise(self.type_, self.value, self.tb)
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> return f(self, context, *args, **kw)
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in 
> decorated_function
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> kwargs['instance'], e, sys.exc_info())
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in 
> __exit__
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> six.reraise(self.type_, self.value, self.tb)
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in 
> decorated_function
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> return function(self, context, *args, **kwargs)
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4089, 
> in get_diagnostics
> 2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher     
> method='get_diagnostics')
> 
> 2017-07-07 16:30:10.017 23077 ERROR oslo_messaging.rpc.dispatcher 
> InstanceInvalidState: Instance 6ab60005-ccbf-4bc2-95ac-7daf31716754 in 
> power_state 4. Cannot get_diagnostics while the instance is in this state.
> 
> I don't think it should be trying to gather diags on shutdown instances, 
> and if it did, it shouldn't just create a never-ending stream of errors.
> If anyone has any info on if this might be a bug that is fixed in the 
> latest release, or if I can turn off this behavior, it would be appreciated.

get_diagnostics() doesn't run automatically. Something is triggering a 
call to get_diagnostics() for each instance on the box (the internal 
compute manager only has a get_diagnostics(instance) call that takes one 
instance at a time). Not sure what is triggering that...

I agree with you that ERRORs shouldn't be spewed into the nova-compute 
logs like the above, though. That should be fixed. Would you mind 
submitting a bug for that on Launchpad, Peter?

Thank you!
-jay