Open Stack

Good for now.

The issue should be related to some situations that VM stop instance task is taking longer and it seems one of the periodic task is timing out.

To avoid the exception, may  try to increase some of the timeout values. Or increase the looping interval for retry…

Thanks.
LK


From: Kwan, Louie
Sent: Tuesday, February 20, 2018 5:17 PM
To: 'OpenStack Development Mailing List (not for usage questions)'
Subject: [openstack-dev] [masakari] [masakari-monitors] : Masakari notification failed.

Hi Masakari community,

I would like to get your help  to understand what may be causing the Masakari notification failed.  I do get  success cases which the engine got the notification, VM got shutdown and rebooted ok.

Having said that, there are some cases that the notification failed and it seems there are some conflicts going on. 20% to 40% chance.

Feb 20 21:53:21 masakari-2 masakari-engine[3807]: 2018-02-20 21:53:21.517 WARNING masakari.engine.drivers.taskflow.driver [req-ce909151-1afb-4f2f-abf4-f25d54f25c6b service None] Task 'masakari.engine.drivers.taskflow.instance_failure.StopInstanceTask;instance:recovery' (e85dec06-1498-482c-a63a-51f855745c32) transitioned into state 'FAILURE' from state 'RUNNING'
Feb 20 21:53:21 masakari-2 masakari-engine[3807]: 1 predecessors (most recent first):
Feb 20 21:53:21 masakari-2 masakari-engine[3807]:   Flow 'instance_recovery_engine': Conflict: Conflict

Is it normal that masakari notification would be failed because of timing or conflicting events? FYI, I only have one VM and one active notification.

Enclosed is the log file I got from the engine.

I do appreciate if anyone of you can provide some insight what to do with the failure. Any tip where to look at etc? Timeout?

Thanks.
Louie


| notification_uuid                    | generated_time             | status   | type | source_host_uuid                     | payload                                                                                                                                  |
+--------------------------------------+----------------------------+----------+------+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| 42ccee84-0ea5-4163-84a5-028a0bb914a3 | 2018-02-20T21:52:03.000000 | failed   | VM   | 66c8b5b9-03f5-4843-8a9c-fa83af807a9b | {u'instance_uuid': u'565da9ba-3c0c-4087-83ca-32a5a1b00a55', u'vir_domain_event': u'STOPPED_FAILED', u'event': u'QEMU_GUEST_AGENT_ERROR'} |
| aa4184f3-b002-4ba8-a403-f22ccd4ce6b5 | 2018-02-20T21:42:54.000000 | finished | VM   | 66c8b5b9-03f5-4843-8a9c-fa83af807a9b | {u'instance_uuid': u'565da9ba-3c0c-4087-83ca-32a5a1b00a55', u'vir_domain_event': u'STOPPED_FAILED', u'event': u'QEMU_GUEST_AGENT_ERROR'} |

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180222/c2af2269/attachment.html>