[openstack-dev] [masakari] BUG in Masakari Installation and Procedure and/or Documentation

Waines, Greg Greg.Waines at windriver.com
Fri Jan 26 20:59:10 UTC 2018 

Update on this.

It turned out that i had incorrectly set the ‘project_name’ and ‘username’ in  /etc/masakarimonitors/masakarimonitors.conf

Setting both these attributes to ‘admin’ made it such that the instancemonitor’s notification to masakari-engine was successful.
e.g.
stack at devstack-masakari-louie:~/devstack$ masakari notification-list
+--------------------------------------+----------------------------+---------+--------------------------------------+------+
| notification_uuid                    | generated_time             | status  | source_host_uuid                     | type |
+--------------------------------------+----------------------------+---------+--------------------------------------+------+
| b8c6c561-7a93-40a2-8d73-3783024865b4 | 2018-01-26T19:41:29.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM   |
+--------------------------------------+----------------------------+---------+--------------------------------------+------+
stack at devstack-masakari-louie:~/devstack$


However I now get the following error in masakari-engine, when the masakari-engine attempts to do the VM Recovery

Jan 26 19:41:28 devstack-masakari-louie masakari-engine[11795]: 2018-01-26 19:41:28.968 TRACE masakari.engine.drivers.taskflow.driver EndpointNotFound: publicURL endpoint for compute service named Compute Service not found


Why is masakari-engine looking for a publicURL endpoint for service_type=’compute’ and service_name=’Compute Service’ ?
See below that the Service Name = ‘nova’ ... NOT ‘Compute Service’

stack at devstack-masakari-louie:~/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+----------------------------------------------+
| ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                          |
+----------------------------------+-----------+--------------+----------------+---------+-----------+----------------------------------------------+
| 0111643ef1584decb523524a3db5ce18 | RegionOne | nova_legacy  | compute_legacy | True    | public    | http://10.10.10.14/compute/v2/$(project_id)s |
| 01790448c22f49e69774adf290fba728 | RegionOne | gnocchi      | metric         | True    | internal  | http://10.10.10.14/metric                    |
| 0b31693c6650499a981d580721be9e48 | RegionOne | vitrage      | rca            | True    | internal  | http://10.10.10.14:8999                      |
| 40f66ed61b4e4310829aa69e11c75554 | RegionOne | neutron      | network        | True    | public    | http://10.10.10.14:9696/                     |
| 47479cf64af944b996b1fbca42efd945 | RegionOne | nova         | compute        | True    | public    | http://10.10.10.14/compute/v2.1              |
| 49dccfc61e8246a2a2c0b8d12b3db91a | RegionOne | vitrage      | rca            | True    | admin     | http://10.10.10.14:8999                      |
| 5261ba0327de4c2d92842147636ee770 | RegionOne | masakari     | ha             | True    | internal  | http://10.10.10.14:15868/v1/$(tenant_id)s    |
| 5df28622c6f449ebad12d9b62110cd08 | RegionOne | gnocchi      | metric         | True    | admin     | http://10.10.10.14/metric                    |
| 64f8f401431042a0ab1d053ca4f4df02 | RegionOne | glance       | image          | True    | public    | http://10.10.10.14/image                     |
| 69ad6b9d0b0b4d0a8da6fa36af8289cb | RegionOne | masakari     | ha             | True    | public    | http://10.10.10.14:15868/v1/$(tenant_id)s    |
| 7dd9d5396e9c49d4a41e2865b841f6a0 | RegionOne | masakari     | ha             | True    | admin     | http://10.10.10.14:15868/v1/$(tenant_id)s    |
| 811fa7f4b3c14612b4aca354dc8ea77e | RegionOne | vitrage      | rca            | True    | public    | http://10.10.10.14:8999                      |
| 8535da724c424363bffe1d033ee033e5 | RegionOne | cinder       | volume         | True    | public    | http://10.10.10.14/volume/v1/$(project_id)s  |
| 853f1783f1014075a03c16f7c3a2568a | RegionOne | keystone     | identity       | True    | admin     | http://10.10.10.14/identity                  |
| 9450f5611ca747f2a049f22ff0996dba | RegionOne | cinderv3     | volumev3       | True    | public    | http://10.10.10.14/volume/v3/$(project_id)s  |
| 9a73696d88a9438cb0ab75a754a08e9d | RegionOne | gnocchi      | metric         | True    | public    | http://10.10.10.14/metric                    |
| b1ff2b4d683c4a58a3b27232699d0058 | RegionOne | cinderv2     | volumev2       | True    | public    | http://10.10.10.14/volume/v2/$(project_id)s  |
| d4e66240faff48f2b5e1d0fcfb73a74b | RegionOne | placement    | placement      | True    | public    | http://10.10.10.14/placement                 |
| fda917fd368a4a479c9c186df1beb8e9 | RegionOne | keystone     | identity       | True    | public    | http://10.10.10.14/identity                  |
+----------------------------------+-----------+--------------+----------------+---------+-----------+----------------------------------------------+
stack at devstack-masakari-louie:~/devstack$

let me know your thoughts,
I don’t mind raising the required BUG in Launchpad if required,
Greg.

p.s. my masakari configurations ... wrt hosts and segments ... are as follows:
stack at devstack-masakari-louie:~/devstack$ masakari segment-list
+--------------------------------------+-----------+-------------+--------------+-----------------+
| uuid                                 | name      | description | service_type | recovery_method |
+--------------------------------------+-----------+-------------+--------------+-----------------+
| 9c6e22bd-7fab-40cb-a8e0-3702137f3227 | segment-1 | -           | COMPUTE      | auto            |
+--------------------------------------+-----------+-------------+--------------+-----------------+
stack at devstack-masakari-louie:~/devstack$ masakari host-list --segment-id 9c6e22bd-7fab-40cb-a8e0-3702137f3227
+--------------------------------------+-------------------------+---------+--------------------+----------+----------------+--------------------------------------+
| uuid                                 | name                    | type    | control_attributes | reserved | on_maintenance | failover_segment_id                  |
+--------------------------------------+-------------------------+---------+--------------------+----------+----------------+--------------------------------------+
| 51bc8b8b-324f-499a-9166-38c22b3842cd | devstack-masakari-louie | COMPUTE | SSH                | False    | False          | 9c6e22bd-7fab-40cb-a8e0-3702137f3227 |
+--------------------------------------+-------------------------+---------+--------------------+----------+----------------+--------------------------------------+
stack at devstack-masakari-louie:~/devstack$





From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Wednesday, January 24, 2018 at 4:13 PM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: [openstack-dev] [masakari] BUG in Masakari Installation and Procedure and/or Documentation

I am looking for some input before I raise a BUG.

I reviewed the following commits which documented the Masakari and MasakariMonitors Installation and Procedures.
i.e.
https://review.openstack.org/#/c/489570/
https://review.openstack.org/#/c/489095/

I created an AIO devstack with Masakari on current/master ... this morning.
I followed the above instructions on configuring and installing Masakari and MasakariMonitors.

I created a VM and then ‘sudo kill -9 <qemu-pid-of-vm>’
and
I got the following error from instance monitoring trying to send the notification message to masakari-engine.
( The request you have made requires authentication. ) ... see below,

Is this a known BUG ?
Greg.


2018-01-24 20:29:16.902 12473 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari-new, uuid=6884cf13-5797-487b-9cb1-053a2e18b60e, time=2018-01-24 20:29:16.902347, event_id=LIFECYCLE, detail=STOPPED_FAILED)

2018-01-24 20:29:16.903 12473 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari-new', 'type': 'VM', 'payload': {'instance_uuid': '6884cf13-5797-487b-9cb1-053a2e18b60e', 'vir_domain_event': 'STOPPED_FAILED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2018, 1, 24, 20, 29, 16, 902347)}}

2018-01-24 20:29:16.977 12473 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9c734f56-aca9-40a9-b2dd-3f372de8c34e), The request you have made requires authentication.): HttpException: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9c734f56-aca9-40a9-b2dd-3f372de8c34e), The request you have made requires authentication.

...

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari [-] Exception caught: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-26a5de94-aaad-4f8f-949e-cbfeb5e31b8b), The request you have made requires authentication.: HttpException: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-26a5de94-aaad-4f8f-949e-cbfeb5e31b8b), The request you have made requires authentication.

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return res.create(self._session)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari HttpException: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-26a5de94-aaad-4f8f-949e-cbfeb5e31b8b), The request you have made requires authentication.

2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180126/4f623e7d/attachment-0001.html>

More information about the OpenStack-dev mailing list