[openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg Greg.Waines at windriver.com
Wed Dec 13 13:17:11 UTC 2017


So i believe the error is:
HttpException: HttpException: Expecting to find domain in project.

I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file .

See the domain related attributes in each file below.

Is the Default vs default causing this problem ?
Are there other domain related attributes that should be set to default ?

stack at devstack-masakari:~$ fgrep -i domain /etc/masakari/masakari.conf
project_domain_name = Default
user_domain_name = Default

stack at devstack-masakari:~$ fgrep -i domain /etc/masakarimonitors/masakarimonitors.conf
#logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s
# Domain ID to scope to (string value)
#domain_id = <None>
# Domain name to scope to (string value)
#domain_name = <None>
# Domain ID containing project (string value)
#project_domain_id = <None>
# Domain name containing project (string value)
#project_domain_name = <None>
project_domain_name = default
# Optional domain ID to use with v3 and v2 parameters. It will be used for both
# the user and project domain in v3 and ignored in v2 authentication. (string
#default_domain_id = <None>
# Optional domain name to use with v3 API and v2 parameters. It will be used for
# both the user and project domain in v3 and ignored in v2 authentication.
#default_domain_name = <None>
# User's domain id (string value)
#user_domain_id = <None>
# User's domain name (string value)
#user_domain_name = <None>
# Indicate whether this resource may be shared with the domain received in the
stack at devstack-masakari:~$


Greg.


From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Tuesday, December 12, 2017 at 7:19 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ?
Greg.

From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 8:05 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.
Any ideas ?
Greg.

i.e.
stack at devstack-masakari:~/devstack$ nova list


+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
| ID                                   | Name        | Status | Task State | Power State | Networks                                                |
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | -          | Running     | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
stack at devstack-masakari:~/devstack$
stack at devstack-masakari:~/devstack$ nova delete vm-1-cirros
Request to delete server vm-1-cirros has been accepted.
stack at devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)
2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}
2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.
... several times ... and then eventually
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return res.create(self._session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
stack at devstack-masakari:~/devstack$


Greg.



From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 7:52 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.

stack at devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &
[1] 25973
stack at devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor

stack at devstack-masakari:~/masakari-monitors$

I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?

All processes seem to be up and running fine:
stack at devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari
stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug
stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379
root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor
root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor
stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari
rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit at devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit at devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit at devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit at devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
stack at devstack-masakari:~/masakari-monitors$


Greg.


From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 7:38 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.


stack at devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack at devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]



[1]+  Exit 1                  sudo masakari-instancemonitor

Greg.



From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 7:17 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

thanks Honjo,

I’ll try this out now.

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.
Is that correct ?

In that case, following the instructions you pointed me at,
I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·

·         ( don’t need corosync and pacemaker )

·

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o

o   sudo masakari-instancemonitor &

Is this correct ?

Greg.




From: Rikimaru Honjo <honjo.rikimaru at po.ntt-tx.co.jp>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

Hello Greg,

I forgot to tell you.

Please use process_list.yaml instead of proc.list.sample.

On 2017/12/07 14:03, Rikimaru Honjo wrote:
Hello Greg,
Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.
You can generate it by "tox -egenconfig".
hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.
masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.
And, we are preparing setting guides.
Please see it if you are good.
masakari:
https://review.openstack.org/#/c/489570/
masakari-monitors:
https://review.openstack.org/#/c/489095/
Best regards,
On 2017/12/06 22:48, Waines, Greg wrote:
I am just getting started working with masakari.
I am working on master.

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .
Specifically:
# install & startup monitors
cd
git clone https://github.com/openstack/masakari-monitors.git
cd masakari-monitors
sudo python setup.py install
cd
sudo mkdir /etc/masakarimonitors
sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf
sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf
sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list
cd ~/masakari-monitors/masakarimonitors/cmd
sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &
sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &
sudo /usr/bin/python ./instancemonitor.py &

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

Is this the correct way to startup masakari instance monitoring ?

Greg.




My Masakari setup in Devstack

sudo useradd -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
sudo su - stack

git clone https://github.com/openstack-dev/devstack
cd devstack

                                                 local.conf file:
[[local|localrc]]
ADMIN_PASSWORD=admin
DATABASE_PASSWORD=admin
RABBIT_PASSWORD=admin
SERVICE_PASSWORD=admin
# setup Neutron services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
# ceilometer
enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer
enable_plugin aodh https://git.openstack.org/openstack/aodh
# heat
enable_plugin heat https://git.openstack.org/openstack/heat
# vitrage
enable_plugin vitrage https://git.openstack.org/openstack/vitrage
enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard
# masakari
enable_plugin masakari git://git.openstack.org/openstack/masakari


./stack.sh





__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Rikimaru Honjo
E-mail:honjo.rikimaru at po.ntt-tx.co.jp<mailto:honjo.rikimaru at po.ntt-tx.co.jp>


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171213/b594f143/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: masakarimonitors.conf
Type: application/octet-stream
Size: 13854 bytes
Desc: masakarimonitors.conf
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171213/b594f143/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: masakari.conf
Type: application/octet-stream
Size: 13694 bytes
Desc: masakari.conf
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171213/b594f143/attachment-0001.obj>


More information about the OpenStack-dev mailing list