[openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Waines, Greg
Greg.Waines at windriver.com
Tue Dec 12 12:19:59 UTC 2017
Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ?
Greg.
From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 8:05 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.
Any ideas ?
Greg.
i.e.
stack at devstack-masakari:~/devstack$ nova list
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | - | Running | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |
+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+
stack at devstack-masakari:~/devstack$
stack at devstack-masakari:~/devstack$ nova delete vm-1-cirros
Request to delete server vm-1-cirros has been accepted.
stack at devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)
2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}
2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.
... several times ... and then eventually
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari payload=event['notification']['payload'])
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self._create(_notification.Notification, **attrs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return res.create(self._session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari json=request.body, headers=request.headers)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self.request(url, 'POST', **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return func(*args, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return super(Session, self).request(*args, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari auth_headers = self.get_auth_headers(auth)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return auth.get_headers(self, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari token = self.get_token(session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self.get_access(session).auth_token
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari self.auth_ref = self.get_auth_ref(session)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self._plugin.get_auth_ref(session, **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari authenticated=False, log=False, **rkwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari return self.request(url, 'POST', **kwargs)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari raise exceptions.from_exception(e)
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.
2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari
stack at devstack-masakari:~/devstack$
Greg.
From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 7:52 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.
stack at devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &
[1] 25973
stack at devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor
stack at devstack-masakari:~/masakari-monitors$
I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?
All processes seem to be up and running fine:
stack at devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari
stack 11625 1 0 12:29 ? 00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
stack 11778 1 0 12:29 ? 00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug
stack 12005 11625 0 12:29 ? 00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
stack 12006 11625 0 12:29 ? 00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug
root 19336 1 0 12:19 ? 00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379
root 25973 25760 0 12:47 pts/0 00:00:00 sudo masakari-instancemonitor
root 25974 25973 0 12:47 pts/0 00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor
stack 26123 25760 0 12:50 pts/0 00:00:00 grep -F --color=auto masakari
rabbitmq 26477 26348 0 12:06 ? 00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit at devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit at devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit at devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit at devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
stack at devstack-masakari:~/masakari-monitors$
Greg.
From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 7:38 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.
stack at devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &
[1] 22959
stack at devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):
File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main
config.parse_args(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args
default_config_files=default_config_files)
File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__
self._check_required_opts()
File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts
raise RequiredOptError(opt.name, group)
oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]
[1]+ Exit 1 sudo masakari-instancemonitor
Greg.
From: Greg Waines <Greg.Waines at windriver.com>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Monday, December 11, 2017 at 7:17 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
thanks Honjo,
I’ll try this out now.
I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.
Is that correct ?
In that case, following the instructions you pointed me at,
I would do the following:
· setup devstack
o i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’
·
· ( don’t need corosync and pacemaker )
·
· install & startup client
o cd
o git clone https://github.com/openstack/python-masakariclient.git
o cd python-masakariclient
o sudo python setup.py build
o sudo python setup.py install
·
· install & startup instance monitor
o cd
o git clone https://github.com/openstack/masakari-monitors.git
o sudo mkdir /etc/masakarimonitors
o cd masakari-monitors
o sudo python setup.py build
o sudo python setup.py install
o
o sudo masakari-instancemonitor &
Is this correct ?
Greg.
From: Rikimaru Honjo <honjo.rikimaru at po.ntt-tx.co.jp>
Reply-To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
Hello Greg,
I forgot to tell you.
Please use process_list.yaml instead of proc.list.sample.
On 2017/12/07 14:03, Rikimaru Honjo wrote:
Hello Greg,
Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.
You can generate it by "tox -egenconfig".
hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.
masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.
And, we are preparing setting guides.
Please see it if you are good.
masakari:
https://review.openstack.org/#/c/489570/
masakari-monitors:
https://review.openstack.org/#/c/489095/
Best regards,
On 2017/12/06 22:48, Waines, Greg wrote:
I am just getting started working with masakari.
I am working on master.
I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.
I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .
Specifically:
# install & startup monitors
cd
git clone https://github.com/openstack/masakari-monitors.git
cd masakari-monitors
sudo python setup.py install
cd
sudo mkdir /etc/masakarimonitors
sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf
sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf
sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list
cd ~/masakari-monitors/masakarimonitors/cmd
sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &
sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &
sudo /usr/bin/python ./instancemonitor.py &
However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.
Is this the correct way to startup masakari instance monitoring ?
Greg.
My Masakari setup in Devstack
sudo useradd -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
sudo su - stack
git clone https://github.com/openstack-dev/devstack
cd devstack
local.conf file:
[[local|localrc]]
ADMIN_PASSWORD=admin
DATABASE_PASSWORD=admin
RABBIT_PASSWORD=admin
SERVICE_PASSWORD=admin
# setup Neutron services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
# ceilometer
enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer
enable_plugin aodh https://git.openstack.org/openstack/aodh
# heat
enable_plugin heat https://git.openstack.org/openstack/heat
# vitrage
enable_plugin vitrage https://git.openstack.org/openstack/vitrage
enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard
# masakari
enable_plugin masakari git://git.openstack.org/openstack/masakari
./stack.sh
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Rikimaru Honjo
E-mail:honjo.rikimaru at po.ntt-tx.co.jp<mailto:honjo.rikimaru at po.ntt-tx.co.jp>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171212/6a719a2c/attachment.html>
More information about the OpenStack-dev
mailing list