Open Stack

I have stable/metaka Octavia which has been running OK until today, whenever I created loadbalancer, the amphorae vm is created with mgmt nic. But look like vip plugin failed.  I can ping to amphorae mgmt. NIC from controller(where Octavia process is running), but look like some rest api call  into amphorae to plug in vip failed :

Ping works:

[localadmin at dmz-eth2-ucs1]logs> ping 192.168.0.7
PING 192.168.0.7 (192.168.0.7) 56(84) bytes of data.
64 bytes from 192.168.0.7: icmp_seq=1 ttl=64 time=1.11 ms
64 bytes from 192.168.0.7: icmp_seq=2 ttl=64 time=0.461 ms
^C


o-cw.log:

2016-12-09 11:03:54.468 31408 DEBUG octavia.controller.worker.tasks.network_tasks [-] Retrieving network details for amphora ae80ae54-395f-4fad-b0de-39f17dd9b19e execute /opt/stack/octavia/octavia/controller/worker/tasks/network_tasks.py:380
2016-12-09 11:03:55.441 31408 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs' (76823522-b504-4d6a-8ba7-c56015cb39a9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{u'ae80ae54-395f-4fad-b0de-39f17dd9b19e': <octavia.network.data_models.AmphoraNetworkConfig object at 0x7faebb7e99d0>}' _task_receiver /usr/local/lib/python2.7/dist-packages/taskflow/listeners/logging.py:178
2016-12-09 11:03:55.444 31408 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraPostVIPPlug' (3b798537-3f20-46a3-abe2-a2c24c569cd9) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/local/lib/python2.7/dist-packages/taskflow/listeners/logging.py:189
2016-12-09 11:03:55.446 31408 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url plug/vip/100.100.100.9 request /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:218
2016-12-09 11:03:55.446 31408 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url https://192.168.0.7:9443/0.5/plug/vip/100.100.100.9 request /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:221
2016-12-09 11:03:55.452 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.
2016-12-09 11:03:56.458 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.
2016-12-09 11:03:57.462 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.
2016-12-09 11:03:58.466 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.
2016-12-09 11:03:59.470 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.
2016-12-09 11:04:00.474 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.
2016-12-09 11:04:02.487 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.
……
ransitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
2016-12-09 11:29:10.509 31408 WARNING octavia.controller.worker.controller_worker [-] Flow 'post-amphora-association-octavia-post-loadbalancer-amp_association-subflow' (f7b0d080-830a-4d6a-bb85-919b6461252f) transitioned into state 'REVERTED' from state 'RUNNING'
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: contacting the amphora timed out
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/octavia/octavia/controller/queue/endpoint.py", line 45, in create_load_balancer
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     self.worker.create_load_balancer(load_balancer_id)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/octavia/octavia/controller/worker/controller_worker.py", line 322, in create_load_balancer
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     post_lb_amp_assoc.run()
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 230, in run
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     for _state in self.run_iter(timeout=timeout):
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 308, in run_iter
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     failure.Failure.reraise_if_any(fails)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     failures[0].reraise()
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 343, in reraise
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     six.reraise(*self._exc_info)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     result = task.execute(**arguments)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 229, in execute
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     loadbalancer, amphorae_network_config)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 131, in post_vip_plug
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     net_info)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 325, in plug_vip
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     json=net_info)
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 246, in request
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher     raise driver_except.TimeOutException()
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher TimeOutException: contacting the amphora timed out
2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher

Can somebody advise what went wrong?  Or suggest way to debug this, I ssh into the vm and can see that amphorae agent is running inside the vm and listening on the API port 9443, but I don’t know where the log is

Thanks
Wanjing



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161209/e23c5170/attachment.html>