<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1970932613;
mso-list-template-ids:736281278;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I have stable/metaka Octavia which has been running OK until today, whenever I created loadbalancer, the amphorae vm is created with mgmt nic. But look like vip plugin failed. I can ping
to amphorae mgmt. NIC from controller(where Octavia process is running), but look like some rest api call into amphorae to plug in vip failed :<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Ping works:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">[localadmin@dmz-eth2-ucs1]logs> ping 192.168.0.7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">PING 192.168.0.7 (192.168.0.7) 56(84) bytes of data.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">64 bytes from 192.168.0.7: icmp_seq=1 ttl=64 time=1.11 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">64 bytes from 192.168.0.7: icmp_seq=2 ttl=64 time=0.461 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">^C<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">o-cw.log:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:54.468 31408 DEBUG octavia.controller.worker.tasks.network_tasks [-] Retrieving network details for amphora ae80ae54-395f-4fad-b0de-39f17dd9b19e execute /opt/stack/octavia/octavia/controller/worker/tasks/network_tasks.py:380<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:55.441 31408 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.GetAmphoraeNetworkConfigs' (76823522-b504-4d6a-8ba7-c56015cb39a9)
transitioned into state 'SUCCESS' from state 'RUNNING' with result '{u'ae80ae54-395f-4fad-b0de-39f17dd9b19e': <octavia.network.data_models.AmphoraNetworkConfig object at 0x7faebb7e99d0>}' _task_receiver /usr/local/lib/python2.7/dist-packages/taskflow/listeners/logging.py:178<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:55.444 31408 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraPostVIPPlug' (3b798537-3f20-46a3-abe2-a2c24c569cd9)
transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/local/lib/python2.7/dist-packages/taskflow/listeners/logging.py:189<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:55.446 31408 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url plug/vip/100.100.100.9 request /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:218<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:55.446 31408 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url https://192.168.0.7:9443/0.5/plug/vip/100.100.100.9 request /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:221<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:55.452 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:56.458 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:57.462 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:58.466 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:03:59.470 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:04:00.474 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:04:02.487 31408 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">……<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">ransitioned into state 'REVERTED' from state 'REVERTING' with result 'None'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 WARNING octavia.controller.worker.controller_worker [-] Flow 'post-amphora-association-octavia-post-loadbalancer-amp_association-subflow' (f7b0d080-830a-4d6a-bb85-919b6461252f)
transitioned into state 'REVERTED' from state 'RUNNING'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: contacting the amphora timed out<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher incoming.message))<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/controller/queue/endpoint.py", line 45, in create_load_balancer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher self.worker.create_load_balancer(load_balancer_id)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/controller/worker/controller_worker.py", line 322, in create_load_balancer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher post_lb_amp_assoc.run()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 230, in run<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher for _state in self.run_iter(timeout=timeout):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 308, in run_iter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(fails)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 336, in reraise_if_any<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher failures[0].reraise()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 343, in reraise<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher result = task.execute(**arguments)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 229, in execute<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher loadbalancer, amphorae_network_config)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 131, in post_vip_plug<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher net_info)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 325, in plug_vip<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher json=net_info)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 246, in request<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher raise driver_except.TimeOutException()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher TimeOutException: contacting the amphora timed out<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2016-12-09 11:29:10.509 31408 ERROR oslo_messaging.rpc.dispatcher
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Can somebody advise what went wrong? Or suggest way to debug this, I ssh into the vm and can see that amphorae agent is running inside the vm and listening on the API port 9443, but I
don’t know where the log is <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Wanjing<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>