<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Dear colleagues,<br>
<br>
trying to launch Amphorae, getting the following error in logs:<br>
<br>
Amphora agent returned unexpected result code 500 with response
{'message': 'Error plugging VIP', 'details': 'SIOCADDRT: Network is
unreachable\nFailed to bring up eth1.\n'}<br>
<br>
While details below, questions are here:<br>
- whether it's enough to assign roles as explained below to special
project for Octavia?<br>
- whether it can be issue with image, created by
diskimage_create.sh?<br>
- any recommendation on where to search for the problem.<br>
<br>
Thank you.<br>
<br>
My environment is:<br>
- Openstack Rocky<br>
- Octavia 4.0<br>
- amphora instance runs in special project "octavia", where users
octavia, nova and neutron have admin role<br>
- amphora image prepared using original git repo process and
elements without modification:<br>
* git clone <octavia><br>
* cd octavia<br>
* diskimage-create/diskimage-create.sh<br>
* openstack image create [ ... ] --tag amphora<br>
<br>
After created, amphora instance successfully connects to management
network and can be accessed by controller:<br>
<br>
2019-05-05 20:46:06.851 18234 DEBUG
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connected to
amphora. Response: <Response [200]> request
/opt/openstack/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:486<br>
2019-05-05 20:46:06.852 18234 DEBUG
octavia.controller.worker.tasks.amphora_driver_tasks [-] Successfuly
connected to amphora 5bec4c09-a209-4e73-a66e-e4fc0fb8ded5:
{'ipvsadm_version': '1:1.28-3', 'api_version': '0.5',
'haproxy_version': '1.6.3-1ubuntu0.2', 'hostname':
'amphora-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5',
'keepalived_version': '1:1.2.24-1ubuntu0.16.04.1'} execute
/opt/openstack/lib/python3.6/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py:372<br>
[ ... ]<br>
2019-05-05 20:46:06.990 18234 DEBUG
octavia.controller.worker.tasks.network_tasks [-] Plumbing VIP for
amphora id: 5bec4c09-a209-4e73-a66e-e4fc0fb8ded5 execute
/opt/openstack/lib/python3.6/site-packages/octavia/controller/worker/tasks/network_tasks.py:382<br>
2019-05-05 20:46:07.003 18234 DEBUG
octavia.network.drivers.neutron.base [-] Neutron extension
security-group found enabled _check_extension_enabled
/opt/openstack/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66<br>
2019-05-05 20:46:07.013 18234 DEBUG
octavia.network.drivers.neutron.base [-] Neutron extension
dns-integration found enabled _check_extension_enabled
/opt/openstack/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66<br>
2019-05-05 20:46:07.025 18234 DEBUG
octavia.network.drivers.neutron.base [-] Neutron extension qos found
enabled _check_extension_enabled
/opt/openstack/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66<br>
2019-05-05 20:46:07.044 18234 DEBUG
octavia.network.drivers.neutron.base [-] Neutron extension
allowed-address-pairs found enabled _check_extension_enabled
/opt/openstack/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66<br>
2019-05-05 20:46:08.406 18234 DEBUG
octavia.network.drivers.neutron.allowed_address_pairs [-] Created
vip port: b0398cc8-6d52-4f12-9f1f-1141b0f10751 for amphora:
5bec4c09-a209-4e73-a66e-e4fc0fb8ded5 _plug_amphora_vip
/opt/openstack/lib/python3.6/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py:97<br>
[ ... ]<br>
2019-05-05 20:46:15.405 18234 DEBUG
octavia.network.drivers.neutron.allowed_address_pairs [-] Retrieving
network details for amphora 5bec4c09-a209-4e73-a66e-e4fc0fb8ded5
_get_amp_net_configs
/opt/openstack/lib/python3.6/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py:596<br>
[ ... ]<br>
2019-05-05 20:46:15.837 18234 DEBUG
octavia.amphorae.drivers.haproxy.rest_api_driver [-]
Post-VIP-Plugging with vrrp_ip 10.0.2.13 vrrp_port
b0398cc8-6d52-4f12-9f1f-1141b0f10751 post_vip_plug
/opt/openstack/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:233<br>
2019-05-05 20:46:15.838 18234 DEBUG
octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url
plug/vip/10.0.2.24 request
/opt/openstack/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:462<br>
2019-05-05 20:46:15.838 18234 DEBUG
octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url
<a class="moz-txt-link-freetext" href="https://172.16.252.35:9443/0.5/plug/vip/10.0.2.24">https://172.16.252.35:9443/0.5/plug/vip/10.0.2.24</a> request
/opt/openstack/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:465<br>
2019-05-05 20:46:16.089 18234 DEBUG
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connected to
amphora. Response: <Response [500]> request
/opt/openstack/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:486<br>
2019-05-05 20:46:16.090 18234 ERROR
octavia.amphorae.drivers.haproxy.exceptions [-] Amphora agent
returned unexpected result code 500 with response {'message': 'Error
plugging VIP', 'details': 'SIOCADDRT: Network is unreachable\nFailed
to bring up eth1.\n'}<br>
<br>
During the process, NEUTRON logs contains the following records that
indicate the following (note "status=DOWN" in neutron-dhcp-agent;
later immediately before to be deleted, it will shed 'ACTIVE'):<br>
<br>
May 5 20:46:13 ardbeg neutron-dhcp-agent: 2019-05-05 20:46:13.857
1804 INFO neutron.agent.dhcp.agent
[req-07833602-9579-403b-a264-76fd3ee408ee
a18f38c780074c6280dde5edad159666 41a02a69918849509f4102b04f8a7de9 -
- -] Trigger reload_allocations for port admin_state_up=True,
allowed_address_pairs=[{u'ip_address': u'10.0.2.24', u'mac_address':
u'72:d0:1c:4c:94:91'}], binding:host_id=ardbeg, binding:profile=,
binding:vif_details=datapath_type=system, ovs_hybrid_plug=False,
port_filter=True, binding:vif_type=ovs, binding:vnic_type=normal,
created_at=2019-05-05T20:46:07Z, description=,
device_id=f1bce6e9-be5b-464b-8f64-686f36e9de1f,
device_owner=compute:nova, dns_assignment=[{u'hostname':
u'amphora-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5', u'ip_address':
u'10.0.2.13', u'fqdn':
u'amphora-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5.loqal.'}],
dns_domain=, dns_name=amphora-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5,
extra_dhcp_opts=[], fixed_ips=[{u'subnet_id':
u'24b10886-3d53-4aee-bdc6-f165b242ae4f', u'ip_address':
u'10.0.2.13'}], id=b0398cc8-6d52-4f12-9f1f-1141b0f10751,
mac_address=72:d0:1c:4c:94:91,
name=octavia-lb-vrrp-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5,
network_id=b24d2830-eec6-4abd-82f2-ac71c8ecbf40,
port_security_enabled=True,
project_id=41a02a69918849509f4102b04f8a7de9, qos_policy_id=None,
revision_number=5,
security_groups=[u'6df53a15-6afc-4c99-b464-03de4f546b4f'],
status=DOWN, tags=[], tenant_id=41a02a69918849509f4102b04f8a7de9,
updated_at=2019-05-05T20:46:13Z<br>
May 5 20:46:14 ardbeg neutron-openvswitch-agent: 2019-05-05
20:46:14.185 31542 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-a4425cdb-afc1-4f6a-9ef9-c8706e3285d6 - - - - -] Port
b0398cc8-6d52-4f12-9f1f-1141b0f10751 updated. Details: {'profile':
{}, 'network_qos_policy_id': None, 'qos_policy_id': None,
'allowed_address_pairs': [{'ip_address':
AuthenticIPNetwork('10.0.2.24'), 'mac_address':
EUI('72:d0:1c:4c:94:91')}], 'admin_state_up': True, 'network_id':
'b24d2830-eec6-4abd-82f2-ac71c8ecbf40', 'segmentation_id': 437,
'fixed_ips': [{'subnet_id': '24b10886-3d53-4aee-bdc6-f165b242ae4f',
'ip_address': '10.0.2.13'}], 'device_owner': u'compute:nova',
'physical_network': None, 'mac_address': '72:d0:1c:4c:94:91',
'device': u'b0398cc8-6d52-4f12-9f1f-1141b0f10751',
'port_security_enabled': True, 'port_id':
'b0398cc8-6d52-4f12-9f1f-1141b0f10751', 'network_type': u'vxlan',
'security_groups': [u'6df53a15-6afc-4c99-b464-03de4f546b4f']}<br>
May 5 20:46:14 ardbeg neutron-openvswitch-agent: 2019-05-05
20:46:14.197 31542 INFO neutron.agent.securitygroups_rpc
[req-a4425cdb-afc1-4f6a-9ef9-c8706e3285d6 - - - - -] Preparing
filters for devices set([u'b0398cc8-6d52-4f12-9f1f-1141b0f10751'])<br>
<br>
Note Nova returns response 200/completed:<br>
<br>
May 5 20:46:14 controller-l neutron-server: 2019-05-05 20:46:14.326
20981 INFO neutron.notifiers.nova [-] Nova event response:
{u'status': u'completed', u'tag':
u'b0398cc8-6d52-4f12-9f1f-1141b0f10751', u'name':
u'network-changed', u'server_uuid':
u'f1bce6e9-be5b-464b-8f64-686f36e9de1f', u'code': 200}<br>
<br>
and "openstack server show" shows both NICs are attached to the
amphorae:<br>
<br>
$ openstack server show f1bce6e9-be5b-464b-8f64-686f36e9de1f<br>
<pre>+-------------------------------------+------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------+
[ ... ]
| addresses | octavia-net=172.16.252.35; u1000-p1000-xbone=10.0.2.13 |
+-------------------------------------+------------------------------------------------------------+
</pre>
Later Octavia worker reports the following:<br>
<br>
2019-05-05 20:46:16.124 18234 DEBUG
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-plug-net-subflow-octavia-amp-post-vip-plug'
(f105ced1-72c6-4116-b582-599a21cdee36) transitioned into state
'REVERTING' from state 'FAILURE' _task_receiver
/opt/openstack/lib/python3.6/site-packages/taskflow/listeners/logging.py:194<br>
2019-05-05 20:46:16.127 18234 WARNING
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-plug-net-subflow-octavia-amp-post-vip-plug'
(f105ced1-72c6-4116-b582-599a21cdee36) transitioned into state
'REVERTED' from state 'REVERTING' with result 'None'<br>
2019-05-05 20:46:16.141 18234 DEBUG
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-plug-net-subflow-reload-amp-after-plug-vip'
(c4d6222e-2508-4a9c-9514-e7f9bcf84e31) transitioned into state
'REVERTING' from state 'SUCCESS' _task_receiver
/opt/openstack/lib/python3.6/site-packages/taskflow/listeners/logging.py:194<br>
2019-05-05 20:46:16.142 18234 WARNING
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-plug-net-subflow-reload-amp-after-plug-vip'
(c4d6222e-2508-4a9c-9514-e7f9bcf84e31) transitioned into state
'REVERTED' from state 'REVERTING' with result 'None'<br>
2019-05-05 20:46:16.146 18234 DEBUG
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-plug-net-subflow-ocatvia-amp-update-vip-data'
(2e1d1a04-282d-43b7-8c4f-fe31e75804ea) transitioned into state
'REVERTING' from state 'SUCCESS' _task_receiver
/opt/openstack/lib/python3.6/site-packages/taskflow/listeners/logging.py:194<br>
2019-05-05 20:46:16.148 18234 WARNING
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-plug-net-subflow-ocatvia-amp-update-vip-data'
(2e1d1a04-282d-43b7-8c4f-fe31e75804ea) transitioned into state
'REVERTED' from state 'REVERTING' with result 'None'<br>
2019-05-05 20:46:16.173 18234 DEBUG
octavia.controller.worker.controller_worker [-] Task
'STANDALONE-octavia-plug-net-subflow-octavia-amp-plug-vip'
(c63a5bed-f531-4ed3-83d2-bce72e835932) transitioned into state
'REVERTING' from state 'SUCCESS' _task_receiver
/opt/openstack/lib/python3.6/site-packages/taskflow/listeners/logging.py:194<br>
2019-05-05 20:46:16.174 18234 WARNING
octavia.controller.worker.tasks.network_tasks [-] Unable to plug VIP
for amphora id 5bec4c09-a209-4e73-a66e-e4fc0fb8ded5 load balancer id
e01c6ff5-179a-4ed5-ae5d-1d00d6c584b8<br>
<br>
and Neutron then deletes port but NOTE that immediately before
deletion port reported by neutron-dhcp-agent as ACTIVE:<br>
<br>
May 5 20:46:17 ardbeg neutron-dhcp-agent: 2019-05-05 20:46:17.080
1804 INFO neutron.agent.dhcp.agent
[req-835e5b91-28e5-44b9-a463-d04a0323294f
a18f38c780074c6280dde5edad159666 41a02a69918849509f4102b04f8a7de9 -
- -] Trigger reload_allocations for port admin_state_up=True,
allowed_address_pairs=[], binding:host_id=ardbeg, binding:profile=,
binding:vif_details=datapath_type=system, ovs_hybrid_plug=False,
port_filter=True, binding:vif_type=ovs, binding:vnic_type=normal,
created_at=2019-05-05T20:46:07Z, description=,
device_id=f1bce6e9-be5b-464b-8f64-686f36e9de1f,
device_owner=compute:nova, dns_assignment=[{u'hostname':
u'amphora-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5', u'ip_address':
u'10.0.2.13', u'fqdn':
u'amphora-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5.loqal.'}],
dns_domain=, dns_name=amphora-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5,
extra_dhcp_opts=[], fixed_ips=[{u'subnet_id':
u'24b10886-3d53-4aee-bdc6-f165b242ae4f', u'ip_address':
u'10.0.2.13'}], id=b0398cc8-6d52-4f12-9f1f-1141b0f10751,
mac_address=72:d0:1c:4c:94:91,
name=octavia-lb-vrrp-5bec4c09-a209-4e73-a66e-e4fc0fb8ded5,
network_id=b24d2830-eec6-4abd-82f2-ac71c8ecbf40,
port_security_enabled=True,
project_id=41a02a69918849509f4102b04f8a7de9, qos_policy_id=None,
revision_number=8,
security_groups=[u'ba20352e-95b9-4c97-a688-59d44e3aa8cf'],
status=ACTIVE, tags=[], tenant_id=41a02a69918849509f4102b04f8a7de9,
updated_at=2019-05-05T20:46:16Z<br>
May 5 20:46:17 controller-l neutron-server: 2019-05-05 20:46:17.086
20981 INFO neutron.wsgi [req-835e5b91-28e5-44b9-a463-d04a0323294f
a18f38c780074c6280dde5edad159666 41a02a69918849509f4102b04f8a7de9 -
default default] 10.0.10.31 "PUT
/v2.0/ports/b0398cc8-6d52-4f12-9f1f-1141b0f10751 HTTP/1.1" status:
200 len: 1395 time: 0.6318841<br>
May 5 20:46:17 controller-l neutron-server: 2019-05-05 20:46:17.153
20981 INFO neutron.wsgi [req-37ee0da3-8dcc-4fb8-9cd3-91c5a8dcedef
a18f38c780074c6280dde5edad159666 41a02a69918849509f4102b04f8a7de9 -
default default] 10.0.10.31 "GET
/v2.0/ports/b0398cc8-6d52-4f12-9f1f-1141b0f10751 HTTP/1.1" status:
200 len: 1395 time: 0.0616651<br>
May 5 20:46:18 controller-l neutron-server: 2019-05-05 20:46:18.179
20981 INFO neutron.wsgi [req-8896542e-5dcb-4e6d-9379-04cd88c4035b
a18f38c780074c6280dde5edad159666 41a02a69918849509f4102b04f8a7de9 -
default default] 10.0.10.31 "DELETE
/v2.0/ports/b0398cc8-6d52-4f12-9f1f-1141b0f10751 HTTP/1.1" status:
204 len: 149 time: 1.0199890<br>
<br>
Thank you.<br>
<br>
<pre class="moz-signature" cols="72">--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison</pre>
</body>
</html>