[OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments...
Hey guys. I'm having problems with a new deploy. I'm relatively new to the project, so I still don't know how to guide myself in solving some errors.
When I create some Network and try to upload a VM the following errors appear in the Neutron container:
###############################
Jul 15 11:31:50 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:31:50.988 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:31:51.009 5683 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:31:51 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/availability_zones (ip 172.29.237.22) !!! Jul 15 11:33:53 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:33:53.860 5676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:33:58.733 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 uwsgi[5678]: Sat Jul 15 11:33:58 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!! Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:33:58.959 5677 INFO neutron.db.segments_db [None req-61542ff4-867d-402f-8239-b45392b2c188 f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Added segment 16ec3850-a4f2-40f3-9dc8-3b4650053935 of type vlan for network 6887bbfb-4680-4ed7-910b-90a65f7731dc Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.095 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:33:59 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.584 5680 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [None req-5ad409c5-c073-4726-bb26-5024d9c9c65b f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:33:59.684 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:32 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:41:32.961 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = connection.create_connection( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise err Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: sock.connect(sa) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 256, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: socket_checkerr(fd) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 54, in socket_checkerr Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise socket.error(err, errno.errorcode[err]) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: BrokenPipeError: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: httplib_response = self._make_request( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn.request(method, url, **httplib_request_kw) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: super(HTTPConnection, self).request(method, url, body=body, headers=headers) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1255, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_request(method, url, body, headers, encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.endheaders(body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_output(message_body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.send(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 950, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.connect() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = self._new_conn() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise NewConnectionError( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 489, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = conn.urlopen( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: retries = retries.increment( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise MaxRetryError(_pool, url, error or ResponseError(cause)) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1022, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.session.request(method, url, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 587, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.send(prep, **send_kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 701, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: r = adapter.send(request, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 565, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise ConnectionError(e, request=request) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 111, in wait Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: listener.cb(fileno) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/common/utils.py", line 924, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return func(*args, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 58, in synced_send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._notify() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 69, in _notify Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.callback(batched_events) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 211, in _send_notifications Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: event.method(event) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 383, in _delete_nova_inventory Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_id = self._get_aggregate_id(event.segment_id) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 370, in _get_aggregate_id Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_uuid = self.p_client.list_aggregates( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 58, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return f(self, *a, **k) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 554, in list_aggregates Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._get(url).json() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 190, in _get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._client.get(url, endpoint_filter=self._ks_filter, Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self.request(url, 'GET', **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 931, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = send(**kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1038, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise exceptions.ConnectFailure(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:41:33.144 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.29.236.2:8780/resource_providers/dd7e74fb-b494-448e-b16a-712a7ad6...: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Removing descriptor: 13 Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:41:33.542 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:42:36 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:42:36.571 5678 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:40 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:42:40.942 5689 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:41 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:42:41 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:42:42 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:42.053 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 3.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:45 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:45.240 5688 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-0270e149-44c9-49c4-af20-47918e00c4ea f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.563 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.564 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 2 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 3 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 4 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 5 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.798 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.799 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 6 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 7 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 8 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 9 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.000 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.001 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 10 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.056 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:54 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:42:54.511 5676 INFO neutron.notifiers.nova [-] Nova event matching ['req-0ffbf4aa-051f-4707-afe7-2908726310c5'] response: {'server_uuid': '9f206d9f-6c96-479f-bbaa-116d967129dc', 'name': 'network-vif-deleted', 'tag': 'caa01668-d5c0-4b37-a185-0ebc458326f7', 'status': 'completed', 'code': 200} Jul 15 11:48:24 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:48:24.062 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:48:31.762 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:48:31.828 5677 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 uwsgi[5677]: Sat Jul 15 11:48:31 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 neutron-api[5686]: 2023-07-15 11:48:32.108 5686 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 uwsgi[5686]: Sat Jul 15 11:48:32 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/networks?limit=21&sort_dir=asc&sort_key=id&router%3Aexternal=True&shared=False (ip 172.29.237.22) !!! Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:48:34.625 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:48:34.641 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:35 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:48:35.978 5682 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:48:36.052 5683 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:48:36.809 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:48:36 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!!
###############################
After the errors appear, the Horizon interface reports the following error:
Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f206d9f-6c96-479f-bbaa-116d967129dc.].
What could be happening? I believe I'm missing something to configure, but I don't know where to start.
Hey,
Can you kindly elaborate what neutron driver do you use for the deployment (OVN/OVS/LXB)? Or, if you unsure and was just relying on the default, what osa version you are using?
Also can you kindly provide output for "openstack network agent list" and "openstack compute service list"?
On Sat, Jul 15, 2023, 22:14 Murilo Morais murilo@evocorp.com.br wrote:
Hey guys. I'm having problems with a new deploy. I'm relatively new to the project, so I still don't know how to guide myself in solving some errors.
When I create some Network and try to upload a VM the following errors appear in the Neutron container:
###############################
Jul 15 11:31:50 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:31:50.988 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:31:51.009 5683 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:31:51 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/availability_zones (ip 172.29.237.22) !!! Jul 15 11:33:53 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:33:53.860 5676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:33:58.733 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 uwsgi[5678]: Sat Jul 15 11:33:58 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!! Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:33:58.959 5677 INFO neutron.db.segments_db [None req-61542ff4-867d-402f-8239-b45392b2c188 f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Added segment 16ec3850-a4f2-40f3-9dc8-3b4650053935 of type vlan for network 6887bbfb-4680-4ed7-910b-90a65f7731dc Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.095 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:33:59 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.584 5680 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [None req-5ad409c5-c073-4726-bb26-5024d9c9c65b f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:33:59.684 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:32 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:41:32.961 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = connection.create_connection( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise err Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: sock.connect(sa) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 256, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: socket_checkerr(fd) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 54, in socket_checkerr Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise socket.error(err, errno.errorcode[err]) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: BrokenPipeError: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: httplib_response = self._make_request( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn.request(method, url, **httplib_request_kw) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: super(HTTPConnection, self).request(method, url, body=body, headers=headers) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1255, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_request(method, url, body, headers, encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.endheaders(body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_output(message_body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.send(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 950, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.connect() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = self._new_conn() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise NewConnectionError( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 489, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = conn.urlopen( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: retries = retries.increment( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise MaxRetryError(_pool, url, error or ResponseError(cause)) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1022, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.session.request(method, url, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 587, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.send(prep, **send_kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 701, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: r = adapter.send(request, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 565, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise ConnectionError(e, request=request) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 111, in wait Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: listener.cb(fileno) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/common/utils.py", line 924, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return func(*args, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 58, in synced_send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._notify() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 69, in _notify Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.callback(batched_events) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 211, in _send_notifications Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: event.method(event) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 383, in _delete_nova_inventory Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_id = self._get_aggregate_id(event.segment_id) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 370, in _get_aggregate_id Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_uuid = self.p_client.list_aggregates( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 58, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return f(self, *a, **k) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 554, in list_aggregates Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._get(url).json() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 190, in _get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._client.get(url, endpoint_filter=self._ks_filter, Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self.request(url, 'GET', **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 931, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = send(**kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1038, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise exceptions.ConnectFailure(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:41:33.144 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.29.236.2:8780/resource_providers/dd7e74fb-b494-448e-b16a-712a7ad6...: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Removing descriptor: 13 Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:41:33.542 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:42:36 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:42:36.571 5678 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:40 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:42:40.942 5689 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:41 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:42:41 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:42:42 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:42.053 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 3.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:45 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:45.240 5688 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-0270e149-44c9-49c4-af20-47918e00c4ea f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.563 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.564 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 2 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 3 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 4 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 5 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.798 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.799 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 6 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 7 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 8 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 9 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.000 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.001 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 10 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.056 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:54 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:42:54.511 5676 INFO neutron.notifiers.nova [-] Nova event matching ['req-0ffbf4aa-051f-4707-afe7-2908726310c5'] response: {'server_uuid': '9f206d9f-6c96-479f-bbaa-116d967129dc', 'name': 'network-vif-deleted', 'tag': 'caa01668-d5c0-4b37-a185-0ebc458326f7', 'status': 'completed', 'code': 200} Jul 15 11:48:24 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:48:24.062 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:48:31.762 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:48:31.828 5677 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 uwsgi[5677]: Sat Jul 15 11:48:31 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 neutron-api[5686]: 2023-07-15 11:48:32.108 5686 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 uwsgi[5686]: Sat Jul 15 11:48:32 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/networks?limit=21&sort_dir=asc&sort_key=id&router%3Aexternal=True&shared=False (ip 172.29.237.22) !!! Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:48:34.625 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:48:34.641 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:35 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:48:35.978 5682 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:48:36.052 5683 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:48:36.809 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:48:36 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!!
###############################
After the errors appear, the Horizon interface reports the following error:
Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f206d9f-6c96-479f-bbaa-116d967129dc.].
What could be happening? I believe I'm missing something to configure, but I don't know where to start.
Hey,
Can you kindly elaborate what neutron driver do you use for the deployment (OVN/OVS/LXB)? Or, if you unsure and was just relying on the default, what osa version you are using?
Also can you kindly provide output for "openstack network agent list" and "openstack compute service list"?
Hi Dmitriy, thanks for answering!
I really didn't send any details of my setup, apologies for that.
I'm using OVS with the following configuration:
provider_networks: - network: container_bridge: br-provider container_type: veth type: vlan range: "100:200" net_name: vlan group_binds: - neutron_openvswitch_agent
- network: container_bridge: br-provider container_type: veth type: flat net_name: flat group_binds: - neutron_openvswitch_agent
neutron_plugin_type: ml2.ovs neutron_ml2_drivers_type: "flat,vlan" neutron_plugin_base: - router - metering
root@dcn2-utility-container-c45f8b09:/# openstack network agent list +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None | :-) | UP | neutron-metadata-agent | | a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None | :-) | UP | neutron-metering-agent | | c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova | :-) | UP | neutron-l3-agent | | da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+
root@dcn2-utility-container-c45f8b09:/# openstack compute service list +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:26.000000 | | c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:27.000000 | | 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2 | nova | enabled | up | 2023-07-18T12:32:27.000000 | +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+
The br-provider bridge exists and is UP.
I noticed that the "neutron-openvswitch-agent" process constantly goes up to 100% in CPU usage. This is normal?
Em ter., 18 de jul. de 2023 às 09:44, Murilo Morais murilo@evocorp.com.br escreveu:
Hi Dmitriy, thanks for answering!
I really didn't send any details of my setup, apologies for that.
I'm using OVS with the following configuration:
provider_networks:
network: container_bridge: br-provider container_type: veth type: vlan range: "100:200" net_name: vlan group_binds: - neutron_openvswitch_agent
network: container_bridge: br-provider container_type: veth type: flat net_name: flat group_binds: - neutron_openvswitch_agent
neutron_plugin_type: ml2.ovs neutron_ml2_drivers_type: "flat,vlan" neutron_plugin_base:
- router
- metering
root@dcn2-utility-container-c45f8b09:/# openstack network agent list
+--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None | :-) | UP | neutron-metadata-agent | | a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None | :-) | UP | neutron-metering-agent | | c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova | :-) | UP | neutron-l3-agent | | da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova | :-) | UP | neutron-dhcp-agent |
+--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+
root@dcn2-utility-container-c45f8b09:/# openstack compute service list
+--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At |
+--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:26.000000 | | c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:27.000000 | | 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2 | nova | enabled | up | 2023-07-18T12:32:27.000000 |
+--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+
The br-provider bridge exists and is UP.
Hey,
Thanks for providing the output. Regarding neutron-openvswitch-agent utilization - that might be a side-effect of neutron-server failures.
Regarding neutron-server failures, reading stack trace again, it seems that neutron-server is failing to reach the placement service: HTTPConnectionPool(host='172.29.236.2', port=8780) Failed to establish a new connection: [Errno 32] EPIPE
I assume that 172.29.236.2 is your internal VIP that's on haproxy? Are you able to reach the IP and connect to the port 8780 from inside of the neutron-server container? You can try leveraging telnet or simple curl from inside of the neutron-server container to verify that connectivity is fine.
Also good to ensure, that haproxy does have placement backends UP, you can verify that with running `echo "show stat" | nc -U /run/haproxy.stat | grep placement`
вт, 18 июл. 2023 г. в 15:14, Murilo Morais murilo@evocorp.com.br:
I noticed that the "neutron-openvswitch-agent" process constantly goes up to 100% in CPU usage. This is normal?
Em ter., 18 de jul. de 2023 às 09:44, Murilo Morais murilo@evocorp.com.br escreveu:
Hi Dmitriy, thanks for answering!
I really didn't send any details of my setup, apologies for that.
I'm using OVS with the following configuration:
provider_networks:
network: container_bridge: br-provider container_type: veth type: vlan range: "100:200" net_name: vlan group_binds: - neutron_openvswitch_agent
network: container_bridge: br-provider container_type: veth type: flat net_name: flat group_binds: - neutron_openvswitch_agent
neutron_plugin_type: ml2.ovs neutron_ml2_drivers_type: "flat,vlan" neutron_plugin_base:
- router
- metering
root@dcn2-utility-container-c45f8b09:/# openstack network agent list +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None | :-) | UP | neutron-metadata-agent | | a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None | :-) | UP | neutron-metering-agent | | c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova | :-) | UP | neutron-l3-agent | | da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+
root@dcn2-utility-container-c45f8b09:/# openstack compute service list +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:26.000000 | | c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:27.000000 | | 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2 | nova | enabled | up | 2023-07-18T12:32:27.000000 | +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+
The br-provider bridge exists and is UP.
Dmitry, good night!
I decided to start from scratch and remove things that I won't be using (at least not now).
When removing the flat network the errors are gone and the VMs start.
I just have another question related to Neutron, but I'll start another thread.
Thanks a lot for the help!
Em ter., 18 de jul. de 2023 às 14:12, Dmitriy Rabotyagov < noonedeadpunk@gmail.com> escreveu:
Hey,
Thanks for providing the output. Regarding neutron-openvswitch-agent utilization - that might be a side-effect of neutron-server failures.
Regarding neutron-server failures, reading stack trace again, it seems that neutron-server is failing to reach the placement service: HTTPConnectionPool(host='172.29.236.2', port=8780) Failed to establish a new connection: [Errno 32] EPIPE
I assume that 172.29.236.2 is your internal VIP that's on haproxy? Are you able to reach the IP and connect to the port 8780 from inside of the neutron-server container? You can try leveraging telnet or simple curl from inside of the neutron-server container to verify that connectivity is fine.
Also good to ensure, that haproxy does have placement backends UP, you can verify that with running `echo "show stat" | nc -U /run/haproxy.stat | grep placement`
вт, 18 июл. 2023 г. в 15:14, Murilo Morais murilo@evocorp.com.br:
I noticed that the "neutron-openvswitch-agent" process constantly goes
up to 100% in CPU usage. This is normal?
Em ter., 18 de jul. de 2023 às 09:44, Murilo Morais <
murilo@evocorp.com.br> escreveu:
Hi Dmitriy, thanks for answering!
I really didn't send any details of my setup, apologies for that.
I'm using OVS with the following configuration:
provider_networks:
network: container_bridge: br-provider container_type: veth type: vlan range: "100:200" net_name: vlan group_binds: - neutron_openvswitch_agent
network: container_bridge: br-provider container_type: veth type: flat net_name: flat group_binds: - neutron_openvswitch_agent
neutron_plugin_type: ml2.ovs neutron_ml2_drivers_type: "flat,vlan" neutron_plugin_base:
- router
- metering
root@dcn2-utility-container-c45f8b09:/# openstack network agent list
+--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+
| ID | Agent Type | Host |
Availability Zone | Alive | State | Binary |
+--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+
| 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None
| :-) | UP | neutron-metadata-agent || a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None
| :-) | UP | neutron-metering-agent || c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova
| :-) | UP | neutron-l3-agent || da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova
| :-) | UP | neutron-dhcp-agent |+--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+
root@dcn2-utility-container-c45f8b09:/# openstack compute service list
+--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host
| Zone | Status | State | Updated At|
+--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+
| a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor |
dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:26.000000 |
| c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler |
dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:27.000000 |
| 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2
| nova | enabled | up |2023-07-18T12:32:27.000000 |
+--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+
The br-provider bridge exists and is UP.
participants (2)
-
Dmitriy Rabotyagov
-
Murilo Morais