Open Stack

Hi,

While executing openstack network agent list we are getting below mentioned
error. We have run "ovn-sbctl chassis-del
650be87c-b581-467a-b523-ce454e753780" command on controller node.

OS: Ubuntu 20
Openstack version: Wallaby
Number of controller/network node: 1 (172.16.30.46)

Number of compute node: 2 (172.16.30.1, 172.16.30.3)
OVN Version: 21.09.0

/var/log/ovn/ovn-northd.log: 

2021-11-15T15:29:22.184Z|00053|northd|WARN|Dropped 7 log messages in last
127 seconds (most recently, 120 seconds ago) due to excessive rate

2021-11-15T15:29:22.184Z|00054|northd|WARN|Chassis does not exist for
Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7

2021-11-15T15:31:11.516Z|00055|northd|WARN|Dropped 14 log messages in last
110 seconds (most recently, 52 seconds ago) due to excessive rate

2021-11-15T15:31:11.516Z|00056|northd|WARN|Chassis does not exist for
Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7

2021-11-15T15:35:25.738Z|00057|northd|WARN|Dropped 6 log messages in last
254 seconds (most recently, 243 seconds ago) due to excessive rate

2021-11-15T15:35:25.738Z|00058|northd|WARN|Chassis does not exist for
Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7

2021-11-15T15:44:34.663Z|00059|northd|WARN|Dropped 12 log messages in last
549 seconds (most recently, 512 seconds ago) due to excessive rate

2021-11-15T15:44:34.663Z|00060|northd|WARN|Chassis does not exist for
Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7

2021-11-15T15:47:39.947Z|00061|northd|WARN|Dropped 2 log messages in last
185 seconds (most recently, 185 seconds ago) due to excessive rate

2021-11-15T15:47:39.948Z|00062|northd|WARN|Chassis does not exist for
Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7


/var/log/neutron/neutron-server.log:
2021-11-15 20:55:24.025 678149 DEBUG futurist.periodics
[req-df627767-441c-4b46-8487-91604cd3033a - - - - -] Submitting periodic
callback
'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.HashRingHealt
hCheckPeriodics.touch_hash_ring_nodes' _process_scheduled
/usr/lib/python3/dist-packages/futurist/periodics.py:641

2021-11-15 20:55:30.531 678146 DEBUG neutron.wsgi [-] (678146) accepted
('172.16.30.46', 49782) server
/usr/lib/python3/dist-packages/eventlet/wsgi.py:992

2021-11-15 20:55:30.893 678146 DEBUG ovsdbapp.backend.ovs_idl.transaction
[req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Running txn n=1
command(idx=0): CheckLivenessCommand() do_commit
/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py:90

2021-11-15 20:55:30.925 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node
8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event
"update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table: NB_Global)
notify
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:666

2021-11-15 20:55:30.927 678147 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-]
ChassisAgentWriteEvent : Matched Chassis_Private, update, None None matches
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:61

2021-11-15 20:55:30.930 678147 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-]
ChassisMetadataAgentWriteEvent : Matched Chassis_Private, update, None None
matches
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:61

2021-11-15 20:55:30.931 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node
8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event
"update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table: NB_Global)
notify
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:666

2021-11-15 20:55:30.934 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node
8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event
"update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table: NB_Global)
notify
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:666

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
[req-db3a5256-a3de-41a9-a135-710ce024a698 92e064e5e77c4e32b97e6c056a86c5eb
7eb59a51503f48ddbc936c40990e2177 - default default] index failed: No
details.: AttributeError: 'Chassis_Private' object has no attribute
'hostname'

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource Traceback (most
recent call last):

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron/api/v2/resource.py", line 98, in
resource

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     result =
method(request=request, **args)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 139, in wrapped

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     setattr(e,
'_RETRY_EXCEEDED', True)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in
__exit__

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
self.force_reraise()

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     raise
self.value

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 135, in wrapped

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
f(*args, **kwargs)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_db/api.py", line 154, in wrapper

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     ectxt.value
= e.inner_exc

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in
__exit__

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
self.force_reraise()

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     raise
self.value

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
f(*args, **kwargs)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 183, in wrapped

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
LOG.debug("Retry wrapper got retriable exception: %s", e)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in
__exit__

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
self.force_reraise()

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     raise
self.value

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 179, in wrapped

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
f(*dup_args, **dup_kwargs)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron/api/v2/base.py", line 369, in index

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
self._items(request, True, parent_id)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron/api/v2/base.py", line 304, in _items

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     obj_list =
obj_getter(request.context, **kwargs)

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/
mech_driver.py", line 1118, in fn

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
op(results, new_method(*args, _driver=self, **kwargs))

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/
mech_driver.py", line 1182, in get_agents

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     agent_dict
= agent.as_dict()

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
"/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/agent/neutro
n_agent.py", line 59, in as_dict

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     'host':
self.chassis.hostname,

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource AttributeError:
'Chassis_Private' object has no attribute 'hostname'

2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource 

2021-11-15 20:55:30.937 678146 INFO neutron.wsgi
[req-db3a5256-a3de-41a9-a135-710ce024a698 92e064e5e77c4e32b97e6c056a86c5eb
7eb59a51503f48ddbc936c40990e2177 - default default] 172.16.30.46 "GET
/v2.0/agents HTTP/1.1" status: 500  len: 368 time: 0.4032691

2021-11-15 20:55:30.938 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node
8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event
"update" for row 2bd437c4-9173-4d1b-ad01-d27cf0a11c8a (table: SB_Global)
notify
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:666

2021-11-15 20:55:30.939 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] ChassisAgentWriteEvent
: Matched Chassis_Private, update, None None matches
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:61

2021-11-15 20:55:30.939 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node
8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event
"update" for row 03ce3887-1007-48fa-9f40-b2fe88246a69 (table:
Chassis_Private) notify
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:666

2021-11-15 20:55:30.940 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -]
ChassisMetadataAgentWriteEvent : Matched Chassis_Private, update, None None
matches
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:61

2021-11-15 20:55:30.940 678146 DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
[req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node
8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event
"update" for row 03ce3887-1007-48fa-9f40-b2fe88246a69 (table:
Chassis_Private) notify
/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/o
vsdb/ovsdb_monitor.py:666

2021-11-15 20:55:37.389 678146 INFO neutron.wsgi
[req-85f09a48-f50f-4776-b110-5d07dd1dc39e 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/ports?device_id=21999c50-740c-4a26-970b-8e07ba37b5ac&fields=binding%3A
host_id&fields=binding%3Avif_type HTTP/1.1" status: 200  len: 271 time:
0.0680673

2021-11-15 20:55:37.476 678146 DEBUG
neutron.pecan_wsgi.hooks.policy_enforcement
[req-4099412f-d724-48d7-8786-add0c5b47f56 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by
policy engine: ['standard_attr_id'] _exclude_attributes_by_policy
/usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.p
y:255

2021-11-15 20:55:37.477 678146 INFO neutron.wsgi
[req-4099412f-d724-48d7-8786-add0c5b47f56 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/ports?tenant_id=7eb59a51503f48ddbc936c40990e2177&device_id=21999c50-74
0c-4a26-970b-8e07ba37b5ac HTTP/1.1" status: 200  len: 1175 time: 0.0565202

2021-11-15 20:55:37.618 678146 DEBUG
neutron.pecan_wsgi.hooks.policy_enforcement
[req-eebf3e86-206a-45ed-92f3-804007a39cca 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by
policy engine: ['standard_attr_id', 'vlan_transparent']
_exclude_attributes_by_policy
/usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.p
y:255

2021-11-15 20:55:37.619 678146 INFO neutron.wsgi
[req-eebf3e86-206a-45ed-92f3-804007a39cca 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/networks?id=ed4ee148-2562-41ed-93ac-7341948ac4dc HTTP/1.1" status: 200
len: 904 time: 0.1111536

2021-11-15 20:55:37.674 678146 INFO neutron.wsgi
[req-71885d0c-4aa2-4377-a014-33768c9160ae 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/floatingips?fixed_ip_address=192.168.100.199&port_id=f9c17ae5-c07b-441
1-a0be-551686e6ba8e HTTP/1.1" status: 200  len: 217 time: 0.0442567

2021-11-15 20:55:37.727 678146 DEBUG
neutron.pecan_wsgi.hooks.policy_enforcement
[req-d35a3b5e-82ea-4e80-b1c6-c8b39205a1f7 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by
policy engine: ['standard_attr_id', 'shared'] _exclude_attributes_by_policy
/usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.p
y:255

2021-11-15 20:55:37.731 678146 INFO neutron.wsgi
[req-d35a3b5e-82ea-4e80-b1c6-c8b39205a1f7 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/subnets?id=74e3b294-5fcc-4685-8bc4-5200be3af09e HTTP/1.1" status: 200
len: 858 time: 0.0468309

2021-11-15 20:55:37.774 678146 INFO neutron.wsgi
[req-7a57b86f-75bf-4381-9447-0b1d7262aae8 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/ports?network_id=ed4ee148-2562-41ed-93ac-7341948ac4dc&device_owner=net
work%3Adhcp HTTP/1.1" status: 200  len: 210 time: 0.0335791

2021-11-15 20:55:37.891 678146 INFO neutron.wsgi
[req-7cf3c09b-b5ee-4372-84c2-d738abd1a47a 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/networks/ed4ee148-2562-41ed-93ac-7341948ac4dc?fields=segments
HTTP/1.1" status: 200  len: 212 time: 0.1059954

2021-11-15 20:55:37.991 678146 INFO neutron.wsgi
[req-3ed1edbd-9f33-4b4e-89c8-a2291f23e8ef 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
/v2.0/networks/ed4ee148-2562-41ed-93ac-7341948ac4dc?fields=provider%3Aphysic
al_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200  len: 277
time: 0.0903363

2021-11-15 20:55:49.083 678146 INFO neutron.wsgi
[req-0328379e-d381-48be-a209-a94acc297baa 9888a6e6da764fd28a72b1a7b25b8967
3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.1 "GET
/v2.0/ports?device_id=82e90d49-bbc4-4c2c-8ae1-7a4bcf84bb7d&fields=binding%3A
host_id&fields=binding%3Avif_type HTTP/1.1" status: 200  len: 271 time:
0.2884536



 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211115/f93cd0af/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 33431 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211115/f93cd0af/attachment-0001.png>