Hello Muhammad:
If the ovn-controller service is enabled in the host (in this case your controller), this service should create again the Chassis register. If the ovn-controller is not enabled, then it is possible to reproduce this issue.
I could reproduce it doing those steps: - Killing the ovn-controller service. Not stopping gracefully (that will delete the "chassis" and the "chassis_private" registers) but sending the kill signal. - Deleting the "chassis" register. That will leave the "chassis_private" register with chassis=[]. However, we don't attend "chassis" events. That means the "AgentCache" won't be updated. - Restart the Neutron server. That will force the OVB SB retrieval with the "chassis_private" register without any chassis. - List the agents --> that will trigger the reported error.
In any case, remember that the OVN agent deletion is only for clean-up purposes. If by any circumstance the ovn-controller in a host does not stop gracefully, it will leave "chassis" and "chassis_private" registers undeleted. To properly delete those registers, you should: - Delete both from OVN SB. - Then delete the agents from Neutron, using the CLI: "openstack network agent delete <ovn_agent_id>".
I'll open a bug to consider this very specific scenario.
Regards.
On Tue, Nov 16, 2021 at 5:22 PM Lajos Katona katonalala@gmail.com wrote:
Hi Muhammad, From Wallaby you should be able to delete agent through Neutron API: https://review.opendev.org/c/openstack/neutron/+/752795
To tell the truth I don't know what happens if you execute DELETE /v2.0/agents/{agent_id} after you have deleted the chassis in sbctl.
Regards Lajos Katona (lajoskatona)
Muhammad Faisal faisal.sheikh@rapidcompute.com ezt írta (időpont: 2021. nov. 15., H, 19:05):
*Hi,*
While executing openstack network agent list we are getting below mentioned error. We have run “ovn-sbctl chassis-del 650be87c-b581-467a-b523-ce454e753780” command on controller node.
*OS:* Ubuntu 20 *Openstack version:* Wallaby *Number of controller/network node:* 1 (172.16.30.46)
*Number of compute node:* 2 (172.16.30.1, 172.16.30.3) *OVN Version:* 21.09.0
/var/log/ovn/ovn-northd.log:
2021-11-15T15:29:22.184Z|00053|northd|WARN|Dropped 7 log messages in last 127 seconds (most recently, 120 seconds ago) due to excessive rate
2021-11-15T15:29:22.184Z|00054|northd|WARN|Chassis does not exist for Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
2021-11-15T15:31:11.516Z|00055|northd|WARN|Dropped 14 log messages in last 110 seconds (most recently, 52 seconds ago) due to excessive rate
2021-11-15T15:31:11.516Z|00056|northd|WARN|Chassis does not exist for Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
2021-11-15T15:35:25.738Z|00057|northd|WARN|Dropped 6 log messages in last 254 seconds (most recently, 243 seconds ago) due to excessive rate
2021-11-15T15:35:25.738Z|00058|northd|WARN|Chassis does not exist for Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
2021-11-15T15:44:34.663Z|00059|northd|WARN|Dropped 12 log messages in last 549 seconds (most recently, 512 seconds ago) due to excessive rate
2021-11-15T15:44:34.663Z|00060|northd|WARN|Chassis does not exist for Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
2021-11-15T15:47:39.947Z|00061|northd|WARN|Dropped 2 log messages in last 185 seconds (most recently, 185 seconds ago) due to excessive rate
2021-11-15T15:47:39.948Z|00062|northd|WARN|Chassis does not exist for Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
/var/log/neutron/neutron-server.log: 2021-11-15 20:55:24.025 678149 DEBUG futurist.periodics [req-df627767-441c-4b46-8487-91604cd3033a - - - - -] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.HashRingHealthCheckPeriodics.touch_hash_ring_nodes' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-11-15 20:55:30.531 678146 DEBUG neutron.wsgi [-] (678146) accepted ('172.16.30.46', 49782) server /usr/lib/python3/dist-packages/eventlet/wsgi.py:992
2021-11-15 20:55:30.893 678146 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Running txn n=1 command(idx=0): CheckLivenessCommand() do_commit /usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2021-11-15 20:55:30.925 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event "update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table: NB_Global) notify /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
2021-11-15 20:55:30.927 678147 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] ChassisAgentWriteEvent : Matched Chassis_Private, update, None None matches /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
2021-11-15 20:55:30.930 678147 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] ChassisMetadataAgentWriteEvent : Matched Chassis_Private, update, None None matches /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
2021-11-15 20:55:30.931 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event "update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table: NB_Global) notify /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
2021-11-15 20:55:30.934 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event "update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table: NB_Global) notify /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource [req-db3a5256-a3de-41a9-a135-710ce024a698 92e064e5e77c4e32b97e6c056a86c5eb 7eb59a51503f48ddbc936c40990e2177 - default default] index failed: No details.: AttributeError: 'Chassis_Private' object has no attribute 'hostname'
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource Traceback (most recent call last):
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron/api/v2/resource.py", line 98, in resource
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource result = method(request=request, **args)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource self.force_reraise()
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource raise self.value
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 154, in wrapper
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource self.force_reraise()
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource raise self.value
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource self.force_reraise()
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource raise self.value
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron/api/v2/base.py", line 369, in index
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource return self._items(request, True, parent_id)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron/api/v2/base.py", line 304, in _items
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource obj_list = obj_getter(request.context, **kwargs)
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1118, in fn
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource return op(results, new_method(*args, _driver=self, **kwargs))
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1182, in get_agents
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource agent_dict = agent.as_dict()
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py", line 59, in as_dict
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource 'host': self.chassis.hostname,
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource AttributeError: 'Chassis_Private' object has no attribute 'hostname'
2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
2021-11-15 20:55:30.937 678146 INFO neutron.wsgi [req-db3a5256-a3de-41a9-a135-710ce024a698 92e064e5e77c4e32b97e6c056a86c5eb 7eb59a51503f48ddbc936c40990e2177 - default default] 172.16.30.46 "GET /v2.0/agents HTTP/1.1" status: 500 len: 368 time: 0.4032691
2021-11-15 20:55:30.938 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event "update" for row 2bd437c4-9173-4d1b-ad01-d27cf0a11c8a (table: SB_Global) notify /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
2021-11-15 20:55:30.939 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] ChassisAgentWriteEvent : Matched Chassis_Private, update, None None matches /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
2021-11-15 20:55:30.939 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event "update" for row 03ce3887-1007-48fa-9f40-b2fe88246a69 (table: Chassis_Private) notify /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
2021-11-15 20:55:30.940 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] ChassisMetadataAgentWriteEvent : Matched Chassis_Private, update, None None matches /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
2021-11-15 20:55:30.940 678146 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling event "update" for row 03ce3887-1007-48fa-9f40-b2fe88246a69 (table: Chassis_Private) notify /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
2021-11-15 20:55:37.389 678146 INFO neutron.wsgi [req-85f09a48-f50f-4776-b110-5d07dd1dc39e 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/ports?device_id=21999c50-740c-4a26-970b-8e07ba37b5ac&fields=binding%3Ahost_id&fields=binding%3Avif_type HTTP/1.1" status: 200 len: 271 time: 0.0680673
2021-11-15 20:55:37.476 678146 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4099412f-d724-48d7-8786-add0c5b47f56 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by policy engine: ['standard_attr_id'] _exclude_attributes_by_policy /usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:255
2021-11-15 20:55:37.477 678146 INFO neutron.wsgi [req-4099412f-d724-48d7-8786-add0c5b47f56 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/ports?tenant_id=7eb59a51503f48ddbc936c40990e2177&device_id=21999c50-740c-4a26-970b-8e07ba37b5ac HTTP/1.1" status: 200 len: 1175 time: 0.0565202
2021-11-15 20:55:37.618 678146 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-eebf3e86-206a-45ed-92f3-804007a39cca 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by policy engine: ['standard_attr_id', 'vlan_transparent'] _exclude_attributes_by_policy /usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:255
2021-11-15 20:55:37.619 678146 INFO neutron.wsgi [req-eebf3e86-206a-45ed-92f3-804007a39cca 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/networks?id=ed4ee148-2562-41ed-93ac-7341948ac4dc HTTP/1.1" status: 200 len: 904 time: 0.1111536
2021-11-15 20:55:37.674 678146 INFO neutron.wsgi [req-71885d0c-4aa2-4377-a014-33768c9160ae 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/floatingips?fixed_ip_address=192.168.100.199&port_id=f9c17ae5-c07b-4411-a0be-551686e6ba8e HTTP/1.1" status: 200 len: 217 time: 0.0442567
2021-11-15 20:55:37.727 678146 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-d35a3b5e-82ea-4e80-b1c6-c8b39205a1f7 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by policy engine: ['standard_attr_id', 'shared'] _exclude_attributes_by_policy /usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:255
2021-11-15 20:55:37.731 678146 INFO neutron.wsgi [req-d35a3b5e-82ea-4e80-b1c6-c8b39205a1f7 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/subnets?id=74e3b294-5fcc-4685-8bc4-5200be3af09e HTTP/1.1" status: 200 len: 858 time: 0.0468309
2021-11-15 20:55:37.774 678146 INFO neutron.wsgi [req-7a57b86f-75bf-4381-9447-0b1d7262aae8 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/ports?network_id=ed4ee148-2562-41ed-93ac-7341948ac4dc&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 210 time: 0.0335791
2021-11-15 20:55:37.891 678146 INFO neutron.wsgi [req-7cf3c09b-b5ee-4372-84c2-d738abd1a47a 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/networks/ed4ee148-2562-41ed-93ac-7341948ac4dc?fields=segments HTTP/1.1" status: 200 len: 212 time: 0.1059954
2021-11-15 20:55:37.991 678146 INFO neutron.wsgi [req-3ed1edbd-9f33-4b4e-89c8-a2291f23e8ef 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET /v2.0/networks/ed4ee148-2562-41ed-93ac-7341948ac4dc?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200 len: 277 time: 0.0903363
2021-11-15 20:55:49.083 678146 INFO neutron.wsgi [req-0328379e-d381-48be-a209-a94acc297baa 9888a6e6da764fd28a72b1a7b25b8967 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.1 "GET /v2.0/ports?device_id=82e90d49-bbc4-4c2c-8ae1-7a4bcf84bb7d&fields=binding%3Ahost_id&fields=binding%3Avif_type HTTP/1.1" status: 200 len: 271 time: 0.2884536
[image: Email Signature]
