Error while getting network agents

Lajos Katona katonalala at gmail.com
Tue Nov 16 09:42:03 UTC 2021 

Hi Muhammad,
>From Wallaby you should be able to delete agent through Neutron API:
https://review.opendev.org/c/openstack/neutron/+/752795

To tell the truth I don't know what happens if you execute
DELETE /v2.0/agents/{agent_id} after you have deleted the chassis in sbctl.

Regards
Lajos Katona (lajoskatona)

Muhammad Faisal <faisal.sheikh at rapidcompute.com> ezt írta (időpont: 2021.
nov. 15., H, 19:05):

> *Hi,*
>
> While executing openstack network agent list we are getting below
> mentioned error. We have run “ovn-sbctl chassis-del
> 650be87c-b581-467a-b523-ce454e753780” command on controller node.
>
> *OS:* Ubuntu 20
> *Openstack version:* Wallaby
> *Number of controller/network node:* 1 (172.16.30.46)
>
> *Number of compute node:* 2 (172.16.30.1, 172.16.30.3)
> *OVN Version:* 21.09.0
>
> /var/log/ovn/ovn-northd.log:
>
> 2021-11-15T15:29:22.184Z|00053|northd|WARN|Dropped 7 log messages in last
> 127 seconds (most recently, 120 seconds ago) due to excessive rate
>
> 2021-11-15T15:29:22.184Z|00054|northd|WARN|Chassis does not exist for
> Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
>
> 2021-11-15T15:31:11.516Z|00055|northd|WARN|Dropped 14 log messages in last
> 110 seconds (most recently, 52 seconds ago) due to excessive rate
>
> 2021-11-15T15:31:11.516Z|00056|northd|WARN|Chassis does not exist for
> Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
>
> 2021-11-15T15:35:25.738Z|00057|northd|WARN|Dropped 6 log messages in last
> 254 seconds (most recently, 243 seconds ago) due to excessive rate
>
> 2021-11-15T15:35:25.738Z|00058|northd|WARN|Chassis does not exist for
> Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
>
> 2021-11-15T15:44:34.663Z|00059|northd|WARN|Dropped 12 log messages in last
> 549 seconds (most recently, 512 seconds ago) due to excessive rate
>
> 2021-11-15T15:44:34.663Z|00060|northd|WARN|Chassis does not exist for
> Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
>
> 2021-11-15T15:47:39.947Z|00061|northd|WARN|Dropped 2 log messages in last
> 185 seconds (most recently, 185 seconds ago) due to excessive rate
>
> 2021-11-15T15:47:39.948Z|00062|northd|WARN|Chassis does not exist for
> Chassis_Private record, name: 7d07ed32-996f-48da-9ac1-7f0455ea7ec7
>
>
> /var/log/neutron/neutron-server.log:
> 2021-11-15 20:55:24.025 678149 DEBUG futurist.periodics
> [req-df627767-441c-4b46-8487-91604cd3033a - - - - -] Submitting periodic
> callback
> 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.HashRingHealthCheckPeriodics.touch_hash_ring_nodes'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>
> 2021-11-15 20:55:30.531 678146 DEBUG neutron.wsgi [-] (678146) accepted
> ('172.16.30.46', 49782) server
> /usr/lib/python3/dist-packages/eventlet/wsgi.py:992
>
> 2021-11-15 20:55:30.893 678146 DEBUG ovsdbapp.backend.ovs_idl.transaction
> [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Running txn n=1
> command(idx=0): CheckLivenessCommand() do_commit
> /usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
>
> 2021-11-15 20:55:30.925 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node
> 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling
> event "update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table:
> NB_Global) notify
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
>
> 2021-11-15 20:55:30.927 678147 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-]
> ChassisAgentWriteEvent : Matched Chassis_Private, update, None None matches
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
>
> 2021-11-15 20:55:30.930 678147 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-]
> ChassisMetadataAgentWriteEvent : Matched Chassis_Private, update, None None
> matches
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
>
> 2021-11-15 20:55:30.931 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node
> 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling
> event "update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table:
> NB_Global) notify
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
>
> 2021-11-15 20:55:30.934 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-04604530-8f57-428b-8359-e1bd2648d417 - - - - -] Hash Ring: Node
> 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling
> event "update" for row b4926d80-2c2a-488f-bd27-49c7191f0c3a (table:
> NB_Global) notify
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> [req-db3a5256-a3de-41a9-a135-710ce024a698 92e064e5e77c4e32b97e6c056a86c5eb
> 7eb59a51503f48ddbc936c40990e2177 - default default] index failed: No
> details.: AttributeError: 'Chassis_Private' object has no attribute
> 'hostname'
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource Traceback
> (most recent call last):
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron/api/v2/resource.py", line 98, in
> resource
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     result =
> method(request=request, **args)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> setattr(e, '_RETRY_EXCEEDED', True)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in
> __exit__
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> self.force_reraise()
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
> force_reraise
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     raise
> self.value
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
> f(*args, **kwargs)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_db/api.py", line 154, in wrapper
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> ectxt.value = e.inner_exc
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in
> __exit__
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> self.force_reraise()
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
> force_reraise
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     raise
> self.value
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
> f(*args, **kwargs)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> LOG.debug("Retry wrapper got retriable exception: %s", e)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in
> __exit__
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> self.force_reraise()
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
> force_reraise
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     raise
> self.value
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
> f(*dup_args, **dup_kwargs)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron/api/v2/base.py", line 369, in index
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
> self._items(request, True, parent_id)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron/api/v2/base.py", line 304, in _items
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     obj_list
> = obj_getter(request.context, **kwargs)
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py",
> line 1118, in fn
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     return
> op(results, new_method(*args, _driver=self, **kwargs))
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py",
> line 1182, in get_agents
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> agent_dict = agent.as_dict()
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource   File
> "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py",
> line 59, in as_dict
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource     'host':
> self.chassis.hostname,
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
> AttributeError: 'Chassis_Private' object has no attribute 'hostname'
>
> 2021-11-15 20:55:30.936 678146 ERROR neutron.api.v2.resource
>
> 2021-11-15 20:55:30.937 678146 INFO neutron.wsgi
> [req-db3a5256-a3de-41a9-a135-710ce024a698 92e064e5e77c4e32b97e6c056a86c5eb
> 7eb59a51503f48ddbc936c40990e2177 - default default] 172.16.30.46 "GET
> /v2.0/agents HTTP/1.1" status: 500  len: 368 time: 0.4032691
>
> 2021-11-15 20:55:30.938 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node
> 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling
> event "update" for row 2bd437c4-9173-4d1b-ad01-d27cf0a11c8a (table:
> SB_Global) notify
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
>
> 2021-11-15 20:55:30.939 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] ChassisAgentWriteEvent
> : Matched Chassis_Private, update, None None matches
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
>
> 2021-11-15 20:55:30.939 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node
> 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling
> event "update" for row 03ce3887-1007-48fa-9f40-b2fe88246a69 (table:
> Chassis_Private) notify
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
>
> 2021-11-15 20:55:30.940 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -]
> ChassisMetadataAgentWriteEvent : Matched Chassis_Private, update, None None
> matches
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:61
>
> 2021-11-15 20:55:30.940 678146 DEBUG
> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor
> [req-5efca86e-8945-43c5-a492-76e8ad16ef6e - - - - -] Hash Ring: Node
> 8318cb4d-c002-41f2-a17d-bb33d35d8b99 (host: controller-khi01) handling
> event "update" for row 03ce3887-1007-48fa-9f40-b2fe88246a69 (table:
> Chassis_Private) notify
> /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:666
>
> 2021-11-15 20:55:37.389 678146 INFO neutron.wsgi
> [req-85f09a48-f50f-4776-b110-5d07dd1dc39e 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/ports?device_id=21999c50-740c-4a26-970b-8e07ba37b5ac&fields=binding%3Ahost_id&fields=binding%3Avif_type
> HTTP/1.1" status: 200  len: 271 time: 0.0680673
>
> 2021-11-15 20:55:37.476 678146 DEBUG
> neutron.pecan_wsgi.hooks.policy_enforcement
> [req-4099412f-d724-48d7-8786-add0c5b47f56 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by
> policy engine: ['standard_attr_id'] _exclude_attributes_by_policy
> /usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:255
>
> 2021-11-15 20:55:37.477 678146 INFO neutron.wsgi
> [req-4099412f-d724-48d7-8786-add0c5b47f56 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/ports?tenant_id=7eb59a51503f48ddbc936c40990e2177&device_id=21999c50-740c-4a26-970b-8e07ba37b5ac
> HTTP/1.1" status: 200  len: 1175 time: 0.0565202
>
> 2021-11-15 20:55:37.618 678146 DEBUG
> neutron.pecan_wsgi.hooks.policy_enforcement
> [req-eebf3e86-206a-45ed-92f3-804007a39cca 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by
> policy engine: ['standard_attr_id', 'vlan_transparent']
> _exclude_attributes_by_policy
> /usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:255
>
> 2021-11-15 20:55:37.619 678146 INFO neutron.wsgi
> [req-eebf3e86-206a-45ed-92f3-804007a39cca 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/networks?id=ed4ee148-2562-41ed-93ac-7341948ac4dc HTTP/1.1" status:
> 200  len: 904 time: 0.1111536
>
> 2021-11-15 20:55:37.674 678146 INFO neutron.wsgi
> [req-71885d0c-4aa2-4377-a014-33768c9160ae 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/floatingips?fixed_ip_address=192.168.100.199&port_id=f9c17ae5-c07b-4411-a0be-551686e6ba8e
> HTTP/1.1" status: 200  len: 217 time: 0.0442567
>
> 2021-11-15 20:55:37.727 678146 DEBUG
> neutron.pecan_wsgi.hooks.policy_enforcement
> [req-d35a3b5e-82ea-4e80-b1c6-c8b39205a1f7 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] Attributes excluded by
> policy engine: ['standard_attr_id', 'shared'] _exclude_attributes_by_policy
> /usr/lib/python3/dist-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:255
>
> 2021-11-15 20:55:37.731 678146 INFO neutron.wsgi
> [req-d35a3b5e-82ea-4e80-b1c6-c8b39205a1f7 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/subnets?id=74e3b294-5fcc-4685-8bc4-5200be3af09e HTTP/1.1" status:
> 200  len: 858 time: 0.0468309
>
> 2021-11-15 20:55:37.774 678146 INFO neutron.wsgi
> [req-7a57b86f-75bf-4381-9447-0b1d7262aae8 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/ports?network_id=ed4ee148-2562-41ed-93ac-7341948ac4dc&device_owner=network%3Adhcp
> HTTP/1.1" status: 200  len: 210 time: 0.0335791
>
> 2021-11-15 20:55:37.891 678146 INFO neutron.wsgi
> [req-7cf3c09b-b5ee-4372-84c2-d738abd1a47a 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/networks/ed4ee148-2562-41ed-93ac-7341948ac4dc?fields=segments
> HTTP/1.1" status: 200  len: 212 time: 0.1059954
>
> 2021-11-15 20:55:37.991 678146 INFO neutron.wsgi
> [req-3ed1edbd-9f33-4b4e-89c8-a2291f23e8ef 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.3 "GET
> /v2.0/networks/ed4ee148-2562-41ed-93ac-7341948ac4dc?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type
> HTTP/1.1" status: 200  len: 277 time: 0.0903363
>
> 2021-11-15 20:55:49.083 678146 INFO neutron.wsgi
> [req-0328379e-d381-48be-a209-a94acc297baa 9888a6e6da764fd28a72b1a7b25b8967
> 3e076bf2747b40ed8792ebf85d14d719 - default default] 172.16.30.1 "GET
> /v2.0/ports?device_id=82e90d49-bbc4-4c2c-8ae1-7a4bcf84bb7d&fields=binding%3Ahost_id&fields=binding%3Avif_type
> HTTP/1.1" status: 200  len: 271 time: 0.2884536
>
> [image: Email Signature]
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211116/9a3744ae/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 33431 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211116/9a3744ae/attachment-0001.png>

More information about the openstack-discuss mailing list