[wallaby][neutron][ovn] SSL connection to OVN-NB/SB OVSDB
Hi, I am using Openstack Wallaby release with OVN on Ubuntu 20.04. My environment consists of 2 compute nodes and 1 controller node. ovs-vswitchd (Open vSwitch) 2.15.0 Ubuntu Kernel Version: 5.4.0-88-generic compute node1 172.16.30.1 compute node2 172.16.30.3 controller/Network node IP 172.16.30.46 I want to configure the ovn southbound and northbound database to listen on SSL connection. Set a certificate, private key, and CA certificate on both compute nodes and controller nodes in /etc/neutron/plugins/ml2/ml2_conf.ini and using string ssl:IP:Port to connect the southbound/northbound database but I am unable to establish connection on SSL. It's not connecting to ovsdb-server on 6641/6642. Error in the neutron logs is like below: 2021-10-12 17:15:27.728 50561 WARNING neutron.quota.resource_registry [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] security_group_rule is already registered 2021-10-12 17:15:27.754 50561 WARNING keystonemiddleware.auth_token [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2021-10-12 17:15:27.761 50561 INFO oslo_service.service [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Starting 1 workers 2021-10-12 17:15:27.768 50561 INFO neutron.service [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Neutron service started, listening on 0.0.0.0:9696 2021-10-12 17:15:27.776 50561 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:27.779 50561 CRITICAL neutron [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-373793 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:27.779 50561 ERROR neutron Traceback (most recent call last): 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module> 2021-10-12 17:15:27.779 50561 ERROR neutron sys.exit(main()) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main 2021-10-12 17:15:27.779 50561 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/__init__.py", line 68, in boot_server 2021-10-12 17:15:27.779 50561 ERROR neutron server_func() 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server 2021-10-12 17:15:27.779 50561 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 94, in serve_wsgi 2021-10-12 17:15:27.779 50561 ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/registry.py", line 60, in publish 2021-10-12 17:15:27.779 50561 ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 149, in publish 2021-10-12 17:15:27.779 50561 ERROR neutron return self.notify(resource, event, trigger, payload=payload) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 110, in _wrapped 2021-10-12 17:15:27.779 50561 ERROR neutron raise db_exc.RetryRequest(e) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-10-12 17:15:27.779 50561 ERROR neutron self.force_reraise() 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-10-12 17:15:27.779 50561 ERROR neutron raise self.value 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 105, in _wrapped 2021-10-12 17:15:27.779 50561 ERROR neutron return function(*args, **kwargs) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 174, in notify 2021-10-12 17:15:27.779 50561 ERROR neutron raise exceptions.CallbackFailure(errors=errors) 2021-10-12 17:15:27.779 50561 ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-373793 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:27.779 50561 ERROR neutron 2021-10-12 17:15:27.783 50572 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-373774 process, after_init: Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 294, in post_fork_initialize 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 357, in _wait_for_pg_drop_event 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager ovn_conf.get_ovn_nb_connection(), self.nb_schema_helper, self, 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 136, in nb_schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbNbOvnIdl.schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 721, in __get__ 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return self.func(owner) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 102, in schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager cls._schema_helper = idlutils.get_schema_helper(cls.connection_string, 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return create_schema_helper(fetch_schema_json(connection, schema_name)) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in fetch_schema_json 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager 2021-10-12 17:15:27.787 50572 INFO neutron.wsgi [-] (50572) wsgi starting up on http://0.0.0.0:9696 2021-10-12 17:15:27.924 50572 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-10-12 17:15:27.925 50572 INFO neutron.wsgi [-] (50572) wsgi exited, is_accepting=True 2021-10-12 17:15:29.709 50573 INFO neutron.common.config [-] Logging enabled! 2021-10-12 17:15:29.710 50573 INFO neutron.common.config [-] /usr/bin/neutron-server version 18.0.0 2021-10-12 17:15:29.712 50573 INFO neutron.common.config [-] Logging enabled! 2021-10-12 17:15:29.713 50573 INFO neutron.common.config [-] /usr/bin/neutron-server version 18.0.0 2021-10-12 17:15:29.899 50573 INFO keyring.backend [-] Loading KWallet 2021-10-12 17:15:29.904 50573 INFO keyring.backend [-] Loading SecretService 2021-10-12 17:15:29.907 50573 INFO keyring.backend [-] Loading Windows 2021-10-12 17:15:29.907 50573 INFO keyring.backend [-] Loading chainer 2021-10-12 17:15:29.908 50573 INFO keyring.backend [-] Loading macOS 2021-10-12 17:15:29.927 50573 INFO neutron.manager [-] Loading core plugin: ml2 2021-10-12 17:15:30.355 50573 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['flat', 'geneve'] 2021-10-12 17:15:30.357 50573 INFO neutron.plugins.ml2.drivers.type_flat [-] Arbitrary flat physical_network names allowed 2021-10-12 17:15:30.358 50573 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve'] 2021-10-12 17:15:30.358 50573 INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve']) 2021-10-12 17:15:30.359 50573 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['geneve'] 2021-10-12 17:15:30.359 50573 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] 2021-10-12 17:15:32.555 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension driver 'port_security' 2021-10-12 17:15:32.555 50573 INFO neutron.plugins.ml2.extensions.port_security [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] PortSecurityExtensionDriver initialization complete 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension driver 'qos' 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing mechanism driver 'ovn' 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting OVNMechanismDriver 2021-10-12 17:15:32.562 50573 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Firewall driver configuration is ignored 2021-10-12 17:15:32.586 50573 INFO neutron.services.logapi.drivers.ovn.driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] OVN logging driver registered 2021-10-12 17:15:32.588 50573 INFO neutron.plugins.ml2.plugin [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Modular L2 Plugin initialization complete 2021-10-12 17:15:32.589 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Got port-security extension from driver 'port_security' 2021-10-12 17:15:32.589 50573 INFO neutron.extensions.vlantransparent [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Disabled vlantransparent extension. 2021-10-12 17:15:32.589 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: ovn-router 2021-10-12 17:15:32.597 50573 INFO neutron.services.ovn_l3.plugin [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting OVNL3RouterPlugin 2021-10-12 17:15:32.597 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: qos 2021-10-12 17:15:32.600 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: metering 2021-10-12 17:15:32.603 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: port_forwarding 2021-10-12 17:15:32.605 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading service plugin ovn-router, it is required by port_forwarding 2021-10-12 17:15:32.606 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: segments 2021-10-12 17:15:32.684 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: auto_allocate 2021-10-12 17:15:32.685 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: tag 2021-10-12 17:15:32.687 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: timestamp 2021-10-12 17:15:32.689 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: network_ip_availability 2021-10-12 17:15:32.691 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: flavors 2021-10-12 17:15:32.693 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: revisions 2021-10-12 17:15:32.695 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension manager. 2021-10-12 17:15:32.696 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension address-group not supported by any of loaded plugins 2021-10-12 17:15:32.697 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: address-scope 2021-10-12 17:15:32.697 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension router-admin-state-down-before-update not supported by any of loaded plugins 2021-10-12 17:15:32.698 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: agent 2021-10-12 17:15:32.699 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension agent-resources-synced not supported by any of loaded plugins 2021-10-12 17:15:32.700 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: allowed-address-pairs 2021-10-12 17:15:32.701 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: auto-allocated-topology 2021-10-12 17:15:32.701 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: availability_zone 2021-10-12 17:15:32.702 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension availability_zone_filter not supported by any of loaded plugins 2021-10-12 17:15:32.703 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension data-plane-status not supported by any of loaded plugins 2021-10-12 17:15:32.703 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: default-subnetpools 2021-10-12 17:15:32.704 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dhcp_agent_scheduler not supported by any of loaded plugins 2021-10-12 17:15:32.705 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dns-integration not supported by any of loaded plugins 2021-10-12 17:15:32.706 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dns-domain-ports not supported by any of loaded plugins 2021-10-12 17:15:32.706 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dvr not supported by any of loaded plugins 2021-10-12 17:15:32.707 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension empty-string-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.708 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension expose-l3-conntrack-helper not supported by any of loaded plugins 2021-10-12 17:15:32.708 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: expose-port-forwarding-in-fip 2021-10-12 17:15:32.709 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: external-net 2021-10-12 17:15:32.710 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: extra_dhcp_opt 2021-10-12 17:15:32.710 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: extraroute 2021-10-12 17:15:32.711 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension extraroute-atomic not supported by any of loaded plugins 2021-10-12 17:15:32.712 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension filter-validation not supported by any of loaded plugins 2021-10-12 17:15:32.712 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: floating-ip-port-forwarding-description 2021-10-12 17:15:32.713 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: fip-port-details 2021-10-12 17:15:32.714 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: flavors 2021-10-12 17:15:32.715 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: floating-ip-port-forwarding 2021-10-12 17:15:32.715 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension floatingip-pools not supported by any of loaded plugins 2021-10-12 17:15:32.716 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: ip_allocation 2021-10-12 17:15:32.717 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension ip-substring-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.717 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: l2_adjacency 2021-10-12 17:15:32.718 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: router 2021-10-12 17:15:32.719 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-conntrack-helper not supported by any of loaded plugins 2021-10-12 17:15:32.720 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: ext-gw-mode 2021-10-12 17:15:32.721 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-ha not supported by any of loaded plugins 2021-10-12 17:15:32.721 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-flavors not supported by any of loaded plugins 2021-10-12 17:15:32.722 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins 2021-10-12 17:15:32.723 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3_agent_scheduler not supported by any of loaded plugins 2021-10-12 17:15:32.724 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension logging not supported by any of loaded plugins 2021-10-12 17:15:32.725 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: metering 2021-10-12 17:15:32.725 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: metering_source_and_destination_fields 2021-10-12 17:15:32.726 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: multi-provider 2021-10-12 17:15:32.727 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: net-mtu 2021-10-12 17:15:32.727 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: net-mtu-writable 2021-10-12 17:15:32.728 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: network_availability_zone 2021-10-12 17:15:32.729 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: network-ip-availability 2021-10-12 17:15:32.729 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension network-segment-range not supported by any of loaded plugins 2021-10-12 17:15:32.730 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: pagination 2021-10-12 17:15:32.731 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-device-profile 2021-10-12 17:15:32.731 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension port-mac-address-regenerate not supported by any of loaded plugins 2021-10-12 17:15:32.732 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-numa-affinity-policy 2021-10-12 17:15:32.733 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-resource-request 2021-10-12 17:15:32.733 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: binding 2021-10-12 17:15:32.734 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension binding-extended not supported by any of loaded plugins 2021-10-12 17:15:32.735 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-security 2021-10-12 17:15:32.735 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: project-id 2021-10-12 17:15:32.736 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: provider 2021-10-12 17:15:32.736 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos 2021-10-12 17:15:32.737 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-bw-limit-direction 2021-10-12 17:15:32.738 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-bw-minimum-ingress 2021-10-12 17:15:32.738 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-default 2021-10-12 17:15:32.739 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-fip 2021-10-12 17:15:32.740 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension qos-gateway-ip not supported by any of loaded plugins 2021-10-12 17:15:32.740 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-port-network-policy 2021-10-12 17:15:32.741 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-rule-type-details 2021-10-12 17:15:32.741 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-rules-alias 2021-10-12 17:15:32.742 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: quotas 2021-10-12 17:15:32.743 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: quota_details 2021-10-12 17:15:32.744 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: rbac-policies 2021-10-12 17:15:32.744 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-address-group not supported by any of loaded plugins 2021-10-12 17:15:32.745 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: rbac-address-scope 2021-10-12 17:15:32.746 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-security-groups not supported by any of loaded plugins 2021-10-12 17:15:32.746 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-subnetpool not supported by any of loaded plugins 2021-10-12 17:15:32.747 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: revision-if-match 2021-10-12 17:15:32.748 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-revisions 2021-10-12 17:15:32.748 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: router_availability_zone 2021-10-12 17:15:32.749 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension router-service-type not supported by any of loaded plugins 2021-10-12 17:15:32.749 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-groups-normalized-cidr 2021-10-12 17:15:32.750 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension port-security-groups-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.751 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-groups-remote-address-group 2021-10-12 17:15:32.756 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-group 2021-10-12 17:15:32.757 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: segment 2021-10-12 17:15:32.758 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: segments-peer-subnet-host-routes 2021-10-12 17:15:32.758 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: service-type 2021-10-12 17:15:32.759 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: sorting 2021-10-12 17:15:32.759 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-segment 2021-10-12 17:15:32.760 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-description 2021-10-12 17:15:32.760 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension stateful-security-group not supported by any of loaded plugins 2021-10-12 17:15:32.761 50573 WARNING neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Did not find expected name "Stdattrs_common" in /usr/lib/python3/dist-packages/neutron/extensions/stdattrs_common.py 2021-10-12 17:15:32.762 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet-dns-publish-fixed-ip not supported by any of loaded plugins 2021-10-12 17:15:32.762 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet_onboard not supported by any of loaded plugins 2021-10-12 17:15:32.763 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: subnet-segmentid-writable 2021-10-12 17:15:32.763 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet-service-types not supported by any of loaded plugins 2021-10-12 17:15:32.764 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: subnet_allocation 2021-10-12 17:15:32.765 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnetpool-prefix-ops not supported by any of loaded plugins 2021-10-12 17:15:32.765 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins 2021-10-12 17:15:32.766 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-tag 2021-10-12 17:15:32.767 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-timestamp 2021-10-12 17:15:32.767 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension trunk not supported by any of loaded plugins 2021-10-12 17:15:32.768 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension trunk-details not supported by any of loaded plugins 2021-10-12 17:15:32.769 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension uplink-status-propagation not supported by any of loaded plugins 2021-10-12 17:15:32.769 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension vlan-transparent not supported by any of loaded plugins 2021-10-12 17:15:32.771 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:network 2021-10-12 17:15:32.771 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:subnet 2021-10-12 17:15:32.772 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:subnetpool 2021-10-12 17:15:32.772 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:port 2021-10-12 17:15:32.774 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:router 2021-10-12 17:15:32.774 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:floatingip 2021-10-12 17:15:32.778 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of CountableResource for resource:rbac_policy 2021-10-12 17:15:32.778 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group 2021-10-12 17:15:32.779 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group_rule 2021-10-12 17:15:32.781 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:router 2021-10-12 17:15:32.781 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] router is already registered 2021-10-12 17:15:32.781 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:floatingip 2021-10-12 17:15:32.782 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] floatingip is already registered 2021-10-12 17:15:32.783 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of CountableResource for resource:rbac_policy 2021-10-12 17:15:32.783 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] rbac_policy is already registered 2021-10-12 17:15:32.783 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group 2021-10-12 17:15:32.783 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] security_group is already registered 2021-10-12 17:15:32.784 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group_rule 2021-10-12 17:15:32.784 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] security_group_rule is already registered 2021-10-12 17:15:32.810 50573 WARNING keystonemiddleware.auth_token [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2021-10-12 17:15:32.816 50573 INFO oslo_service.service [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting 1 workers 2021-10-12 17:15:32.824 50573 INFO neutron.service [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Neutron service started, listening on 0.0.0.0:9696 2021-10-12 17:15:32.831 50573 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:32.834 50573 CRITICAL neutron [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-904549 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:32.834 50573 ERROR neutron Traceback (most recent call last): 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module> 2021-10-12 17:15:32.834 50573 ERROR neutron sys.exit(main()) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main 2021-10-12 17:15:32.834 50573 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/__init__.py", line 68, in boot_server 2021-10-12 17:15:32.834 50573 ERROR neutron server_func() 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server 2021-10-12 17:15:32.834 50573 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 94, in serve_wsgi 2021-10-12 17:15:32.834 50573 ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/registry.py", line 60, in publish 2021-10-12 17:15:32.834 50573 ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 149, in publish 2021-10-12 17:15:32.834 50573 ERROR neutron return self.notify(resource, event, trigger, payload=payload) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 110, in _wrapped 2021-10-12 17:15:32.834 50573 ERROR neutron raise db_exc.RetryRequest(e) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-10-12 17:15:32.834 50573 ERROR neutron self.force_reraise() 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-10-12 17:15:32.834 50573 ERROR neutron raise self.value 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 105, in _wrapped 2021-10-12 17:15:32.834 50573 ERROR neutron return function(*args, **kwargs) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 174, in notify 2021-10-12 17:15:32.834 50573 ERROR neutron raise exceptions.CallbackFailure(errors=errors) 2021-10-12 17:15:32.834 50573 ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-904549 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:32.834 50573 ERROR neutron 2021-10-12 17:15:32.838 50582 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-904522 process, after_init: Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 294, in post_fork_initialize 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 357, in _wait_for_pg_drop_event 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager ovn_conf.get_ovn_nb_connection(), self.nb_schema_helper, self, 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 136, in nb_schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbNbOvnIdl.schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 721, in __get__ 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return self.func(owner) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 102, in schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager cls._schema_helper = idlutils.get_schema_helper(cls.connection_string, 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return create_schema_helper(fetch_schema_json(connection, schema_name)) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in fetch_schema_json 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager 2021-10-12 17:15:32.842 50582 INFO neutron.wsgi [-] (50582) wsgi starting up on http://0.0.0.0:9696 2021-10-12 17:15:32.961 50582 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-10-12 17:15:32.963 50582 INFO neutron.wsgi [-] (50582) wsgi exited, is_accepting=True 2021-10-12 17:15:34.722 50583 INFO neutron.common.config [-] Logging enabled! I would really appreciate any input in this regard. Best regards, Faisal Sheikh
Hi, To configure the OVN Northbound and Southbound databases connection with SSL you need to run: $ ovn-nbctl set-ssl <PRIV-KEY> <CERT> <CA-CERT> $ ovn-sbctl set-ssl <PRIV-KEY> <CERT> <CA-CERT> Then, for Neutron you need to set these six configuration options (3 for Northbound and 3 for Southbound): # /etc/neutron/plugins/ml2/ml2_conf.ini [ovn] ovn_sb_ca_cert="<CA-CERT>" ovn_sb_certificate="<CERT>" ovn_sb_private_key="<PRIV-KEY>" ovn_nb_ca_cert="<CA-CERT>" ovn_nb_certificate="<CERT>" ovn_nb_private_key="<PRIV-KEY>" And last, configure the OVN metadata agent. Do the same as above at /etc/neutron/neutron_ovn_metadata_agent.ini That should be it! Hope it helps, Lucas On Wed, Oct 13, 2021 at 5:02 PM Faisal Sheikh <faisalsheikh.cyber@gmail.com> wrote:
Hi,
I am using Openstack Wallaby release with OVN on Ubuntu 20.04. My environment consists of 2 compute nodes and 1 controller node. ovs-vswitchd (Open vSwitch) 2.15.0 Ubuntu Kernel Version: 5.4.0-88-generic compute node1 172.16.30.1 compute node2 172.16.30.3 controller/Network node IP 172.16.30.46
I want to configure the ovn southbound and northbound database to listen on SSL connection. Set a certificate, private key, and CA certificate on both compute nodes and controller nodes in /etc/neutron/plugins/ml2/ml2_conf.ini and using string ssl:IP:Port to connect the southbound/northbound database but I am unable to establish connection on SSL. It's not connecting to ovsdb-server on 6641/6642. Error in the neutron logs is like below:
2021-10-12 17:15:27.728 50561 WARNING neutron.quota.resource_registry [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] security_group_rule is already registered 2021-10-12 17:15:27.754 50561 WARNING keystonemiddleware.auth_token [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2021-10-12 17:15:27.761 50561 INFO oslo_service.service [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Starting 1 workers 2021-10-12 17:15:27.768 50561 INFO neutron.service [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Neutron service started, listening on 0.0.0.0:9696 2021-10-12 17:15:27.776 50561 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:27.779 50561 CRITICAL neutron [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-373793 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:27.779 50561 ERROR neutron Traceback (most recent call last): 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module> 2021-10-12 17:15:27.779 50561 ERROR neutron sys.exit(main()) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main 2021-10-12 17:15:27.779 50561 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/__init__.py", line 68, in boot_server 2021-10-12 17:15:27.779 50561 ERROR neutron server_func() 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server 2021-10-12 17:15:27.779 50561 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 94, in serve_wsgi 2021-10-12 17:15:27.779 50561 ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/registry.py", line 60, in publish 2021-10-12 17:15:27.779 50561 ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 149, in publish 2021-10-12 17:15:27.779 50561 ERROR neutron return self.notify(resource, event, trigger, payload=payload) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 110, in _wrapped 2021-10-12 17:15:27.779 50561 ERROR neutron raise db_exc.RetryRequest(e) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-10-12 17:15:27.779 50561 ERROR neutron self.force_reraise() 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-10-12 17:15:27.779 50561 ERROR neutron raise self.value 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 105, in _wrapped 2021-10-12 17:15:27.779 50561 ERROR neutron return function(*args, **kwargs) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 174, in notify 2021-10-12 17:15:27.779 50561 ERROR neutron raise exceptions.CallbackFailure(errors=errors) 2021-10-12 17:15:27.779 50561 ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-373793 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:27.779 50561 ERROR neutron 2021-10-12 17:15:27.783 50572 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-373774 process, after_init: Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 294, in post_fork_initialize 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 357, in _wait_for_pg_drop_event 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager ovn_conf.get_ovn_nb_connection(), self.nb_schema_helper, self, 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 136, in nb_schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbNbOvnIdl.schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 721, in __get__ 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return self.func(owner) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 102, in schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager cls._schema_helper = idlutils.get_schema_helper(cls.connection_string, 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return create_schema_helper(fetch_schema_json(connection, schema_name)) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in fetch_schema_json 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager 2021-10-12 17:15:27.787 50572 INFO neutron.wsgi [-] (50572) wsgi starting up on http://0.0.0.0:9696 2021-10-12 17:15:27.924 50572 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-10-12 17:15:27.925 50572 INFO neutron.wsgi [-] (50572) wsgi exited, is_accepting=True 2021-10-12 17:15:29.709 50573 INFO neutron.common.config [-] Logging enabled! 2021-10-12 17:15:29.710 50573 INFO neutron.common.config [-] /usr/bin/neutron-server version 18.0.0 2021-10-12 17:15:29.712 50573 INFO neutron.common.config [-] Logging enabled! 2021-10-12 17:15:29.713 50573 INFO neutron.common.config [-] /usr/bin/neutron-server version 18.0.0 2021-10-12 17:15:29.899 50573 INFO keyring.backend [-] Loading KWallet 2021-10-12 17:15:29.904 50573 INFO keyring.backend [-] Loading SecretService 2021-10-12 17:15:29.907 50573 INFO keyring.backend [-] Loading Windows 2021-10-12 17:15:29.907 50573 INFO keyring.backend [-] Loading chainer 2021-10-12 17:15:29.908 50573 INFO keyring.backend [-] Loading macOS 2021-10-12 17:15:29.927 50573 INFO neutron.manager [-] Loading core plugin: ml2 2021-10-12 17:15:30.355 50573 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['flat', 'geneve'] 2021-10-12 17:15:30.357 50573 INFO neutron.plugins.ml2.drivers.type_flat [-] Arbitrary flat physical_network names allowed 2021-10-12 17:15:30.358 50573 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve'] 2021-10-12 17:15:30.358 50573 INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve']) 2021-10-12 17:15:30.359 50573 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['geneve'] 2021-10-12 17:15:30.359 50573 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] 2021-10-12 17:15:32.555 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension driver 'port_security' 2021-10-12 17:15:32.555 50573 INFO neutron.plugins.ml2.extensions.port_security [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] PortSecurityExtensionDriver initialization complete 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension driver 'qos' 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing mechanism driver 'ovn' 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting OVNMechanismDriver 2021-10-12 17:15:32.562 50573 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Firewall driver configuration is ignored 2021-10-12 17:15:32.586 50573 INFO neutron.services.logapi.drivers.ovn.driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] OVN logging driver registered 2021-10-12 17:15:32.588 50573 INFO neutron.plugins.ml2.plugin [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Modular L2 Plugin initialization complete 2021-10-12 17:15:32.589 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Got port-security extension from driver 'port_security' 2021-10-12 17:15:32.589 50573 INFO neutron.extensions.vlantransparent [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Disabled vlantransparent extension. 2021-10-12 17:15:32.589 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: ovn-router 2021-10-12 17:15:32.597 50573 INFO neutron.services.ovn_l3.plugin [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting OVNL3RouterPlugin 2021-10-12 17:15:32.597 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: qos 2021-10-12 17:15:32.600 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: metering 2021-10-12 17:15:32.603 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: port_forwarding 2021-10-12 17:15:32.605 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading service plugin ovn-router, it is required by port_forwarding 2021-10-12 17:15:32.606 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: segments 2021-10-12 17:15:32.684 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: auto_allocate 2021-10-12 17:15:32.685 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: tag 2021-10-12 17:15:32.687 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: timestamp 2021-10-12 17:15:32.689 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: network_ip_availability 2021-10-12 17:15:32.691 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: flavors 2021-10-12 17:15:32.693 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: revisions 2021-10-12 17:15:32.695 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension manager. 2021-10-12 17:15:32.696 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension address-group not supported by any of loaded plugins 2021-10-12 17:15:32.697 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: address-scope 2021-10-12 17:15:32.697 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension router-admin-state-down-before-update not supported by any of loaded plugins 2021-10-12 17:15:32.698 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: agent 2021-10-12 17:15:32.699 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension agent-resources-synced not supported by any of loaded plugins 2021-10-12 17:15:32.700 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: allowed-address-pairs 2021-10-12 17:15:32.701 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: auto-allocated-topology 2021-10-12 17:15:32.701 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: availability_zone 2021-10-12 17:15:32.702 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension availability_zone_filter not supported by any of loaded plugins 2021-10-12 17:15:32.703 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension data-plane-status not supported by any of loaded plugins 2021-10-12 17:15:32.703 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: default-subnetpools 2021-10-12 17:15:32.704 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dhcp_agent_scheduler not supported by any of loaded plugins 2021-10-12 17:15:32.705 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dns-integration not supported by any of loaded plugins 2021-10-12 17:15:32.706 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dns-domain-ports not supported by any of loaded plugins 2021-10-12 17:15:32.706 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dvr not supported by any of loaded plugins 2021-10-12 17:15:32.707 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension empty-string-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.708 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension expose-l3-conntrack-helper not supported by any of loaded plugins 2021-10-12 17:15:32.708 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: expose-port-forwarding-in-fip 2021-10-12 17:15:32.709 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: external-net 2021-10-12 17:15:32.710 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: extra_dhcp_opt 2021-10-12 17:15:32.710 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: extraroute 2021-10-12 17:15:32.711 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension extraroute-atomic not supported by any of loaded plugins 2021-10-12 17:15:32.712 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension filter-validation not supported by any of loaded plugins 2021-10-12 17:15:32.712 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: floating-ip-port-forwarding-description 2021-10-12 17:15:32.713 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: fip-port-details 2021-10-12 17:15:32.714 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: flavors 2021-10-12 17:15:32.715 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: floating-ip-port-forwarding 2021-10-12 17:15:32.715 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension floatingip-pools not supported by any of loaded plugins 2021-10-12 17:15:32.716 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: ip_allocation 2021-10-12 17:15:32.717 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension ip-substring-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.717 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: l2_adjacency 2021-10-12 17:15:32.718 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: router 2021-10-12 17:15:32.719 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-conntrack-helper not supported by any of loaded plugins 2021-10-12 17:15:32.720 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: ext-gw-mode 2021-10-12 17:15:32.721 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-ha not supported by any of loaded plugins 2021-10-12 17:15:32.721 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-flavors not supported by any of loaded plugins 2021-10-12 17:15:32.722 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins 2021-10-12 17:15:32.723 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3_agent_scheduler not supported by any of loaded plugins 2021-10-12 17:15:32.724 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension logging not supported by any of loaded plugins 2021-10-12 17:15:32.725 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: metering 2021-10-12 17:15:32.725 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: metering_source_and_destination_fields 2021-10-12 17:15:32.726 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: multi-provider 2021-10-12 17:15:32.727 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: net-mtu 2021-10-12 17:15:32.727 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: net-mtu-writable 2021-10-12 17:15:32.728 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: network_availability_zone 2021-10-12 17:15:32.729 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: network-ip-availability 2021-10-12 17:15:32.729 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension network-segment-range not supported by any of loaded plugins 2021-10-12 17:15:32.730 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: pagination 2021-10-12 17:15:32.731 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-device-profile 2021-10-12 17:15:32.731 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension port-mac-address-regenerate not supported by any of loaded plugins 2021-10-12 17:15:32.732 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-numa-affinity-policy 2021-10-12 17:15:32.733 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-resource-request 2021-10-12 17:15:32.733 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: binding 2021-10-12 17:15:32.734 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension binding-extended not supported by any of loaded plugins 2021-10-12 17:15:32.735 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-security 2021-10-12 17:15:32.735 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: project-id 2021-10-12 17:15:32.736 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: provider 2021-10-12 17:15:32.736 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos 2021-10-12 17:15:32.737 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-bw-limit-direction 2021-10-12 17:15:32.738 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-bw-minimum-ingress 2021-10-12 17:15:32.738 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-default 2021-10-12 17:15:32.739 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-fip 2021-10-12 17:15:32.740 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension qos-gateway-ip not supported by any of loaded plugins 2021-10-12 17:15:32.740 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-port-network-policy 2021-10-12 17:15:32.741 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-rule-type-details 2021-10-12 17:15:32.741 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-rules-alias 2021-10-12 17:15:32.742 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: quotas 2021-10-12 17:15:32.743 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: quota_details 2021-10-12 17:15:32.744 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: rbac-policies 2021-10-12 17:15:32.744 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-address-group not supported by any of loaded plugins 2021-10-12 17:15:32.745 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: rbac-address-scope 2021-10-12 17:15:32.746 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-security-groups not supported by any of loaded plugins 2021-10-12 17:15:32.746 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-subnetpool not supported by any of loaded plugins 2021-10-12 17:15:32.747 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: revision-if-match 2021-10-12 17:15:32.748 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-revisions 2021-10-12 17:15:32.748 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: router_availability_zone 2021-10-12 17:15:32.749 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension router-service-type not supported by any of loaded plugins 2021-10-12 17:15:32.749 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-groups-normalized-cidr 2021-10-12 17:15:32.750 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension port-security-groups-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.751 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-groups-remote-address-group 2021-10-12 17:15:32.756 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-group 2021-10-12 17:15:32.757 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: segment 2021-10-12 17:15:32.758 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: segments-peer-subnet-host-routes 2021-10-12 17:15:32.758 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: service-type 2021-10-12 17:15:32.759 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: sorting 2021-10-12 17:15:32.759 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-segment 2021-10-12 17:15:32.760 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-description 2021-10-12 17:15:32.760 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension stateful-security-group not supported by any of loaded plugins 2021-10-12 17:15:32.761 50573 WARNING neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Did not find expected name "Stdattrs_common" in /usr/lib/python3/dist-packages/neutron/extensions/stdattrs_common.py 2021-10-12 17:15:32.762 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet-dns-publish-fixed-ip not supported by any of loaded plugins 2021-10-12 17:15:32.762 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet_onboard not supported by any of loaded plugins 2021-10-12 17:15:32.763 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: subnet-segmentid-writable 2021-10-12 17:15:32.763 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet-service-types not supported by any of loaded plugins 2021-10-12 17:15:32.764 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: subnet_allocation 2021-10-12 17:15:32.765 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnetpool-prefix-ops not supported by any of loaded plugins 2021-10-12 17:15:32.765 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins 2021-10-12 17:15:32.766 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-tag 2021-10-12 17:15:32.767 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-timestamp 2021-10-12 17:15:32.767 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension trunk not supported by any of loaded plugins 2021-10-12 17:15:32.768 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension trunk-details not supported by any of loaded plugins 2021-10-12 17:15:32.769 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension uplink-status-propagation not supported by any of loaded plugins 2021-10-12 17:15:32.769 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension vlan-transparent not supported by any of loaded plugins 2021-10-12 17:15:32.771 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:network 2021-10-12 17:15:32.771 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:subnet 2021-10-12 17:15:32.772 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:subnetpool 2021-10-12 17:15:32.772 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:port 2021-10-12 17:15:32.774 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:router 2021-10-12 17:15:32.774 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:floatingip 2021-10-12 17:15:32.778 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of CountableResource for resource:rbac_policy 2021-10-12 17:15:32.778 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group 2021-10-12 17:15:32.779 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group_rule 2021-10-12 17:15:32.781 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:router 2021-10-12 17:15:32.781 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] router is already registered 2021-10-12 17:15:32.781 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:floatingip 2021-10-12 17:15:32.782 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] floatingip is already registered 2021-10-12 17:15:32.783 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of CountableResource for resource:rbac_policy 2021-10-12 17:15:32.783 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] rbac_policy is already registered 2021-10-12 17:15:32.783 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group 2021-10-12 17:15:32.783 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] security_group is already registered 2021-10-12 17:15:32.784 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group_rule 2021-10-12 17:15:32.784 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] security_group_rule is already registered 2021-10-12 17:15:32.810 50573 WARNING keystonemiddleware.auth_token [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2021-10-12 17:15:32.816 50573 INFO oslo_service.service [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting 1 workers 2021-10-12 17:15:32.824 50573 INFO neutron.service [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Neutron service started, listening on 0.0.0.0:9696 2021-10-12 17:15:32.831 50573 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:32.834 50573 CRITICAL neutron [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-904549 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:32.834 50573 ERROR neutron Traceback (most recent call last): 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module> 2021-10-12 17:15:32.834 50573 ERROR neutron sys.exit(main()) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main 2021-10-12 17:15:32.834 50573 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/__init__.py", line 68, in boot_server 2021-10-12 17:15:32.834 50573 ERROR neutron server_func() 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server 2021-10-12 17:15:32.834 50573 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 94, in serve_wsgi 2021-10-12 17:15:32.834 50573 ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/registry.py", line 60, in publish 2021-10-12 17:15:32.834 50573 ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 149, in publish 2021-10-12 17:15:32.834 50573 ERROR neutron return self.notify(resource, event, trigger, payload=payload) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 110, in _wrapped 2021-10-12 17:15:32.834 50573 ERROR neutron raise db_exc.RetryRequest(e) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-10-12 17:15:32.834 50573 ERROR neutron self.force_reraise() 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-10-12 17:15:32.834 50573 ERROR neutron raise self.value 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 105, in _wrapped 2021-10-12 17:15:32.834 50573 ERROR neutron return function(*args, **kwargs) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 174, in notify 2021-10-12 17:15:32.834 50573 ERROR neutron raise exceptions.CallbackFailure(errors=errors) 2021-10-12 17:15:32.834 50573 ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-904549 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:32.834 50573 ERROR neutron 2021-10-12 17:15:32.838 50582 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-904522 process, after_init: Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 294, in post_fork_initialize 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 357, in _wait_for_pg_drop_event 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager ovn_conf.get_ovn_nb_connection(), self.nb_schema_helper, self, 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 136, in nb_schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbNbOvnIdl.schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 721, in __get__ 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return self.func(owner) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 102, in schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager cls._schema_helper = idlutils.get_schema_helper(cls.connection_string, 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return create_schema_helper(fetch_schema_json(connection, schema_name)) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in fetch_schema_json 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager 2021-10-12 17:15:32.842 50582 INFO neutron.wsgi [-] (50582) wsgi starting up on http://0.0.0.0:9696 2021-10-12 17:15:32.961 50582 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-10-12 17:15:32.963 50582 INFO neutron.wsgi [-] (50582) wsgi exited, is_accepting=True 2021-10-12 17:15:34.722 50583 INFO neutron.common.config [-] Logging enabled!
I would really appreciate any input in this regard.
Best regards, Faisal Sheikh
Hi Lucas, Thanks for your help. I was missing these two commands. $ ovn-nbctl set-ssl <PRIV-KEY> <CERT> <CA-CERT> $ ovn-sbctl set-ssl <PRIV-KEY> <CERT> <CA-CERT> It worked for me and now SSL connection is established with OVN NB/SB. kudos. BR, Muhammad Faisal Sheikh On Fri, Oct 15, 2021 at 1:44 PM Lucas Alvares Gomes <lucasagomes@gmail.com> wrote:
Hi,
To configure the OVN Northbound and Southbound databases connection with SSL you need to run:
$ ovn-nbctl set-ssl <PRIV-KEY> <CERT> <CA-CERT> $ ovn-sbctl set-ssl <PRIV-KEY> <CERT> <CA-CERT>
Then, for Neutron you need to set these six configuration options (3 for Northbound and 3 for Southbound):
# /etc/neutron/plugins/ml2/ml2_conf.ini [ovn] ovn_sb_ca_cert="<CA-CERT>" ovn_sb_certificate="<CERT>" ovn_sb_private_key="<PRIV-KEY>" ovn_nb_ca_cert="<CA-CERT>" ovn_nb_certificate="<CERT>" ovn_nb_private_key="<PRIV-KEY>"
And last, configure the OVN metadata agent. Do the same as above at /etc/neutron/neutron_ovn_metadata_agent.ini
That should be it!
Hope it helps, Lucas
On Wed, Oct 13, 2021 at 5:02 PM Faisal Sheikh <faisalsheikh.cyber@gmail.com> wrote:
Hi,
I am using Openstack Wallaby release with OVN on Ubuntu 20.04. My environment consists of 2 compute nodes and 1 controller node. ovs-vswitchd (Open vSwitch) 2.15.0 Ubuntu Kernel Version: 5.4.0-88-generic compute node1 172.16.30.1 compute node2 172.16.30.3 controller/Network node IP 172.16.30.46
I want to configure the ovn southbound and northbound database to listen on SSL connection. Set a certificate, private key, and CA certificate on both compute nodes and controller nodes in /etc/neutron/plugins/ml2/ml2_conf.ini and using string ssl:IP:Port to connect the southbound/northbound database but I am unable to establish connection on SSL. It's not connecting to ovsdb-server on 6641/6642. Error in the neutron logs is like below:
2021-10-12 17:15:27.728 50561 WARNING neutron.quota.resource_registry [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] security_group_rule is already registered 2021-10-12 17:15:27.754 50561 WARNING keystonemiddleware.auth_token [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2021-10-12 17:15:27.761 50561 INFO oslo_service.service [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Starting 1 workers 2021-10-12 17:15:27.768 50561 INFO neutron.service [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Neutron service started, listening on 0.0.0.0:9696 2021-10-12 17:15:27.776 50561 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:27.779 50561 CRITICAL neutron [req-94a8acc2-e0ca-4431-8bed-f26864e7740a - - - - -] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-373793 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:27.779 50561 ERROR neutron Traceback (most recent call last): 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module> 2021-10-12 17:15:27.779 50561 ERROR neutron sys.exit(main()) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main 2021-10-12 17:15:27.779 50561 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/__init__.py", line 68, in boot_server 2021-10-12 17:15:27.779 50561 ERROR neutron server_func() 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server 2021-10-12 17:15:27.779 50561 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 94, in serve_wsgi 2021-10-12 17:15:27.779 50561 ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/registry.py", line 60, in publish 2021-10-12 17:15:27.779 50561 ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 149, in publish 2021-10-12 17:15:27.779 50561 ERROR neutron return self.notify(resource, event, trigger, payload=payload) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 110, in _wrapped 2021-10-12 17:15:27.779 50561 ERROR neutron raise db_exc.RetryRequest(e) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-10-12 17:15:27.779 50561 ERROR neutron self.force_reraise() 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-10-12 17:15:27.779 50561 ERROR neutron raise self.value 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 105, in _wrapped 2021-10-12 17:15:27.779 50561 ERROR neutron return function(*args, **kwargs) 2021-10-12 17:15:27.779 50561 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 174, in notify 2021-10-12 17:15:27.779 50561 ERROR neutron raise exceptions.CallbackFailure(errors=errors) 2021-10-12 17:15:27.779 50561 ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-373793 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:27.779 50561 ERROR neutron 2021-10-12 17:15:27.783 50572 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-373774 process, after_init: Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 294, in post_fork_initialize 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 357, in _wait_for_pg_drop_event 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager ovn_conf.get_ovn_nb_connection(), self.nb_schema_helper, self, 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 136, in nb_schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbNbOvnIdl.schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 721, in __get__ 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return self.func(owner) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 102, in schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager cls._schema_helper = idlutils.get_schema_helper(cls.connection_string, 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager return create_schema_helper(fetch_schema_json(connection, schema_name)) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in fetch_schema_json 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:27.784 50572 ERROR neutron_lib.callbacks.manager 2021-10-12 17:15:27.787 50572 INFO neutron.wsgi [-] (50572) wsgi starting up on http://0.0.0.0:9696 2021-10-12 17:15:27.924 50572 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-10-12 17:15:27.925 50572 INFO neutron.wsgi [-] (50572) wsgi exited, is_accepting=True 2021-10-12 17:15:29.709 50573 INFO neutron.common.config [-] Logging enabled! 2021-10-12 17:15:29.710 50573 INFO neutron.common.config [-] /usr/bin/neutron-server version 18.0.0 2021-10-12 17:15:29.712 50573 INFO neutron.common.config [-] Logging enabled! 2021-10-12 17:15:29.713 50573 INFO neutron.common.config [-] /usr/bin/neutron-server version 18.0.0 2021-10-12 17:15:29.899 50573 INFO keyring.backend [-] Loading KWallet 2021-10-12 17:15:29.904 50573 INFO keyring.backend [-] Loading SecretService 2021-10-12 17:15:29.907 50573 INFO keyring.backend [-] Loading Windows 2021-10-12 17:15:29.907 50573 INFO keyring.backend [-] Loading chainer 2021-10-12 17:15:29.908 50573 INFO keyring.backend [-] Loading macOS 2021-10-12 17:15:29.927 50573 INFO neutron.manager [-] Loading core plugin: ml2 2021-10-12 17:15:30.355 50573 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['flat', 'geneve'] 2021-10-12 17:15:30.357 50573 INFO neutron.plugins.ml2.drivers.type_flat [-] Arbitrary flat physical_network names allowed 2021-10-12 17:15:30.358 50573 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve'] 2021-10-12 17:15:30.358 50573 INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve']) 2021-10-12 17:15:30.359 50573 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['geneve'] 2021-10-12 17:15:30.359 50573 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] 2021-10-12 17:15:30.360 50573 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] 2021-10-12 17:15:30.415 50573 INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' 2021-10-12 17:15:30.456 50573 INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] 2021-10-12 17:15:32.555 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension driver 'port_security' 2021-10-12 17:15:32.555 50573 INFO neutron.plugins.ml2.extensions.port_security [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] PortSecurityExtensionDriver initialization complete 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension driver 'qos' 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing mechanism driver 'ovn' 2021-10-12 17:15:32.556 50573 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting OVNMechanismDriver 2021-10-12 17:15:32.562 50573 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Firewall driver configuration is ignored 2021-10-12 17:15:32.586 50573 INFO neutron.services.logapi.drivers.ovn.driver [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] OVN logging driver registered 2021-10-12 17:15:32.588 50573 INFO neutron.plugins.ml2.plugin [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Modular L2 Plugin initialization complete 2021-10-12 17:15:32.589 50573 INFO neutron.plugins.ml2.managers [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Got port-security extension from driver 'port_security' 2021-10-12 17:15:32.589 50573 INFO neutron.extensions.vlantransparent [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Disabled vlantransparent extension. 2021-10-12 17:15:32.589 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: ovn-router 2021-10-12 17:15:32.597 50573 INFO neutron.services.ovn_l3.plugin [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting OVNL3RouterPlugin 2021-10-12 17:15:32.597 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: qos 2021-10-12 17:15:32.600 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: metering 2021-10-12 17:15:32.603 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: port_forwarding 2021-10-12 17:15:32.605 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading service plugin ovn-router, it is required by port_forwarding 2021-10-12 17:15:32.606 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: segments 2021-10-12 17:15:32.684 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: auto_allocate 2021-10-12 17:15:32.685 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: tag 2021-10-12 17:15:32.687 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: timestamp 2021-10-12 17:15:32.689 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: network_ip_availability 2021-10-12 17:15:32.691 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: flavors 2021-10-12 17:15:32.693 50573 INFO neutron.manager [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loading Plugin: revisions 2021-10-12 17:15:32.695 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Initializing extension manager. 2021-10-12 17:15:32.696 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension address-group not supported by any of loaded plugins 2021-10-12 17:15:32.697 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: address-scope 2021-10-12 17:15:32.697 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension router-admin-state-down-before-update not supported by any of loaded plugins 2021-10-12 17:15:32.698 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: agent 2021-10-12 17:15:32.699 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension agent-resources-synced not supported by any of loaded plugins 2021-10-12 17:15:32.700 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: allowed-address-pairs 2021-10-12 17:15:32.701 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: auto-allocated-topology 2021-10-12 17:15:32.701 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: availability_zone 2021-10-12 17:15:32.702 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension availability_zone_filter not supported by any of loaded plugins 2021-10-12 17:15:32.703 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension data-plane-status not supported by any of loaded plugins 2021-10-12 17:15:32.703 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: default-subnetpools 2021-10-12 17:15:32.704 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dhcp_agent_scheduler not supported by any of loaded plugins 2021-10-12 17:15:32.705 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dns-integration not supported by any of loaded plugins 2021-10-12 17:15:32.706 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dns-domain-ports not supported by any of loaded plugins 2021-10-12 17:15:32.706 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension dvr not supported by any of loaded plugins 2021-10-12 17:15:32.707 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension empty-string-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.708 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension expose-l3-conntrack-helper not supported by any of loaded plugins 2021-10-12 17:15:32.708 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: expose-port-forwarding-in-fip 2021-10-12 17:15:32.709 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: external-net 2021-10-12 17:15:32.710 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: extra_dhcp_opt 2021-10-12 17:15:32.710 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: extraroute 2021-10-12 17:15:32.711 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension extraroute-atomic not supported by any of loaded plugins 2021-10-12 17:15:32.712 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension filter-validation not supported by any of loaded plugins 2021-10-12 17:15:32.712 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: floating-ip-port-forwarding-description 2021-10-12 17:15:32.713 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: fip-port-details 2021-10-12 17:15:32.714 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: flavors 2021-10-12 17:15:32.715 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: floating-ip-port-forwarding 2021-10-12 17:15:32.715 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension floatingip-pools not supported by any of loaded plugins 2021-10-12 17:15:32.716 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: ip_allocation 2021-10-12 17:15:32.717 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension ip-substring-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.717 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: l2_adjacency 2021-10-12 17:15:32.718 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: router 2021-10-12 17:15:32.719 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-conntrack-helper not supported by any of loaded plugins 2021-10-12 17:15:32.720 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: ext-gw-mode 2021-10-12 17:15:32.721 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-ha not supported by any of loaded plugins 2021-10-12 17:15:32.721 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-flavors not supported by any of loaded plugins 2021-10-12 17:15:32.722 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins 2021-10-12 17:15:32.723 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension l3_agent_scheduler not supported by any of loaded plugins 2021-10-12 17:15:32.724 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension logging not supported by any of loaded plugins 2021-10-12 17:15:32.725 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: metering 2021-10-12 17:15:32.725 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: metering_source_and_destination_fields 2021-10-12 17:15:32.726 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: multi-provider 2021-10-12 17:15:32.727 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: net-mtu 2021-10-12 17:15:32.727 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: net-mtu-writable 2021-10-12 17:15:32.728 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: network_availability_zone 2021-10-12 17:15:32.729 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: network-ip-availability 2021-10-12 17:15:32.729 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension network-segment-range not supported by any of loaded plugins 2021-10-12 17:15:32.730 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: pagination 2021-10-12 17:15:32.731 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-device-profile 2021-10-12 17:15:32.731 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension port-mac-address-regenerate not supported by any of loaded plugins 2021-10-12 17:15:32.732 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-numa-affinity-policy 2021-10-12 17:15:32.733 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-resource-request 2021-10-12 17:15:32.733 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: binding 2021-10-12 17:15:32.734 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension binding-extended not supported by any of loaded plugins 2021-10-12 17:15:32.735 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: port-security 2021-10-12 17:15:32.735 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: project-id 2021-10-12 17:15:32.736 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: provider 2021-10-12 17:15:32.736 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos 2021-10-12 17:15:32.737 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-bw-limit-direction 2021-10-12 17:15:32.738 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-bw-minimum-ingress 2021-10-12 17:15:32.738 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-default 2021-10-12 17:15:32.739 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-fip 2021-10-12 17:15:32.740 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension qos-gateway-ip not supported by any of loaded plugins 2021-10-12 17:15:32.740 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-port-network-policy 2021-10-12 17:15:32.741 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-rule-type-details 2021-10-12 17:15:32.741 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: qos-rules-alias 2021-10-12 17:15:32.742 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: quotas 2021-10-12 17:15:32.743 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: quota_details 2021-10-12 17:15:32.744 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: rbac-policies 2021-10-12 17:15:32.744 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-address-group not supported by any of loaded plugins 2021-10-12 17:15:32.745 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: rbac-address-scope 2021-10-12 17:15:32.746 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-security-groups not supported by any of loaded plugins 2021-10-12 17:15:32.746 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension rbac-subnetpool not supported by any of loaded plugins 2021-10-12 17:15:32.747 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: revision-if-match 2021-10-12 17:15:32.748 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-revisions 2021-10-12 17:15:32.748 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: router_availability_zone 2021-10-12 17:15:32.749 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension router-service-type not supported by any of loaded plugins 2021-10-12 17:15:32.749 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-groups-normalized-cidr 2021-10-12 17:15:32.750 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension port-security-groups-filtering not supported by any of loaded plugins 2021-10-12 17:15:32.751 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-groups-remote-address-group 2021-10-12 17:15:32.756 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: security-group 2021-10-12 17:15:32.757 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: segment 2021-10-12 17:15:32.758 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: segments-peer-subnet-host-routes 2021-10-12 17:15:32.758 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: service-type 2021-10-12 17:15:32.759 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: sorting 2021-10-12 17:15:32.759 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-segment 2021-10-12 17:15:32.760 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-description 2021-10-12 17:15:32.760 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension stateful-security-group not supported by any of loaded plugins 2021-10-12 17:15:32.761 50573 WARNING neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Did not find expected name "Stdattrs_common" in /usr/lib/python3/dist-packages/neutron/extensions/stdattrs_common.py 2021-10-12 17:15:32.762 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet-dns-publish-fixed-ip not supported by any of loaded plugins 2021-10-12 17:15:32.762 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet_onboard not supported by any of loaded plugins 2021-10-12 17:15:32.763 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: subnet-segmentid-writable 2021-10-12 17:15:32.763 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnet-service-types not supported by any of loaded plugins 2021-10-12 17:15:32.764 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: subnet_allocation 2021-10-12 17:15:32.765 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension subnetpool-prefix-ops not supported by any of loaded plugins 2021-10-12 17:15:32.765 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins 2021-10-12 17:15:32.766 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-tag 2021-10-12 17:15:32.767 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Loaded extension: standard-attr-timestamp 2021-10-12 17:15:32.767 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension trunk not supported by any of loaded plugins 2021-10-12 17:15:32.768 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension trunk-details not supported by any of loaded plugins 2021-10-12 17:15:32.769 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension uplink-status-propagation not supported by any of loaded plugins 2021-10-12 17:15:32.769 50573 INFO neutron.api.extensions [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Extension vlan-transparent not supported by any of loaded plugins 2021-10-12 17:15:32.771 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:network 2021-10-12 17:15:32.771 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:subnet 2021-10-12 17:15:32.772 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:subnetpool 2021-10-12 17:15:32.772 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:port 2021-10-12 17:15:32.774 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:router 2021-10-12 17:15:32.774 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:floatingip 2021-10-12 17:15:32.778 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of CountableResource for resource:rbac_policy 2021-10-12 17:15:32.778 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group 2021-10-12 17:15:32.779 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group_rule 2021-10-12 17:15:32.781 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:router 2021-10-12 17:15:32.781 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] router is already registered 2021-10-12 17:15:32.781 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:floatingip 2021-10-12 17:15:32.782 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] floatingip is already registered 2021-10-12 17:15:32.783 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of CountableResource for resource:rbac_policy 2021-10-12 17:15:32.783 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] rbac_policy is already registered 2021-10-12 17:15:32.783 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group 2021-10-12 17:15:32.783 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] security_group is already registered 2021-10-12 17:15:32.784 50573 INFO neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Creating instance of TrackedResource for resource:security_group_rule 2021-10-12 17:15:32.784 50573 WARNING neutron.quota.resource_registry [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] security_group_rule is already registered 2021-10-12 17:15:32.810 50573 WARNING keystonemiddleware.auth_token [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2021-10-12 17:15:32.816 50573 INFO oslo_service.service [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Starting 1 workers 2021-10-12 17:15:32.824 50573 INFO neutron.service [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Neutron service started, listening on 0.0.0.0:9696 2021-10-12 17:15:32.831 50573 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:32.834 50573 CRITICAL neutron [req-06f63d07-b8d8-4c20-aa87-bdf06a3b17f5 - - - - -] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-904549 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:32.834 50573 ERROR neutron Traceback (most recent call last): 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module> 2021-10-12 17:15:32.834 50573 ERROR neutron sys.exit(main()) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main 2021-10-12 17:15:32.834 50573 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/__init__.py", line 68, in boot_server 2021-10-12 17:15:32.834 50573 ERROR neutron server_func() 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server 2021-10-12 17:15:32.834 50573 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 94, in serve_wsgi 2021-10-12 17:15:32.834 50573 ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/registry.py", line 60, in publish 2021-10-12 17:15:32.834 50573 ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 149, in publish 2021-10-12 17:15:32.834 50573 ERROR neutron return self.notify(resource, event, trigger, payload=payload) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 110, in _wrapped 2021-10-12 17:15:32.834 50573 ERROR neutron raise db_exc.RetryRequest(e) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-10-12 17:15:32.834 50573 ERROR neutron self.force_reraise() 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-10-12 17:15:32.834 50573 ERROR neutron raise self.value 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/db/utils.py", line 105, in _wrapped 2021-10-12 17:15:32.834 50573 ERROR neutron return function(*args, **kwargs) 2021-10-12 17:15:32.834 50573 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 174, in notify 2021-10-12 17:15:32.834 50573 ERROR neutron raise exceptions.CallbackFailure(errors=errors) 2021-10-12 17:15:32.834 50573 ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-904549 failed with "Could not retrieve schema from ssl:172.16.30.46:6641" 2021-10-12 17:15:32.834 50573 ERROR neutron 2021-10-12 17:15:32.838 50582 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to ssl:172.16.30.46:6641 to retrieve schema: Unknown error -1 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-904522 process, after_init: Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 294, in post_fork_initialize 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 357, in _wait_for_pg_drop_event 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager ovn_conf.get_ovn_nb_connection(), self.nb_schema_helper, self, 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 136, in nb_schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbNbOvnIdl.schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 721, in __get__ 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return self.func(owner) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 102, in schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager cls._schema_helper = idlutils.get_schema_helper(cls.connection_string, 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager return create_schema_helper(fetch_schema_json(connection, schema_name)) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in fetch_schema_json 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:172.16.30.46:6641 2021-10-12 17:15:32.840 50582 ERROR neutron_lib.callbacks.manager 2021-10-12 17:15:32.842 50582 INFO neutron.wsgi [-] (50582) wsgi starting up on http://0.0.0.0:9696 2021-10-12 17:15:32.961 50582 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-10-12 17:15:32.963 50582 INFO neutron.wsgi [-] (50582) wsgi exited, is_accepting=True 2021-10-12 17:15:34.722 50583 INFO neutron.common.config [-] Logging enabled!
I would really appreciate any input in this regard.
Best regards, Faisal Sheikh
participants (2)
-
Faisal Sheikh
-
Lucas Alvares Gomes