Dear Sir,

This our testbed:
Openstack (stein) + Opendaylight (neon) + ovs (v2.11.1)

We have trouble when creating vnf on compute node using OpenStack CLI:
#openstack vnf create vnfd1 vnf1

Here are the log message and ovs information.
COMPUTE NODE1:
nova-compute.log
=============================
Attempting claim on node cmp001: memory 2048 MB, disk 15 GB, vcpus 1 CPU
Total memory: 7976 MB, used: 6656.00 MB
memory limit not specified, defaulting to unlimited
Total disk: 96 GB, used: 45.00 GB
disk limit not specified, defaulting to unlimited
Total vcpu: 4 VCPU, used: 3.00 VCPU
vcpu limit not specified, defaulting to unlimited
Claim successful on node cmp001
Creating image
ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-a514ba2a-7ada-4a35-944e-0c861055112e eacae20dfeb9442eb8643bb8784b03d1 caee3c14634f42438ec8479eb49ba388 - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--db=tcp:127.0.0.1:6640', '--', '--may-exist', 'add-br', 'br-int', '--', 'set', 'Bridge', 'br-int', 'datapath_type=netdev']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json --db=tcp:127.0.0.1:6640 -- --may-exist add-br br-int -- set Bridge br-int datapath_type=netdev
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: tcp:127.0.0.1:6640: database connection failed (Connection refused)\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
ERROR vif_plug_ovs.ovsdb.impl_vsctl  Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--db=tcp:127.0.0.1:6640', '--', '--may-exist', 'add-port', 'br-int', 'vhu5e8b5246-cd', '--', 'set', 'Interface', 'vhu5e8b5246-cd', 'external_ids:iface-id=5e8b5246-cd05-4735-9692-29d51db6e72d', 'external_ids:iface-status=active', 'external_ids:attached-mac=fa:16:3e:8f:b1:a1', 'external_ids:vm-uuid=30675d24-bd74-409c-ba36-4c59a72283ec', 'type=dpdkvhostuser']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json --db=tcp:127.0.0.1:6640 -- --may-exist add-port br-int vhu5e8b5246-cd -- set Interface vhu5e8b5246-cd external_ids:iface-id=5e8b5246-cd05-4735-9692-29d51db6e72d external_ids:iface-status=active external_ids:attached-mac=fa:16:3e:8f:b1:a1 external_ids:vm-uuid=30675d24-bd74-409c-ba36-4c59a72283ec type=dpdkvhostuser
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: tcp:127.0.0.1:6640: database connection failed (Connection refused)\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-03-09 06:30:36.594 3282 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-a514ba2a-7ada-4a35-944e-0c861055112e eacae20dfeb9442eb8643bb8784b03d1 caee3c14634f42438ec8479eb49ba388 - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--db=tcp:127.0.0.1:6640', '--', '--columns=mtu_request', 'list', 'Interface']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json --db=tcp:127.0.0.1:6640 -- --columns=mtu_request list Interface
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: tcp:127.0.0.1:6640: database connection failed (Connection refused)\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-03-09 06:30:36.594 3282 ERROR os_vif [req-a514ba2a-7ada-4a35-944e-0c861055112e eacae20dfeb9442eb8643bb8784b03d1 caee3c14634f42438ec8479eb49ba388 - default default] Failed to plug vif VIFVHostUser(active=False,address=fa:16:3e:8f:b1:a1,has_traffic_filtering=False,id=5e8b5246-cd05-4735-9692-29d51db6e72d,mode='client',network=Network(6be19cbf-c6a7-4735-a0ee-42e1c5a664e2),path='/var/run/openvswitch/vhu5e8b5246-cd',plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='vhu5e8b5246-cd'): oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json --db=tcp:127.0.0.1:6640 -- --columns=mtu_request list Interface
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: tcp:127.0.0.1:6640: database connection failed (Connection refused)\n'
=============================
ovs information:
===========================
_uuid               : 4a56f11f-f839-4032-ba85-2eec47260144
bridges             : [1a600d85-9849-459a-8453-9a28eee63dc4, f21a1111-3c01-4988-99b7-ca5ab4a73051]
cur_cfg             : 653
datapath_types      : [netdev, system]
db_version          : "7.16.1"
dpdk_initialized    : false
dpdk_version        : none
external_ids        : {hostname="cmp001", "odl_os_hostconfig_config_odl_l2"="{\"allowed_network_types\": [\"local\", \"flat\", \"vlan\", \"vxlan\", \"gre\"], \"bridge_mappings\": {}, \"datapath_type\": \"netdev\", \"supported_vnic_types\": [{\"vif_details\": {\"uuid\": \"4a56f11f-f839-4032-ba85-2eec47260144\", \"host_addresses\": [\"cmp001\"], \"has_datapath_type_netdev\": true, \"support_vhost_user\": true, \"port_prefix\": \"vhu\", \"vhostuser_socket_dir\": \"/var/run/openvswitch\", \"vhostuser_ovs_plug\": true, \"vhostuser_mode\": \"client\", \"vhostuser_socket\": \"/var/run/openvswitch/vhu$PORT_ID\"}, \"vif_type\": \"vhostuser\", \"vnic_type\": \"normal\"}]}", odl_os_hostconfig_hostid="cmp001", rundir="/var/run/openvswitch", system-id="ec007ae5-4b61-4f8e-af2b-b1b97413d20d"}
iface_types         : [erspan, geneve, gre, internal, "ip6erspan", "ip6gre", lisp, patch, stt, system, tap, vxlan]
manager_options     : [11b3ada9-d9b3-42bb-97fc-d254364acecd]
next_cfg            : 653
other_config        : {local_ip="10.1.0.5", provider_mappings="physnet1:br-floating"}
ovs_version         : "2.11.1"
ssl                 : []
statistics          : {}
system_type         : ubuntu
system_version      : "18.04"
===========================
# ovs-vsctl get bridge br-int datapath_type
system
===========================
COMPUTE NODE 2:
nova-compute.log
===========================
ERROR nova.compute.manager Failed to build and run instance: libvirt.libvirtError: internal error: process exited while connecting to monitor: 
qemu-system-x86_64: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhub701eeae-c0: Failed to connect socket /var/run/openvswitch/vhub701eeae-c0: No such file or directory
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2353, in _build_and_run_instance
    block_device_info=block_device_info)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3204, in spawn
    destroy_disks_on_failure=True)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 5724, in _create_domain_and_network
    destroy_disks_on_failure)
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 5693, in _create_domain_and_network
    post_xml_callback=post_xml_callback)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 5627, in _create_domain
    guest.launch(pause=pause)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 144, in launch
    self._encoded_xml, errors='ignore')
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 139, in launch
    return self._domain.createWithFlags(flags)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 190, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 148, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 129, in execute
    six.reraise(c, e, tb)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 83, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1110, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirt.libvirtError: internal error: process exited while connecting to monitor: 2021-03-10T07:59:04.064555Z qemu-system-x86_64: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhub701eeae-c0: Failed to connect socket /var/run/openvswitch/vhub701eeae-c0: No such file or directory
INFO nova.compute.manager Took 0.68 seconds to deallocate network for instance.
INFO nova.scheduler.client.report Deleted allocation for instance 308de941-1192-4dbd-8c4d-d7860139d6d4
===========================
ovs information:
===========================
_uuid               : b8f5b869-e794-4052-8896-3ee5d323a0b7
bridges             : [5b6ab0c7-d8f1-43b0-9ab7-1e77280cd3ff, 70810190-6796-4500-bda4-dae856bcede1]
cur_cfg             : 1267
datapath_types      : [netdev, system]
db_version          : "7.16.1"
dpdk_initialized    : false
dpdk_version        : none
external_ids        : {hostname="cmp002", "odl_os_hostconfig_config_odl_l2"="{\"allowed_network_types\": [\"local\", \"flat\", \"vlan\", \"vxlan\", \"gre\"], \"bridge_mappings\": {}, \"datapath_type\": \"netdev\", \"supported_vnic_types\": [{\"vif_details\": {\"uuid\": \"b8f5b869-e794-4052-8896-3ee5d323a0b7\", \"host_addresses\": [\"cmp002\"], \"has_datapath_type_netdev\": true, \"support_vhost_user\": true, \"port_prefix\": \"vhu\", \"vhostuser_socket_dir\": \"/var/run/openvswitch\", \"vhostuser_ovs_plug\": true, \"vhostuser_mode\": \"client\", \"vhostuser_socket\": \"/var/run/openvswitch/vhu$PORT_ID\"}, \"vif_type\": \"vhostuser\", \"vnic_type\": \"normal\"}]}", odl_os_hostconfig_hostid="cmp002", rundir="/var/run/openvswitch", system-id="dddc6d6c-0e7c-4813-97c9-7c72eaf9f46b"}
iface_types         : [erspan, geneve, gre, internal, "ip6erspan", "ip6gre", lisp, patch, stt, system, tap, vxlan]
manager_options     : [29b4ef4b-e2ba-43f3-aa42-b1bacaf9dd19, 75ae95f6-d251-4152-9e0d-5b65e5f7bb77]
next_cfg            : 1267
other_config        : {local_ip="10.1.0.6", provider_mappings="physnet1:br-floating"}
ovs_version         : "2.11.1"
ssl                 : []
statistics          : {}
system_type         : ubuntu
system_version      : "18.04"
===========================
# ovs-vsctl get bridge br-int datapath_type
netdev
===========================

We don't use DPDK on our Openvswitch.
Can anyone give us some advice on this issue?

Thanks for helping us.

Sender notified by
Mailtrack 03/10/21, 04:09:46 PM