I'm seeing very similar errors on patches we are trying to merge in openstack-ansible. Same sort of trouble with Centos breaking and Ubuntu working. References to logs with errors like yours are in the comments on this patch https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/768514
I was trying to find something specific we are doing wrong on
Centos in openstack-ansible deployments but feels like there maybe
some common Centos related factor here.
Hi:Recently, we added a CI task[0] for octavia in the kolla project. and we tested octavia based on the ubuntu and centos systems.The ubuntu system worked as expected but Centos did not work.I tried to debug it and result is following:Octavia can not create a load balancer on centos8. because octavia-worker failed to plug a vip port on amphora vm.[1]2021-01-31 08:20:12.065 22 ERROR octavia.compute.drivers.nova_driver [-] Error attaching network None with ip None and port 26a39187-e95a-4131-91e7-24289e777f36 to amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) : novaclient.exceptions.ClientException: Unknown Error (HTTP 504)2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) into port 26a39187-e95a-4131-91e7-24289e777f36.: octavia.common.exceptions.ComputeUnknownException: Unknown exception from the compute driver: Unknown Error (HTTP 504).2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last):2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/compute/drivers/nova_driver.py", line 318, in attach_network_or_port2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id)2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return methods[-1].func(obj, *args, **kwargs)2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/v2/servers.py", line 2063, in interface_attach2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs obj_class=NetworkInterface)2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs resp, body = self.api.client.post(url, body=body)2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.request(url, 'POST', **kwargs)2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/client.py", line 78, in request2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs raise exceptions.from_response(resp, body, url, method)2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs novaclient.exceptions.ClientException: Unknown Error (HTTP 504)Octavia-work called Neutron API to create a port. And called nova-api to attach the vip to amphora.Neutron created port successfully, but nova failed to attach the port to instance.[2]2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [req-ab0e8d9b-664b-430f-8006-cad713b0c826 401ba22da5f8427fbda5fce24600041b 8ce8b97f710f43d7af2b8f9b1e0463c8 - default default] [instance: a210ec88-b554-487f-a125-30b5c7473060] attaching network adapter failed.: libvirt.libvirtError: Unable to read from monitor: Connection reset by peer2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] Traceback (most recent call last):2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2149, in attach_interface2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] guest.attach_device(cfg, persistent=True, live=live)2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 304, in attach_device2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] self._domain.attachDeviceFlags(device_xml, flags=flags)2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] result = proxy_call(self._autowrap, f, *args, **kwargs)2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = execute(f, *args, **kwargs)2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] six.reraise(c, e, tb)2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] raise value2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = meth(*args, **kwargs)2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 630, in attachDeviceFlags2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] libvirt.libvirtError: Unable to read from monitor: Connection reset by peer2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060]Nova-compute called libvirt to attach the device. And libvirt also failed to attach the device[3]2021-01-31 08:20:23.884+0000: 86663: error : qemuMonitorIORead:491 : Unable to read from monitor: Connection reset by peer2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:618 : Error on monitor Unable to read from monitor: Connection reset by peer2021-01-31 08:20:23.884+0000: 86663: info : virObjectRef:402 : OBJECT_REF: obj=0x7f004c00b6102021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:649 : Triggering error callback2021-01-31 08:20:23.884+0000: 86663: debug : qemuProcessHandleMonitorError:346 : Received error on 0x7f004c0095b0 'instance-00000001'2021-01-31 08:20:23.884+0000: 64768: debug : qemuMonitorSend:958 : Send command resulted in error Unable to read from monitor: Connection reset by peerI also tried to use kolla/ubuntu-source-nova-libvirt instead of kolla/centos-source-nova-libvirt. and it worked as expected.I think the root cause is that libvirt failed to attach a network device. but i don't know how to resolve this problem.could anyone help me?ThanksWuchunyang