[nova][octavia][kolla] failed to create loadbalancer on Centos8 system
Hi: Recently, we added a CI task[0] for octavia in the kolla project. and we tested octavia based on the ubuntu and centos systems. The ubuntu system worked as expected but Centos did not work. I tried to debug it and result is following: Octavia can not create a load balancer on centos8. because octavia-worker failed to plug a vip port on amphora vm.[1] 2021-01-31 08:20:12.065 22 ERROR octavia.compute.drivers.nova_driver [-] Error attaching network None with ip None and port 26a39187-e95a-4131-91e7-24289e777f36 to amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) : novaclient.exceptions.ClientException: Unknown Error (HTTP 504) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) into port 26a39187-e95a-4131-91e7-24289e777f36.: octavia.common.exceptions.ComputeUnknownException: Unknown exception from the compute driver: Unknown Error (HTTP 504). 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last): 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/compute/drivers/nova_driver.py", line 318, in attach_network_or_port 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return methods[-1].func(obj, *args, **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/v2/servers.py", line 2063, in interface_attach 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs obj_class=NetworkInterface) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs resp, body = self.api.client.post(url, body=body) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.request(url, 'POST', **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/client.py", line 78, in request 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs raise exceptions.from_response(resp, body, url, method) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs novaclient.exceptions.ClientException: Unknown Error (HTTP 504) Octavia-work called Neutron API to create a port. And called nova-api to attach the vip to amphora. Neutron created port successfully, but nova failed to attach the port to instance.[2] 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [req-ab0e8d9b-664b-430f-8006-cad713b0c826 401ba22da5f8427fbda5fce24600041b 8ce8b97f710f43d7af2b8f9b1e0463c8 - default default] [instance: a210ec88-b554-487f-a125-30b5c7473060] attaching network adapter failed.: libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] Traceback (most recent call last): 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2149, in attach_interface 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] guest.attach_device(cfg, persistent=True, live=live) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 304, in attach_device 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] self._domain.attachDeviceFlags(device_xml, flags=flags) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] result = proxy_call(self._autowrap, f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = execute(f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] six.reraise(c, e, tb) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] raise value 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = meth(*args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 630, in attachDeviceFlags 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] Nova-compute called libvirt to attach the device. And libvirt also failed to attach the device[3] 2021-01-31 08:20:23.884+0000: 86663: error : qemuMonitorIORead:491 : Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:618 : Error on monitor Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: info : virObjectRef:402 : OBJECT_REF: obj=0x7f004c00b610 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:649 : Triggering error callback 2021-01-31 08:20:23.884+0000: 86663: debug : qemuProcessHandleMonitorError:346 : Received error on 0x7f004c0095b0 'instance-00000001' 2021-01-31 08:20:23.884+0000: 64768: debug : qemuMonitorSend:958 : Send command resulted in error Unable to read from monitor: Connection reset by peer I also tried to use kolla/ubuntu-source-nova-libvirt instead of kolla/centos-source-nova-libvirt. and it worked as expected. I think the root cause is that libvirt failed to attach a network device. but i don't know how to resolve this problem. could anyone help me? Thanks Wuchunyang [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/754285 [1]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... [2]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... [3]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/...
I'm seeing very similar errors on patches we are trying to merge in openstack-ansible. Same sort of trouble with Centos breaking and Ubuntu working. References to logs with errors like yours are in the comments on this patch https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/768514 I was trying to find something specific we are doing wrong on Centos in openstack-ansible deployments but feels like there maybe some common Centos related factor here. On 04/02/2021 10:11, W Ch wrote:
Hi: Recently, we added a CI task[0] for octavia in the kolla project. and we tested octavia based on the ubuntu and centos systems. The ubuntu system worked as expected but Centos did not work. I tried to debug it and result is following: Octavia can not create a load balancer on centos8. because octavia-worker failed to plug a vip port on amphora vm.[1] 2021-01-31 08:20:12.065 22 ERROR octavia.compute.drivers.nova_driver [-] Error attaching network None with ip None and port 26a39187-e95a-4131-91e7-24289e777f36 to amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) : novaclient.exceptions.ClientException: Unknown Error (HTTP 504) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) into port 26a39187-e95a-4131-91e7-24289e777f36.: octavia.common.exceptions.ComputeUnknownException: Unknown exception from the compute driver: Unknown Error (HTTP 504). 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last): 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/compute/drivers/nova_driver.py", line 318, in attach_network_or_port 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return methods[-1].func(obj, *args, **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/v2/servers.py", line 2063, in interface_attach 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs obj_class=NetworkInterface) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs resp, body = self.api.client.post <http://self.api.client.post>(url, body=body) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.request(url, 'POST', **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/client.py", line 78, in request 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs raise exceptions.from_response(resp, body, url, method) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs novaclient.exceptions.ClientException: Unknown Error (HTTP 504) Octavia-work called Neutron API to create a port. And called nova-api to attach the vip to amphora. Neutron created port successfully, but nova failed to attach the port to instance.[2] 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [req-ab0e8d9b-664b-430f-8006-cad713b0c826 401ba22da5f8427fbda5fce24600041b 8ce8b97f710f43d7af2b8f9b1e0463c8 - default default] [instance: a210ec88-b554-487f-a125-30b5c7473060] attaching network adapter failed.: libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] Traceback (most recent call last): 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2149, in attach_interface 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] guest.attach_device(cfg, persistent=True, live=live) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 304, in attach_device 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] self._domain.attachDeviceFlags(device_xml, flags=flags) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] result = proxy_call(self._autowrap, f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = execute(f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] six.reraise(c, e, tb) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] raise value 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = meth(*args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 630, in attachDeviceFlags 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] Nova-compute called libvirt to attach the device. And libvirt also failed to attach the device[3] 2021-01-31 08:20:23.884+0000: 86663: error : qemuMonitorIORead:491 : Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:618 : Error on monitor Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: info : virObjectRef:402 : OBJECT_REF: obj=0x7f004c00b610 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:649 : Triggering error callback 2021-01-31 08:20:23.884+0000: 86663: debug : qemuProcessHandleMonitorError:346 : Received error on 0x7f004c0095b0 'instance-00000001' 2021-01-31 08:20:23.884+0000: 64768: debug : qemuMonitorSend:958 : Send command resulted in error Unable to read from monitor: Connection reset by peer I also tried to use kolla/ubuntu-source-nova-libvirt instead of kolla/centos-source-nova-libvirt. and it worked as expected. I think the root cause is that libvirt failed to attach a network device. but i don't know how to resolve this problem. could anyone help me? Thanks Wuchunyang [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/754285 <https://review.opendev.org/c/openstack/kolla-ansible/+/754285> [1]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... <https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/octavia/octavia-worker.txt#1116> [2]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... <https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/nova/nova-compute.txt#2546> [3]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... <https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/libvirt/libvirtd.txt#194472>
I just wanted to add that we started seeing this in mid December. The last patch landed that was passing CentOS 8 CI was 14th of December. 04.02.2021, 12:40, "Jonathan Rosser" <jonathan.rosser@rd.bbc.co.uk>:
I'm seeing very similar errors on patches we are trying to merge in openstack-ansible. Same sort of trouble with Centos breaking and Ubuntu working. References to logs with errors like yours are in the comments on this patch https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/768514
I was trying to find something specific we are doing wrong on Centos in openstack-ansible deployments but feels like there maybe some common Centos related factor here. On 04/02/2021 10:11, W Ch wrote:
Hi:
Recently, we added a CI task[0] for octavia in the kolla project. and we tested octavia based on the ubuntu and centos systems.
The ubuntu system worked as expected but Centos did not work.
I tried to debug it and result is following:
Octavia can not create a load balancer on centos8. because octavia-worker failed to plug a vip port on amphora vm.[1]
2021-01-31 08:20:12.065 22 ERROR octavia.compute.drivers.nova_driver [-] Error attaching network None with ip None and port 26a39187-e95a-4131-91e7-24289e777f36 to amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) : novaclient.exceptions.ClientException: Unknown Error (HTTP 504) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) into port 26a39187-e95a-4131-91e7-24289e777f36.: octavia.common.exceptions.ComputeUnknownException: Unknown exception from the compute driver: Unknown Error (HTTP 504). 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last): 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/compute/drivers/nova_driver.py", line 318, in attach_network_or_port 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return methods[-1].func(obj, *args, **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/v2/servers.py", line 2063, in interface_attach 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs obj_class=NetworkInterface) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs resp, body = self.api.client.post(url, body=body) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.request(url, 'POST', **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/client.py", line 78, in request 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs raise exceptions.from_response(resp, body, url, method) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs novaclient.exceptions.ClientException: Unknown Error (HTTP 504)
Octavia-work called Neutron API to create a port. And called nova-api to attach the vip to amphora.
Neutron created port successfully, but nova failed to attach the port to instance.[2]
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [req-ab0e8d9b-664b-430f-8006-cad713b0c826 401ba22da5f8427fbda5fce24600041b 8ce8b97f710f43d7af2b8f9b1e0463c8 - default default] [instance: a210ec88-b554-487f-a125-30b5c7473060] attaching network adapter failed.: libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] Traceback (most recent call last): 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2149, in attach_interface 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] guest.attach_device(cfg, persistent=True, live=live) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 304, in attach_device 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] self._domain.attachDeviceFlags(device_xml, flags=flags) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] result = proxy_call(self._autowrap, f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = execute(f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] six.reraise(c, e, tb) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] raise value 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = meth(*args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 630, in attachDeviceFlags 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060]
Nova-compute called libvirt to attach the device. And libvirt also failed to attach the device[3]
2021-01-31 08:20:23.884+0000: 86663: error : qemuMonitorIORead:491 : Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:618 : Error on monitor Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: info : virObjectRef:402 : OBJECT_REF: obj=0x7f004c00b610 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:649 : Triggering error callback 2021-01-31 08:20:23.884+0000: 86663: debug : qemuProcessHandleMonitorError:346 : Received error on 0x7f004c0095b0 'instance-00000001' 2021-01-31 08:20:23.884+0000: 64768: debug : qemuMonitorSend:958 : Send command resulted in error Unable to read from monitor: Connection reset by peer
I also tried to use kolla/ubuntu-source-nova-libvirt instead of kolla/centos-source-nova-libvirt. and it worked as expected.
I think the root cause is that libvirt failed to attach a network device. but i don't know how to resolve this problem.
could anyone help me?
Thanks
Wuchunyang
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/754285 [1]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... [2]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... [3]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/...
-- Kind Regards, Dmitriy Rabotyagov
On Thu, 4 Feb 2021 at 10:25, Dmitriy Rabotyagov <noonedeadpunk@ya.ru> wrote:
Hi,
We see pretty much the same for OpenStack-Ansible CI since mid December I guess. And it's specificly CentOS 8 job, while ubuntu bionic and focal and debian buster are working properly. In the meanwhile nova and all other project tests can create instances on centos 8.
The operation that fails is attaching a network interface to the amphora. I'd guess it's related to CentOS 8.3 which was released on 8th December. Mark
04.02.2021, 12:17, "W Ch" <wchy1001@gmail.com>:
Hi:
Recently, we added a CI task[0] for octavia in the kolla project. and we tested octavia based on the ubuntu and centos systems.
The ubuntu system worked as expected but Centos did not work.
I tried to debug it and result is following:
Octavia can not create a load balancer on centos8. because octavia-worker failed to plug a vip port on amphora vm.[1]
2021-01-31 08:20:12.065 22 ERROR octavia.compute.drivers.nova_driver [-] Error attaching network None with ip None and port 26a39187-e95a-4131-91e7-24289e777f36 to amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) : novaclient.exceptions.ClientException: Unknown Error (HTTP 504) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) into port 26a39187-e95a-4131-91e7-24289e777f36.: octavia.common.exceptions.ComputeUnknownException: Unknown exception from the compute driver: Unknown Error (HTTP 504). 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last): 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/compute/drivers/nova_driver.py", line 318, in attach_network_or_port 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return methods[-1].func(obj, *args, **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/v2/servers.py", line 2063, in interface_attach 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs obj_class=NetworkInterface) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs resp, body = self.api.client.post(url, body=body) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.request(url, 'POST', **kwargs) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/client.py", line 78, in request 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs raise exceptions.from_response(resp, body, url, method) 2021-01-31 08:20:12.066 22 ERROR octavia.network.drivers.neutron.allowed_address_pairs novaclient.exceptions.ClientException: Unknown Error (HTTP 504)
Octavia-work called Neutron API to create a port. And called nova-api to attach the vip to amphora.
Neutron created port successfully, but nova failed to attach the port to instance.[2]
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [req-ab0e8d9b-664b-430f-8006-cad713b0c826 401ba22da5f8427fbda5fce24600041b 8ce8b97f710f43d7af2b8f9b1e0463c8 - default default] [instance: a210ec88-b554-487f-a125-30b5c7473060] attaching network adapter failed.: libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] Traceback (most recent call last): 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2149, in attach_interface 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] guest.attach_device(cfg, persistent=True, live=live) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 304, in attach_device 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] self._domain.attachDeviceFlags(device_xml, flags=flags) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] result = proxy_call(self._autowrap, f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = execute(f, *args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] six.reraise(c, e, tb) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] raise value 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] rv = meth(*args, **kwargs) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 630, in attachDeviceFlags 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060] libvirt.libvirtError: Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: a210ec88-b554-487f-a125-30b5c7473060]
Nova-compute called libvirt to attach the device. And libvirt also failed to attach the device[3]
2021-01-31 08:20:23.884+0000: 86663: error : qemuMonitorIORead:491 : Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:618 : Error on monitor Unable to read from monitor: Connection reset by peer 2021-01-31 08:20:23.884+0000: 86663: info : virObjectRef:402 : OBJECT_REF: obj=0x7f004c00b610 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:649 : Triggering error callback 2021-01-31 08:20:23.884+0000: 86663: debug : qemuProcessHandleMonitorError:346 : Received error on 0x7f004c0095b0 'instance-00000001' 2021-01-31 08:20:23.884+0000: 64768: debug : qemuMonitorSend:958 : Send command resulted in error Unable to read from monitor: Connection reset by peer
I also tried to use kolla/ubuntu-source-nova-libvirt instead of kolla/centos-source-nova-libvirt. and it worked as expected.
I think the root cause is that libvirt failed to attach a network device. but i don't know how to resolve this problem.
could anyone help me?
Thanks
Wuchunyang
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/754285 [1]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... [2]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/... [3]: https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/...
-- Kind Regards, Dmitriy Rabotyagov
participants (4)
-
Dmitriy Rabotyagov
-
Jonathan Rosser
-
Mark Goddard
-
W Ch