Open Stack

I'm seeing very similar errors on patches we are trying to merge in 
openstack-ansible. Same sort of trouble with Centos breaking and Ubuntu 
working. References to logs with errors like yours are in the comments 
on this patch 
https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/768514

I was trying to find something specific we are doing wrong on Centos in 
openstack-ansible deployments but feels like there maybe some common 
Centos related factor here.

On 04/02/2021 10:11, W Ch wrote:
> Hi:
> Recently, we added a CI task[0] for octavia in the kolla project. and 
> we tested octavia based on the ubuntu and centos systems.
> The ubuntu system worked as expected but Centos did not work.
> I tried to debug it and result is following:
> Octavia can not create a load balancer on centos8. because 
> octavia-worker failed to plug a vip port on amphora vm.[1]
> 2021-01-31 08:20:12.065 22 ERROR octavia.compute.drivers.nova_driver 
> [-] Error attaching network None with ip None and port 
> 26a39187-e95a-4131-91e7-24289e777f36 to amphora (compute_id: 
> a210ec88-b554-487f-a125-30b5c7473060) : 
> novaclient.exceptions.ClientException: Unknown Error (HTTP 504)
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs [-] Error 
> plugging amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) 
> into port 26a39187-e95a-4131-91e7-24289e777f36.: 
> octavia.common.exceptions.ComputeUnknownException: Unknown exception 
> from the compute driver: Unknown Error (HTTP 504).
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs Traceback (most 
> recent call last):
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/compute/drivers/nova_driver.py", 
> line 318, in attach_network_or_port
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id)
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/api_versions.py", 
> line 393, in substitution
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs return 
> methods[-1].func(obj, *args, **kwargs)
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/v2/servers.py", 
> line 2063, in interface_attach
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs 
> obj_class=NetworkInterface)
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/base.py", 
> line 363, in _create
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs resp, body = 
> self.api.client.post <http://self.api.client.post>(url, body=body)
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", 
> line 401, in post
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs return 
> self.request(url, 'POST', **kwargs)
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/client.py", 
> line 78, in request
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs raise 
> exceptions.from_response(resp, body, url, method)
> 2021-01-31 08:20:12.066 22 ERROR 
> octavia.network.drivers.neutron.allowed_address_pairs 
> novaclient.exceptions.ClientException: Unknown Error (HTTP 504)
> Octavia-work called Neutron API to create a port. And called nova-api 
> to attach the vip to amphora.
> Neutron created port successfully, but nova failed to attach the port 
> to instance.[2]
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver 
> [req-ab0e8d9b-664b-430f-8006-cad713b0c826 
> 401ba22da5f8427fbda5fce24600041b 8ce8b97f710f43d7af2b8f9b1e0463c8 - 
> default default] [instance: a210ec88-b554-487f-a125-30b5c7473060] 
> attaching network adapter failed.: libvirt.libvirtError: Unable to 
> read from monitor: Connection reset by peer
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] Traceback (most recent call last):
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", 
> line 2149, in attach_interface
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] guest.attach_device(cfg, 
> persistent=True, live=live)
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", 
> line 304, in attach_device
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] 
> self._domain.attachDeviceFlags(device_xml, flags=flags)
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", 
> line 190, in doit
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] result = 
> proxy_call(self._autowrap, f, *args, **kwargs)
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", 
> line 148, in proxy_call
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] rv = execute(f, *args, **kwargs)
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", 
> line 129, in execute
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] six.reraise(c, e, tb)
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] raise value
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", 
> line 83, in tworker
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] rv = meth(*args, **kwargs)
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] File 
> "/usr/lib64/python3.6/site-packages/libvirt.py", line 630, in 
> attachDeviceFlags
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] if ret == -1: raise libvirtError 
> ('virDomainAttachDeviceFlags() failed', dom=self)
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060] libvirt.libvirtError: Unable to 
> read from monitor: Connection reset by peer
> 2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance: 
> a210ec88-b554-487f-a125-30b5c7473060]
> Nova-compute called libvirt to attach the device. And libvirt also 
> failed to attach the device[3]
> 2021-01-31 08:20:23.884+0000: 86663: error : qemuMonitorIORead:491 : 
> Unable to read from monitor: Connection reset by peer
> 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:618 : Error 
> on monitor Unable to read from monitor: Connection reset by peer
> 2021-01-31 08:20:23.884+0000: 86663: info : virObjectRef:402 : 
> OBJECT_REF: obj=0x7f004c00b610
> 2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:649 : 
> Triggering error callback
> 2021-01-31 08:20:23.884+0000: 86663: debug : 
> qemuProcessHandleMonitorError:346 : Received error on 0x7f004c0095b0 
> 'instance-00000001'
> 2021-01-31 08:20:23.884+0000: 64768: debug : qemuMonitorSend:958 : 
> Send command resulted in error Unable to read from monitor: Connection 
> reset by peer
> I also tried to use kolla/ubuntu-source-nova-libvirt instead of 
> kolla/centos-source-nova-libvirt. and it worked as expected.
> I think the root cause is that libvirt failed to attach a network 
> device. but i don't know how to resolve this problem.
> could anyone help me?
> Thanks
> Wuchunyang
> [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/754285 
> <https://review.opendev.org/c/openstack/kolla-ansible/+/754285>
> [1]: 
> https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/octavia/octavia-worker.txt#1116 
> <https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/octavia/octavia-worker.txt#1116>
> [2]: 
> https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/nova/nova-compute.txt#2546 
> <https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/nova/nova-compute.txt#2546>
> [3]: 
> https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/libvirt/libvirtd.txt#194472 
> <https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/libvirt/libvirtd.txt#194472>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210204/8a92ef76/attachment.html>