Open Stack

Hi:

Recently, we added a CI task[0] for octavia in the kolla project. and we
tested octavia based on the ubuntu and centos systems.

The ubuntu system worked as expected but Centos did not work.

I tried to debug it and result is following:

Octavia can not create a load balancer on centos8. because octavia-worker
failed to plug a vip port on amphora vm.[1]

2021-01-31 08:20:12.065 22 ERROR octavia.compute.drivers.nova_driver [-]
Error attaching network None with ip None and port
26a39187-e95a-4131-91e7-24289e777f36 to amphora (compute_id:
a210ec88-b554-487f-a125-30b5c7473060) :
novaclient.exceptions.ClientException: Unknown Error (HTTP 504)
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging
amphora (compute_id: a210ec88-b554-487f-a125-30b5c7473060) into port
26a39187-e95a-4131-91e7-24289e777f36.:
octavia.common.exceptions.ComputeUnknownException: Unknown exception from
the compute driver: Unknown Error (HTTP 504).
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs Traceback (most
recent call last):
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs File
"/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/compute/drivers/nova_driver.py",
line 318, in attach_network_or_port
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs port_id=port_id)
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs File
"/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/api_versions.py",
line 393, in substitution
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs return
methods[-1].func(obj, *args, **kwargs)
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs File
"/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/v2/servers.py",
line 2063, in interface_attach
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs
obj_class=NetworkInterface)
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs File
"/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/base.py", line
363, in _create
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs resp, body =
self.api.client.post(url, body=body)
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs File
"/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/adapter.py",
line 401, in post
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs return
self.request(url, 'POST', **kwargs)
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs File
"/var/lib/kolla/venv/lib/python3.6/site-packages/novaclient/client.py",
line 78, in request
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs raise
exceptions.from_response(resp, body, url, method)
2021-01-31 08:20:12.066 22 ERROR
octavia.network.drivers.neutron.allowed_address_pairs
novaclient.exceptions.ClientException: Unknown Error (HTTP 504)

Octavia-work called Neutron API to create a port. And called nova-api to
attach the vip to amphora.

Neutron created port successfully, but nova failed to attach the port to
instance.[2]

2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver
[req-ab0e8d9b-664b-430f-8006-cad713b0c826 401ba22da5f8427fbda5fce24600041b
8ce8b97f710f43d7af2b8f9b1e0463c8 - default default] [instance:
a210ec88-b554-487f-a125-30b5c7473060] attaching network adapter failed.:
libvirt.libvirtError: Unable to read from monitor: Connection reset by peer
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] Traceback (most recent call last):
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 2149, in attach_interface
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] guest.attach_device(cfg,
persistent=True, live=live)
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py",
line 304, in attach_device
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060]
self._domain.attachDeviceFlags(device_xml, flags=flags)
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line
190, in doit
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] result = proxy_call(self._autowrap,
f, *args, **kwargs)
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line
148, in proxy_call
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] rv = execute(f, *args, **kwargs)
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line
129, in execute
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] six.reraise(c, e, tb)
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] raise value
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line
83, in tworker
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] rv = meth(*args, **kwargs)
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] File
"/usr/lib64/python3.6/site-packages/libvirt.py", line 630, in
attachDeviceFlags
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060] libvirt.libvirtError: Unable to read
from monitor: Connection reset by peer
2021-01-31 08:20:23.945 8 ERROR nova.virt.libvirt.driver [instance:
a210ec88-b554-487f-a125-30b5c7473060]



Nova-compute called libvirt to attach the device. And libvirt also failed
to attach the device[3]

2021-01-31 08:20:23.884+0000: 86663: error : qemuMonitorIORead:491 : Unable
to read from monitor: Connection reset by peer
2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:618 : Error on
monitor Unable to read from monitor: Connection reset by peer
2021-01-31 08:20:23.884+0000: 86663: info : virObjectRef:402 : OBJECT_REF:
obj=0x7f004c00b610
2021-01-31 08:20:23.884+0000: 86663: debug : qemuMonitorIO:649 : Triggering
error callback
2021-01-31 08:20:23.884+0000: 86663: debug :
qemuProcessHandleMonitorError:346 : Received error on 0x7f004c0095b0
'instance-00000001'
2021-01-31 08:20:23.884+0000: 64768: debug : qemuMonitorSend:958 : Send
command resulted in error Unable to read from monitor: Connection reset by
peer

I also tried to use kolla/ubuntu-source-nova-libvirt instead of
kolla/centos-source-nova-libvirt. and it worked as expected.

I think the root cause is that libvirt failed to attach a network device.
but i don't know how to resolve this problem.

could anyone help me?

Thanks

Wuchunyang


[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/754285
[1]:
https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/octavia/octavia-worker.txt#1116
[2]:
https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/nova/nova-compute.txt#2546
[3]:
https://zuul.opendev.org/t/openstack/build/e4b8c62c44a64b96bc287ba2ba2315f0/log/primary/logs/kolla/libvirt/libvirtd.txt#194472
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210204/23ee8fbe/attachment-0001.html>