April 2023 - openstack-discuss - lists.openstack.org

[qa][ptg] QA Bobcat vPTG summary
by Martin Kopec 06 Apr '23

06 Apr '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Antelope Retrospective == Good things: kudos to all who are still around and help with reviews, patches, bug fixes Bad things: unstable gate due to tox update, timeouts, job flakiness (mainly multinode job) == uWSGI is in the maintenance mode == This doesn't require an immediate action as it's still in the maintenance mode = any bugs should be fixed and until we need new features we don't have to rush into replacing it. We should have it on the watchlist though and potentially test some alternatives - I've put this as one of the priority items for this cycle [2]. == FIPS interop testing == We had a FIPS goal since the last PTG [3]. The patches [4] and [5] were to have FIPS on ubuntu and these [6] and [7] are to add FIPS on a centos job. The next goal is to have a FIPS job on Rocky linux as it's more stable than centos. == Retiring Patrole == We are going to retire Patrole project. 2 main facts that speak in favor of retiring: Fact 1: plugins have enough RBAC tests (and are writing more to test SRBAC) - therefore patrole is not needed, the testing coverage got by the plugins is enough. Fact 2: patrole gates have been broken for a long time and no one was affected by that. == Additional PyPi maintainer cleanup == Only 2 projects from QA left with additional maintainers: * openstack/os-performance-tools - additional maintainer SpamapS (Clint Byrum) * openstack/patrole - additional maintainer DavidPurcell If you know any of them, please, tell them to reach out to me. I've sent emails to them, but maybe the email addresses I found aren't checked regularly anymore. == Tempest/Devstack gate stability == Several precautions have been done such as reduce mysql memory [8] or reduce the number of tests executed per job, increase of a timeout etc [9]. However, those precautions are mainly workarounds. The important note is that the instability was only highlighted by our testing, not caused by it. Many problems are inefficiencies in openstack software, e.g.: * openstack-client issues a new token for every operation * python startup time - entrypoint lookup * privesp processes in aggregate use more memory than nova (every service runs their own copy) * inefficient ansible code, etc Any volunteers who would like to first analyse the situation, find which of those inefficiencies are the biggest culprits and how big effect they have on the system? [1] https://etherpad.opendev.org/p/qa-bobcat-ptg [2] https://etherpad.opendev.org/p/qa-bobcat-priority [3] https://etherpad.opendev.org/p/antelope-ptg-interop [4] https://review.opendev.org/c/zuul/zuul-jobs/+/866881 [5] https://review.opendev.org/c/zuul/zuul-jobs/+/873893 [6] https://review.opendev.org/c/openstack/devstack/+/871606 [7] https://review.opendev.org/c/openstack/tempest/+/873697 [8] https://review.opendev.org/c/openstack/devstack/+/873646 [9] https://review.opendev.org/q/topic:bug%252F2004780 Regards, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[cinder] 2023.2 Bobcat PTG Summary
by Rajat Dhasmana 06 Apr '23

06 Apr '23

Hello, Please find the summary of 2023.2 Bobcat PTG in the wiki page[1]. It includes recordings, summary and action items for each topic. More details can be found in the PTG etherpad[2]. I would recommend everyone to please look for the action items assigned to you to plan it according to the cycle deadlines. [1] https://wiki.openstack.org/wiki/CinderBobcatPTGSummary [2] https://etherpad.opendev.org/p/bobcat-ptg-cinder Thanks Rajat Dhasmana

1 0

[cinder]multi backend SAN and Ceph
by Nguyễn Hữu Khôi 06 Apr '23

06 Apr '23

Hello. I have a question that could we use both SAN and Ceph for multi backend? If yes can we change volume type from SAN to Ceph and otherwise? Thanks.

3 3

[Glance] Weekly Meeting Cancelled
by Pranali Deore 06 Apr '23

06 Apr '23

Hi All, Canceling today's glance weekly meeting, I'm on sick leave today. See you next week ! Thanks, Pranali D

1 0

[cinder]cannot lauch instance with cinder volume
by Nguyễn Hữu Khôi 06 Apr '23

06 Apr '23

Hello guys. I am using kolla-ansible yoga to deploy an openstack cluster, but when I launch an instance with volume, it has errors. but I can create volume first and boot from it. I am using Powerstore 5000Tb. Is it a bug? 672 ERROR oslo_messaging.rpc.server [req-1a822ccd-1d3a-4f9a-a453-22d04f07f816 47cefefdadbe4bb584d8cf983de71eb5 708d7af8178143aab263d3942a124ec8 - - -] Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/driver.py", line 1045, in _attach_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server conn = self.initialize_connection(volume, properties) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/driver.py", line 153, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return self.adapter.initialize_connection(volume, connector, **kwargs) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 272, in initialize_connection 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server connection_properties = self._connect_volume(volume, connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 613, in _connect_volume 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server chap_credentials, volume_lun = self._create_host_and_attach( 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 602, in _create_host_and_attach 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host, chap_credentials = self._create_host_if_not_exist(connector) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/decorator.py", line 232, in fun 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/coordination.py", line 195, in _synchronized 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server return f(*a, **k) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/adapter.py", line 446, in _create_host_if_not_exist 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server host = self.client.create_host(host_name, ports) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/drivers/dell_emc/powerstore/client.py", line 302, in create_host 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg) 2023-04-05 21:24:30.886 672 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Failed to create PowerStore host . Nguyen Huu Khoi

1 1

[kolla-ansible][yoga][neutron][neutron-dhcp]ERRORS.... Need to understand
by Franck VEDEL 06 Apr '23

06 Apr '23

Hello. I come back to you to try to understand my problem. My installation is based on kolla-ansible, Yoga, Centos-Stream. For some time, without understanding why, neutron_dhcp no longer works on external networks and the networks of my students' projects. This generates many error messages, which are difficult to understand. Could this be a bug in this version? Before, it worked very well, but I have the impression that when the number of network increases (i’m not sure), from a moment, dhcp does not work anymore. What is the way to find the problem? Here are examples of errors that won't stop even if no one creates more instances of them 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap73cd30e2-35, namespace qdhcp-395f1edb-de6d-4dd8-b31b-69a53780c012. 2023-04-04 20:36:52.870 7 ERROR neutron.agent.dhcp.agent 2023-04-04 20:36:52.871 7 INFO neutron.agent.dhcp.agent [-] Finished network 395f1edb-de6d-4dd8-b31b-69a53780c012 dhcp configuration 2023-04-04 20:36:52.872 7 INFO neutron.agent.dhcp.agent [-] Starting network 8db375e0-28d2-4379-9330-397dc9b4f2b1 dhcp configuration 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 8db375e0-28d2-4379-9330-397dc9b4f2b1.: neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 218, in call_driver 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 274, in enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 717, in wait_until_true 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent while not predicate(): 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 286, in _enable 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1779, in setup 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent namespace=network.namespace) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 152, in init_l3 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device.addr.add(ip_cidr) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 537, in add 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent add_broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 823, in add_ip_address 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent r_call_timeout) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent neutron.privileged.agent.linux.ip_lib.InvalidArgument: Invalid parameter/value used on interface tap1c220d25-b9, namespace qdhcp-8db375e0-28d2-4379-9330-397dc9b4f2b1. 2023-04-04 20:36:53.587 7 ERROR neutron.agent.dhcp.agent Thanks a lot Franck

2 4

[ironic][ptg] 2023.2 Work Items
by Jay Faulkner 06 Apr '23

06 Apr '23

Hey all, As usual, I documented items we talked about working on in Bobcat in a spec. Please go have a look and review it: https://review.opendev.org/c/openstack/ironic-specs/+/879381 This will hopefully be published very soon at https://specs.openstack.org/openstack/ironic-specs/#themes as our 2023.2 work items. Thanks everyone for participating in our vPTG this cycle! -- Jay Faulkner Ironic PTL

1 0

[tripleo] hold rechecks content provider gate blocker
by Marios Andreou 05 Apr '23

05 Apr '23

Hello tripleo we have a gate blocker at [1] hitting the centos-9-content-provider jobs. The tripleo CI team is tracking this - please hold rechecks until it we have a resolution or workaround regards, marios [1] https://bugs.launchpad.net/tripleo/+bug/2015309

1 1

[kolla] weekly meeting cancelled today
by Michał Nasiadka 05 Apr '23

05 Apr '23

Hello Koalas, Weekly meeting cancelled today. Please note that our weekly meeting time for next week and onwards has changed to 13:00 UTC due to DST changes. Best regards, Michal -- Michał Nasiadka mnasiadka(a)gmail.com

1 0

[neutron][OpenStack-ansible] Performance issues with trunk ports
by John Bartelme 05 Apr '23

05 Apr '23

Hello, I'm currently experiencing some pretty severe performance issues with my openstack-ansible deployed cluster(yoga) while deploying trunk ports and I'm looking for some help determining what might be the cause of this poor performance. In my simplest case I'm deploying 2 servers each with one trunk port each. The first trunk has 2 subports and the second 6 subports. Both servers also have 3 other regular ports. When deploying the first trunk port its subports are often provisioned quickly and the second trunk port takes anywhere from 30 seconds to 18 minutes. This happens even when I isolate neutron-server to a single physical machine with 44(88 threads) and 256GB ram. Further diagnosis has shown me some things i didn't quite understand. My deployment with OpenStack-ansible deploys neutron-server with 16 uWSGI processes and neutron-rpc-server with 16 rpc workers. However the way that the trunk RPC server is implemented it is only run on the parent RPC thread and instead runs in all of the uWSGI processes as well. This means that most of my trunk RPC calls are being handled by the uWSGI instead of the RPC workers. When the parent RPC thread handles the trunk port creation calls I constantly see creation times of 1-1.5 seconds. I've isolated it so that this thread does all of the trunk RPC calls and this works quite well but this doesn't seem ideal. What could be causing such poor performance in the uWSGI side of the house? I'm having a really hard time getting a good feeling for what might be slowing it down so much. I'm wondering if it could be green thread preemption but I really don't know. I've tried setting 'enable-threads' false for uWSGI but I don't think that is improving performance. Putting the profiled decorator on update_subport_bindings shows different places taking longer every time, but in general a lot of time(tottime, i.e. not subfunction time) spent in webob/dec.py(__call__), paste/urlmap.py(__call__), webob/request.py(call_application),webob/request.py(send). What else can I do to try and look for why this is taking so long? As a side question it seems counterintuitive that the uWSGI handles most of the trunk RPC calls and not the RPC workers? A couple other notes about my environment that could indicate my challenges: I had to disable rabbitmq heartbeats for neutron as they kept not getting sent reliably and connections were terminated. I tried with heartbeat_in_pthread both true and false but still had issues. It looks like nova also sometimes experiences this but not near as often. I was overzealous with my vxlan ranges in my first configuration and gave it a range of 10,000,000 not realizing that would create that many rows in the database. Looking into that I saw that pymysql in my cluster takes 3.5 minutes to retrieve those rows. mysql CLI only takes 4 seconds. Perhaps that is just the overhead of pymysql? I've greatly scaled down the vxlan range now. I'm provisioning the 2 servers with a heat template that contains around 200 custom resources. For 198 of the resources they are set to conditionally not create any OpenStack native resources. Deploying this template of mostly no-op resources still takes about 3 minutes. Horizon works but almost every page load take a few seconds to load. I'm not sure if that is normal or not. Thanks for any help anyone can provide. john

4 8