May 2024 - openstack-discuss - lists.openstack.org

[openstack][kolla-ansible]about group node config overwrite
by Nguyễn Hữu Khôi 30 Oct '25

30 Oct '25

Hello Koller. I want to ask if we can overwrite config for a group of nodes. For example, compute01 >> compute10 will have the same nova configuration compute11 >> compute20 will have the same nova configuration. It will be very nice if we support this way. Thank you. Regards Nguyen Huu Khoi

4 7

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

[oslo] python-consul2 is unmaintained
by Takashi Kajinami 21 Apr '25

21 Apr '25

Hello, I recently noticed that python-consul2 library[1], which is used by the consul coordination driver in tooz, hasn't been updated for 3+ years. Looking at issues in the project, there is an indication that the library is not compatible with ACL API in consul >= 1.11[2]. There is another fork created[3] and this includes fixes to support new API endpoints. Although the driver in tooz does not directly use the feature, I think it's better to discuss our plan to mitigate risks caused by the unmaintained library. As far as I can think of, there are three options now 1. Deprecate consul driver and remove it 2. Keep using python-consul2 now. At least basic functionality is proven to work by our functional tests (Note that the tests do not cover all configuration patterns) 3. Replace python-consul2 by py-consul Before we start the overall discussions I'd like to ask a few questions to the wider audience, as our inputs. - Is anyone using the consul coordination driver now ? - Does anyone have experience with python-consul2 and/or py-consul ? Note that this discussion affects the http messaging driver[4] currently proposed, because it also requires client library to communicate with Consul. Actually I noticed the situation when I noticed that the requirement check job is failing because the library is not part of global requirements list... Thank you, Takashi Kajinami [1] https://github.com/poppyred/python-consul2 [2] https://github.com/poppyred/python-consul2/issues/42 [3] https://github.com/criteo-forks/py-consul [4] https://review.opendev.org/c/openstack/oslo.messaging/+/912499 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 3

[octavia] Purging ERROR'ed amphora and load balancers stuck in transitional states
by Paul Browne 11 Apr '25

11 Apr '25

Hello Octavia users, For a while I've been trying to find a systematic way to remove Octavia LBs+Amphora that have become stuck in ERROR or transitional PENDING_* states ; $ openstack loadbalancer amphora list -f value | grep ERROR a2323f14-3aaa-418a-8249-111aaa9c21fe 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR MASTER 10.8.0.242 192.168.3.247 e5b236ba-e7ee-4ed7-9f58-57ce7a408489 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR BACKUP 10.8.0.190 192.168.3.247 6b556f28-93c9-49dd-b6ee-4379288e7957 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR MASTER 10.8.0.39 192.168.3.126 c669db5d-8686-4d5c-9e95-e02030b34301 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR BACKUP 10.8.0.174 192.168.3.126 $ openstack loadbalancer list -f value | grep -vi active 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe k8s-clusterapi-cluster-default-ci-6386871107-kube-upgrade-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.247 PENDING_UPDATE amphora d5e402fe-2c4b-49af-a700-532cb408cee5 k8s-clusterapi-cluster-default-ci-6386871107-latest-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.126 PENDING_DELETE amphora These resources are marked immutable and so cannot be failed over or deleted; $ openstack loadbalancer amphora failover a2323f14-3aaa-418a-8249-111aaa9c21fe Load Balancer 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe is immutable and cannot be updated. (HTTP 409) (Request-ID: req-6e66c4e8-c3d3-4549-a03c-367017c8c8b3) $ openstack loadbalancer failover 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe Invalid state PENDING_UPDATE of loadbalancer resource 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe (HTTP 409) (Request-ID: req-84b44212-e7a8-4101-a16f-18c774c0577e) The backing Nova instances for these Amphora do seem to exist and be in good working order. Is there any API way to purge these out of Octavia's service state, or would (very careful) DB hackery be required here? Many thanks, Paul Browne ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 6

[tc][all][monasca ] Retiring the Monasca project
by Ghanshyam Mann 16 Feb '25

16 Feb '25

Hello everyone, As you might know, the Monasca project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project. I'm sending this email to propose the Monasca retirement and based on the response, I will proceed next. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 -gmann

6 17

[ironic]About the OS supported by Diskimage-builder
by Junya Noguchi (Fujitsu) 03 Feb '25

03 Feb '25

To whom it may concern. My name is Junya Noguchi. I have some questions about the supported OS for Diskimage-builder operation. Please respond. References https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html Does it mean that the OS that says "work but receive only limited testing and support" works but is not supported? References https://docs.openstack.org/diskimage-builder/latest/ This document describes RHEL and other operating systems that are not supported in the previous question. Are these operating systems supported? Also, are there any plans to support more operating systems in the future? Best regards, Junya Noguchi

5 7

[glance][ops] proposal to remove the glance-scrubber
by Brian Rosmaita 20 Jan '25

20 Jan '25

Hello Glance users, The Glance project team wants to determine whether anyone is using the glance-scrubber utility [0]. If not, we are thinking of deprecating it this cycle to be removed in the appropriate future cycle. If you have not previously heard of the glance-scrubber, that's most likely because it's only useful for particular deployment configurations of which yours is not one. See the doc [0] for details if you are curious. Our suspicion is that the glance-scrubber may have outlived its usefulness for modern OpenStack configurations. (The last time I remember it being discussed at a PTG was during the snowstorm in Dublin, which I believe was the Rocky development cycle). On the other hand, the team also has some ideas for improving the scrubber, but there's no point doing that if it's not actually being used in the first place. So if removal of the glance-scrubber would negatively affect you, please respond to this email so we can get some data points. Thanks! [0] https://docs.openstack.org/glance/latest/cli/glancescrubber.html

2 2

[cinder][kubernetes] pvc stuck in reserved state
by Eugen Block 10 Jan '25

10 Jan '25

Hi, we have a couple of k8s clusters in our openstack environment (Victoria) and are facing some problems when pods get moved to different worker nodes. The PVCs get stuck in "reserved" state, and they only get attached after resetting the volumes to "available" state. First I thought it was due to the missing multiattach volume type which seems to work fine in our customer's cluster. I also found the "duplicate connector" error messages in the nova-compute.log: > Unable to update attachment.(Invalid volume: duplicate connectors > detected on volume But after adding the multiattach volume type and retyping the affected volumes, the PVCs still get stuck in "reserved" state while I don't see any error messages on the hypervisors anymore. So it seems that something has changed/improved, but now I don't have a trace where to look next. Could this be a cinder-csi driver issue or something? Our cinder backend is ceph (Pacific). Not sure which information could help so please ask for anything I can provide. Any pointers are highly appreciated! Thanks! Eugen

1 2

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 24 Dec '24

24 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[horizon][ceph] object storage container storage policy missing
by Satish Patel 25 Nov '24

25 Nov '24

Folks, Openstack Release: Zed Recently I have integrated ceph RGW service with openstack/keystone and testing with command line tool "openstack container create foo1" and it works but when I go to Horizon UI I am seeing it doesn't have storage policy dropdown list and because of that I am not able to create container or upload file from Horizon GUI https://ibb.co/KhYW6Rd It works with commands like but not horizon. $ openstack container create foo1 +---------------------------------------+-----------+------------------------------------------------+ | account | container | x-trans-id | +---------------------------------------+-----------+------------------------------------------------+ | AUTH_d4d82849885e4126aef402c408ad2c5c | foo1 | tx00000c057454841b13497-0065b558b4-42ff2eb-ahm | +---------------------------------------+-----------+------------------------------------------------+ $ openstack container list +------+ | Name | +------+ | foo1 | +------+

2 2

[ironic] Having issue with deploying ISO image to node
by James Leong 18 Nov '24

18 Nov '24

Hi all, I am trying to play around with ironic, and I am having an issue with deploying an Ubuntu 20.04 ISO image onto my bare metal node. Since I am quite new to bare metal, I am not entirely sure what went wrong as I couldn't find any error messages shown on my log file (ironic-conductor, ironic-api, ironic_neutron_agent, and nova-compute-ironic). This might be my guess; the ironic inspection is hitting timeout, which is causing the deployment to timeout. However, I have no idea why the inspection is failing at timeout. Any help would be appreciated. Best, James

6 7

[nova][libvirtd] internal error: unable to execute QEMU command 'query-named-block-nodes'
by Satish Patel 22 Oct '24

22 Oct '24

Folks, I have a ceph cluster and recently I configured rbd-mirror to replicate all data to remove ceph cluster for disaster recovery. Yesterday for POC we did a hard cutover on ceph and point openstack to the new cluster. All other vms came back up fine but 2 vms stuck in this error state in libvirt logs 2024-01-31 22:44:37.591+0000: 474597: error : qemuMonitorJSONCheckErrorFull:399 : internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied If it's a wider issue than it should impact all the vms but why only two VM stuck and not starting and libvirt giving me this error in logs nova-compute logs also showing same error 2024-01-31 22:44:40.925 7 INFO nova.compute.manager [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] [instance: 2de0f880-77c7-4d2c-9e01-898c57ad3693] Successfully reverted task state from powering-on on failure for instance. 2024-01-31 22:44:40.944 7 ERROR oslo_messaging.rpc.server [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied

3 8

[kuryr][tc] Final call for Kuryr PTL/DPL liaisons
by Ghanshyam Mann 15 Oct '24

15 Oct '24

Hello Everyone, Kuryr project ends up being a leaderless project for the 2024.2 Dalmatian cycle[1]. There is no volunteer to serve as PTL or DPL liaisons yet. As the next step, I have proposed to mark this project Inactive: - https://review.opendev.org/c/openstack/governance/+/916945 If you would like to serve as a leader and help maintain this project, this is the right time to step up by replying to this email or commenting in gerrit change 916945. [1] https://etherpad.opendev.org/p/2024.2-leaderless#L99 Ref: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… -gmann

4 10

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[openstack][kolla-ansible]about upgrading openstack
by Nguyễn Hữu Khôi 04 Aug '24

04 Aug '24

Hello. I read this doc about upgrading openstack https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html But I would like to know if We install a new kolla-ansible(version to upgrade) on the new deployment host, apply changes with password.yml, global.yml and multinode and keep old /etc/kolla/config. Finally, we use kolla-ansible deploy instead of kolla-ansible upgrade. Are these steps ok? Pls correct me if I am wrong. Thank you. Regards Nguyen Huu Khoi

3 8

[kolla-ansible][letsencrypt] containers are running but not getting certificates.
by Jonathan Proulx 03 Jul '24

03 Jul '24

Hi All, I'm trying to do a test multinode deploy using 2023.2 I have letsencrypt_webserver and letsencrypt_lego contsainers running and I'm seeing random traffic in the /var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly confident they're plumbed through to the public internet properly, but I don't seem to be getting certificates. how can I trigger a renewal attempt so I can maybe see what I've screwed up? Thanks, -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL

3 6

[barbican][openstack-helm][tacker] Request to Include Barbican Installation Guide in OpenStack-Helm Documentation
by JUSO WATARU(十楚　航) 21 Jun '24

21 Jun '24

Hello, We are presently in the process of enhancing the official OpenStack-Helm documentation by including a procedural guide for deploying Tacker using helmchart. Through discussions with members of the openstack-helm community, it has been emphasized that Barbican is an indispensable component for Tacker. Hence, it would be highly beneficial for users to have comprehensive instructions for installing Barbican with helmchart included in the official OpenStack-Helm documentation [1]. Consequently, we suggest that the Barbican community integrate the installation guide for the Barbican component into the official OpenStack-Helm documentation. This addition is poised to significantly enhance the clarity and accessibility of deployment procedures for users. We eagerly welcome your opinions and feedback on this proposal, as we aim to ensure that the documentation for openstack deployment with helmchart meets the needs of our community. [1] https://docs.openstack.org/openstack-helm/latest/install/openstack.html Regards, Wataru

2 2

Issue in Stack.sh
by Lohit S 14 Jun '24

14 Jun '24

How to fix this issue while stack.sh is executed +functions-common:_run_under_systemd:1655 sudo systemctl start devstack(a)q-svc.service +functions-common:run_process:1687 time_stop run_process +functions-common:time_stop:2421 local name +functions-common:time_stop:2422 local end_time +functions-common:time_stop:2423 local elapsed_time +functions-common:time_stop:2424 local total +functions-common:time_stop:2425 local start_time +functions-common:time_stop:2427 name=run_process +functions-common:time_stop:2428 start_time=1716869530330 +functions-common:time_stop:2430 [[ -z 1716869530330 ]] ++functions-common:time_stop:2433 date +%s%3N +functions-common:time_stop:2433 end_time=1716869534546 +functions-common:time_stop:2434 elapsed_time=4216 +functions-common:time_stop:2435 total=16163 +functions-common:time_stop:2437 _TIME_START[$name]= +functions-common:time_stop:2438 _TIME_TOTAL[$name]=20379 +lib/neutron:start_neutron_service_and_check:640 neutron_url= http://192.168.122.150:9696/ +lib/neutron:start_neutron_service_and_check:642 is_service_enabled tls-proxy +functions-common:is_service_enabled:2077 return 1 +lib/neutron:start_neutron_service_and_check:646 '[' '!' -z networking ']' +lib/neutron:start_neutron_service_and_check:647 neutron_url= http://192.168.122.150:9696/networking +lib/neutron:start_neutron_service_and_check:649 echo 'Waiting for Neutron to start...' Waiting for Neutron to start... +lib/neutron:start_neutron_service_and_check:651 local 'testcmd=wget --no-proxy -q -O- http://192.168.122.150:9696/networking' +lib/neutron:start_neutron_service_and_check:652 test_with_retry 'wget --no-proxy -q -O- http://192.168.122.150:9696/networking' 'Neutron did not start' 60 +functions-common:test_with_retry:2359 local 'testcmd=wget --no-proxy -q -O- http://192.168.122.150:9696/networking' +functions-common:test_with_retry:2360 local 'failmsg=Neutron did not start' +functions-common:test_with_retry:2361 local until=60 +functions-common:test_with_retry:2362 local sleep=0.5 +functions-common:test_with_retry:2364 time_start test_with_retry +functions-common:time_start:2407 local name=test_with_retry +functions-common:time_start:2408 local start_time= +functions-common:time_start:2409 [[ -n '' ]] ++functions-common:time_start:2412 date +%s%3N +functions-common:time_start:2412 _TIME_START[$name]=1716869535329 +functions-common:test_with_retry:2365 timeout 60 sh -c 'while ! wget --no-proxy -q -O- http://192.168.122.150:9696/networking; do sleep 0.5; done' +functions-common:test_with_retry:2366 die 2366 'Neutron did not start' +functions-common:die:285 local exitcode=0 [Call Trace] ./stack.sh:1322:start_neutron_service_and_check /home/lohit22/opensource/openstack/devstack/Org/devstack/lib/neutron:652:test_with_retry /home/lohit22/opensource/openstack/devstack/Org/devstack/functions-common:2366:die [ERROR] /home/lohit22/opensource/openstack/devstack/Org/devstack/functions-common:2366 Neutron did not start exit_trap: cleaning up child processes Error on exit

2 2

[openstack charm][tc] openstack-charms still need PTL appointment
by Ghanshyam Mann 11 Jun '24

11 Jun '24

Hello Felipe, openstack-charm team, I tried to reach out to you on IRC but could not connect. This is regarding openstack-charm PTL appointment for the 2024.2 cycle. I am not sure if you noticed or not but there are some comments for you to fix on your PTL appointment. Please check the below Gerrit change and fix the same or let us know if there is a change in the plan. - https://review.opendev.org/c/openstack/governance/+/914254 -gmann

2 3

[swift] clean up orphan segment parts
by Rafael Tavares Oliveira 07 Jun '24

07 Jun '24

Hey there! We've been using swift and we came across this bug https://bugs.launchpad.net/swift/+bug/1813202. We decided to work it out and have an ongoing patch to solve this issue alongside ronaldo.maia(a)luizalabs.com, we would like to share and get some feedback before we submit it. To solve this, we took inspiration in obj/expirer.py: In S3 controllers we register task objects in an internal account, each task object points to an orphaned slo object, which can be created in two situations, when a multipart-upload is not completed or when a multipart object is overwritten. These tasks will be checked by the daemon (a new one we made called "s3-slo-expirer") after some time and deleted from the base accounts. Rafael

2 2

[all][tc] Eventlet migration: Call to action
by Jay Faulkner 07 Jun '24

07 Jun '24

Hi all, As you may have read in Goutham's TC summary, the existing eventlet goal[0] is unlikely to gain consensus to merge. This appears to be due to the difference in needs between OpenStack projects, and belief that it's unlikely a single solution would serve them all. Our current situation is: 1) Eventlet usage must be discontinued. The temporary maintainers are just that -- temporary -- and we should not expect eventlet to continue to be a valid path moving forward. 2) We have basic agreement that shared tooling, such as oslo libraries, should not dictate a specific threading model. In lieu of specific technical leadership and guidance from the TC, this leaves it to individual projects to decide what threading model to use instead of eventlet monkey patching. The unmerged spec[0] may be a good starting point if you're wondering where to start. Importantly: do not let this work wait! It is a significant change and will take a long time to complete. Thanks, Jay Faulkner 0: https://review.opendev.org/c/openstack/governance/+/902585

6 7

Antelope Barbican/Keystone: failure to delete secrets (Keystone TrustNotFound)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I've been validating Yoga -> Antelope on our preprod cloud. It was working successfully but "suddenly" (i.e. without a kwown/identified change!) when deleting Magnum clusters, the deletion of Barbican secrets (after stack deletion) fail. In the Magnum log, I find the following error: ---- Failed to delete trust: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a4400250-e8d5-46ec-a28a-dcb97a682512) (traceback skipped for readability) 2024-05-29 10:43:09.750 2078 INFO magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17… from Barbican. 2024-05-29 10:43:09.750 2078 INFO barbicanclient.base [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Calculated Containers uuid ref: containers/edd8487a-6cf1-4f4c-9f53-173dfae7a8b2 2024-05-29 10:43:09.760 2078 ERROR barbicanclient.client [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] 4xx Client error: Not Found: Secrets container not found. 2024-05-29 10:43:09.761 2078 ERROR magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Error recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17…: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. ---- In Keystone log, I find with the same timestamp the following error that seems related to the one above: ---- 2024-05-29 10:43:09.690 1713 WARNING keystone.api._shared.authentication [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Could not find trust: cf4b1b6082594025a1ff9e48df383566.: keystone.exception.TrustNotFound: Could not find trust: cf4b1b6082594025a1ff9e48df383566. 2024-05-29 10:43:09.693 1713 WARNING keystone.server.flask.application [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Authorization failed. The request you have made requires authentication. from 134.158.77.23: keystone.exception.Unauthorized: The request you have made requires authentication. 2024-05-29 10:43:09.746 1723 ERROR keystone.server.flask.application [None req-8c76f304-c2a3-418c-8372-02c706ed356e 44d8cb3192b3485790ababe29991491a - 5b67a8641dd041089e018ceea9884eac - 5b67a8641dd041089e018ceea9884eac -] Could not find user: 527ec32509d849e8a49b41c3dc4c4c32.: keystone.exception.UserNotFound: Could not find user: 527ec32509d849e8a49b41c3dc4c4c32. ---- It is not clear for me if the ERROR is releated to the 2 warnings as the request is not the same but it seems to happen every time this way. In Barbican logs, there is not much things related except a 401 status for the request corresponding ot the TrustNotFound. I'm looking for advices on how to troubleshoot this. In particular, when the trust should be created and whether I should find somewhere a matching error at the time of the trust creation. It seems the error is only affecting deletion but I may be wrong. for the record, when starting Magnum there is an error complaining about /etc/magnum/keystone_auth_default_policy.json being not found but ignored it so far. And in Keystone an error seems to occur at Magnum cluster creation time: --- 2024-05-29 10:17:17.530 1711 WARNING py.warnings [None req-f484cc35-3e54-4d7f-8688-e6442c90344f a22b1199c74f4a819a49f3ac037a5e44 1dd67113c8bb4ef0a31dfca53335ffbd - - defaul t default] /usr/lib/python3.9/site-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_services": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is re quired ---- Thanks in advance for any hint. Best regards, Michel

2 3