- openstack-discuss - lists.openstack.org

[devstack][neutron] DNS Resolution Issue After Server Reboot with DevStack
by xuq.fnstxz＠fujitsu.com 27 Nov '24

27 Nov '24

Hi everyone, I am reaching out to seek your assistance regarding an issue I am experiencing with my DevStack installation. After successfully installing DevStack stable/2024.2 on ubuntu 22.04, I configured the network and set up DNS for private subnet. stack@controller:~$ openstack subnet list +--------------------------------------+---------------------+--------------------------------------+---------------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------------------+--------------------------------------+---------------------+ | 2b6411d5-9c19-4a45-bd47-58dc748f5e00 | public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 192.168.1.0/24 | | 69629170-2e0e-49c0-b741-61560558001c | ipv6-public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 2001:db8::/64 | | c74bf062-b67a-45cb-8a50-73583498dfcd | ipv6-private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | fd3a:2f86:ca98::/64 | | f2e10d3d-f785-4cf9-8804-1969269623c2 | private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | 10.0.0.0/26 | +--------------------------------------+---------------------+--------------------------------------+---------------------+ stack@controller:~$ openstack subnet set --dns-nameserver 10.12.255.254 --dns-nameserver 192.168.1.95 f2e10d3d-f785-4cf9-8804-1969269623c2 10.12.255.254 is the real DNS server address and 192.168.1.95 is my server IP address. Initially, when I launched a VM instance, it was able to resolve the proxy and connect to the external network without any issues. jenkins@np0000000531:~$ curl opendev.org <a href="https://opendev.org/">Temporary Redirect</a>. However, after rebooting the server and launching the instance again, the instance is unable to resolve the proxy. jenkins@np0000000531:~$ curl opendev.org curl: (5) Could not resolve proxy: my.proxy.com I have verified that the systemd-resolved service is in an active state, but the logs show a large number of entries with the following message: Nov 13 11:53:51 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.95. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 10.12.255.254. Nov 13 11:54:11 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.95. Nov 13 11:54:21 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. I have checked the network and routing table of the VM instance, as well as the bridge settings on the server. Everything appears to be the same as it was before the reboot. Could you please help me identify the cause of this issue and suggest any potential solutions? If you need any additional information, please let me know, and I will be happy to provide it. Thank you for your time and assistance. Best regards, Inori.

2 3

[all] 2025 Open Infrastructure Foundation Individual Director nominations are open
by Jeremy Stanley 27 Nov '24

27 Nov '24

To those who aren't following the foundation mailing list, please be aware that the nomination period for the 2025 Open Infrastructure Foundation Board of Director Individual Member elections is open, and will close at 23:59 UTC on December 13. If you've considered taking on some additional leadership responsibility, this could be your opportunity! See Wes's post for more information: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/th… -- Jeremy Stanley

1 0

[kolla] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 26 Nov '24

26 Nov '24

Hi Kolla Community, I would like to understand the current guidelines for adding or removing a controller node from an OpenStack cluster. With the introduction of quorum queues, we’ve observed that adding a new controller node causes a loss of quorum within the cluster. Consequently, during the reinstallation of a controller node, Masakari disables all the nova-compute services. Is this behaviour expected with quorum queues, or could this be addressed differently? Additionally, I’ve noticed that as soon as the Kolla containers are restarted in the cluster, clients are unable to connect, causing the cluster to go down. Could you please share insights or recommendations on how to handle this scenario effectively? Looking forward to hearing your thoughts. Thanks, Ishan Cisco Systems

2 2

[keystone] Weekly meeting canceled 27-Nov-2024
by Dave Wilde 26 Nov '24

26 Nov '24

Hello, Due to the Thanksgiving holiday in the US I'm canceling the weekly meeting and reviewathon this week. Please reach out here or on IRC if you need anything. Thanks! /Dave

1 0

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[openstack helm] [Ansible] Openstack-helm Ansible inventory.yaml missing some value
by daniel890723＠gmail.com 26 Nov '24

26 Nov '24

Hello today i'm testing installation with completely new enviroment i was following the guide from Openstack-helm document and install k8s cluster using ansible and deploy-env playbook but i encounter this error when setup ssh cluster TASK [deploy-env : Generate ssh key pair] ****************************************************************************************************************** skipping: [node-1] fatal: [primary]: FAILED! => {"msg": "Failed to change ownership of the temporary files Ansible (via chmod nor setfacl) needs to create despite connecting as a privileged user. Unprivileged become user would be unable to read the file."} and the log with -vvv output: debug2: mux_client_hello_exchange: master version 4 debug3: mux_client_forwards: request forwardings: 0 local, 0 remote debug3: mux_client_request_session: entering debug3: mux_client_request_alive: entering debug3: mux_client_request_alive: done pid = 114038 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 chown: invalid user: ‘zuul’ debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 And by checking the playbook we can see the playbook is using "{{ client_ssh_user }}" on primary node and "{{ cluster_ssh_user }}" with cluster node which is not defined in inventory.yaml Causing it using default user zuul by adding client_ssh_user & cluster_ssh_user in ansible inventory it works normally Is this meant to be like this? or the document missing some variables in inventory.yaml

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-18)
by Goutham Pacha Ravi 26 Nov '24

26 Nov '24

Hello Stackers, We're 18 weeks away from the coordinated release for 2025.1 ("Epoxy") [1]. We're beginning a period of low activity due to year-end holidays across the world. Last week, the TC didn't merge any new governance proposals; however, several are currently in progress. === Weekly Meeting === The TC met in OFTC's #openstack-tc IRC channel on 2024-11-19 [2]. We discussed a series of global requirements upper constraint bumps that were committed and the ensuing CI flakiness. We appreciated the effort to proactively update dependencies to their latest versions earlier in the release cycle so we can have more time to test and stabilize OpenStack software. Sometimes, these failures may interrupt other project priorities. The best way we can help is if project teams identify a relatively quick-running and stable integration job to run against changes to the global requirements repository. The TC and OpenDev administrators worked with Launchpad administrators to regain control of trackers belonging to the Watcher project team. It was noted that Launchpad trackers for any project should not be owned/administered by individuals. Instead, there must be a Launchpad team driving all trackers associated with an OpenStack initiative, and "openstack-admins" must own this team. This approach ensures the sustainability of project teams and protects projects from the impact of changing individual priorities. We also briefly discussed the "eventlet-removal" goal proposal. The goal document is being rewritten [3] to simplify what is being done and the timelines we're aiming for. I'd love for the community to take a look at this change and review it. The next meeting of the OpenStack Technical Committee is today, 2024-11-25, at 1800 UTC in OFTC's #openstack-tc channel. The agenda for this meeting is in our meeting wiki [4]. I hope you're able to join us. === Governance Proposals === ==== Open for Review ==== Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 ==== Merged ==== Fix doc job for pillow 11.0.0 | https://review.opendev.org/c/openstack/governance/+/935967 Add Ubuntu Noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meetings: Join us at our weekly meeting. The Technical Committee meets every week on Tuesdays at 1800 UTC [4]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting IRC Log 2024-11-19: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.log.html [3] Eventlet removal Community goal: https://review.opendev.org/c/openstack/governance/+/931254 [4] TC Meeting Agenda, 2024-11-26: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[docs][dev] replace_global_vars not working in project-deploy-guide
by Jakub Darmach 25 Nov '24

25 Nov '24

Hello everyone, I've been updating documentation in kolla-ansible and noticed that while replacing variable content works in https://docs.openstack.org/kolla-ansible it fails to replace the variable with proper content in https://docs.openstack.org/project-deploy-guide/kolla-ansible. If I understand correctly, it's being substituted with var having proper content in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… Later on used in https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… And now for the example - |KOLLA_BRANCH_NAME| variable: Working substitution: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#t… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… Failed to render content: https://docs.openstack.org/project-deploy-guide/kolla-ansible/latest/quicks… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… The var seems to be used the same in both examples, I suspect slightly different configuration on project end of the things. Can anyone advise where should I start looking? Best regards, Jakub

2 1

[neutron] Bug deputy report for week of November 18th
by Brian Haley 25 Nov '24

25 Nov '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - https://review.opendev.org/c/openstack/neutron/+/935804 High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2088453 - [UT] Neutron tests failing with eventlet 0.37.0 - https://review.opendev.org/c/openstack/neutron/+/935524 * https://bugs.launchpad.net/neutron/+bug/2089157 - [neutron-specs] CI job "openstack-tox-docs" broken - https://review.opendev.org/c/openstack/neutron-specs/+/935836 * https://bugs.launchpad.net/neutron/+bug/2089250 - neutron-vpnaas: Automatic VPN Agent failover not working with OVN driver - needs owner * https://bugs.launchpad.net/neutron/+bug/2089252 - neutron-vpnaas: LibreSwan 4 connections not listed as ACTIVE - Submitter said he would look this week Medium ------ * https://bugs.launchpad.net/neutron/+bug/2089169 - [OVN] Test ``TestCreateNeutronPgDrop.test_non_existing`` is randomly failing - https://review.opendev.org/c/openstack/neutron/+/935802

1 0

[cinder] Meeting canceled this week, 27 Nov
by Jon Bernard 25 Nov '24

25 Nov '24

Hello all, schools are closed this week so I won't be able to host the weekly meeting due to the holidays. I'll see you all the following week. Take care, -- Jon

1 0

[horizon][ceph] object storage container storage policy missing
by Satish Patel 25 Nov '24

25 Nov '24

Folks, Openstack Release: Zed Recently I have integrated ceph RGW service with openstack/keystone and testing with command line tool "openstack container create foo1" and it works but when I go to Horizon UI I am seeing it doesn't have storage policy dropdown list and because of that I am not able to create container or upload file from Horizon GUI https://ibb.co/KhYW6Rd It works with commands like but not horizon. $ openstack container create foo1 +---------------------------------------+-----------+------------------------------------------------+ | account | container | x-trans-id | +---------------------------------------+-----------+------------------------------------------------+ | AUTH_d4d82849885e4126aef402c408ad2c5c | foo1 | tx00000c057454841b13497-0065b558b4-42ff2eb-ahm | +---------------------------------------+-----------+------------------------------------------------+ $ openstack container list +------+ | Name | +------+ | foo1 | +------+

2 2

[Neutron] - Unable to ping my instance from my Openstack controller node
by ABATCHA OLLOH 23 Nov '24

23 Nov '24

Hello Everyone, Could you give me any advice and help me to better understand neutron. On my VirtualBox VM I properly installed Openstack using packstack (all-in-one installation). I have access to the horizon dashboard. I'm able to launch an instance and associate floating IP but from the controller node I cannot reach my instance. - Here are my interfaces config: - enp0s3: for mgmt - enp0s8: for openstack services - enp0s9: for neutron_external_interface (used as a public interface) - enp0s10: interface on my controller to be connected to the neutron_external_interface - VirtualBox Promiscuous mode : Allow All [root@packstack ~(keystone_admin)]# ip -br -c a lo UNKNOWN 127.0.0.1/8 ::1/128 enp0s3 UP 9.10.93.8/24 fe80::a00:27ff:fe2e:150a/64 enp0s8 UP 9.11.93.8/24 fe80::a00:27ff:fec7:56ab/64 enp0s9 UP fe80::a00:27ff:fef9:3cc7/64 enp0s10 UP 9.12.93.15/24 fe80::a00:27ff:feff:3641/64 ovs-system DOWN br-tun DOWN br-int DOWN br-ex UNKNOWN 9.12.93.8/24 fe80::b021:85ff:fe8a:9d44/64 qbr9eefea66-89 UP qvo9eefea66-89@qvb9eefea66-89 UP fe80::1409:2aff:feb4:e37d/64 qvb9eefea66-89@qvo9eefea66-89 UP fe80::8c84:15ff:fe7d:8896/64 tap9eefea66-89 UNKNOWN fe80::fc16:3eff:fec6:5f5c/64 - Security Group Rules associated to my instance - Egress IPv4 Any Any 0.0.0.0/0 - Ingress IPv4 ICMP Any 0.0.0.0/0 - Ingress IPv4 TCP 22 (SSH) 0.0.0.0/0 - Ingress IPv4 TCP 80 (HTTP) 0.0.0.0/0 - Instances status [root@packstack ~(keystone_admin)]# openstack server list +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | testrocky | ACTIVE | private1=10.0.0.202, 9.12.93.203 | N/A (booted from volume) | m1.devops | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ - From the router inside my openstack I can ping the floating ip [root@packstack ~(keystone_admin)]# ip netns list qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. 64 bytes from 9.12.93.203: icmp_seq=1 ttl=64 time=6.67 ms 64 bytes from 9.12.93.203: icmp_seq=2 ttl=64 time=1.51 ms9.12.93.203 [root@packstack ~(keystone_admin)]# ip netns qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ip route default via 9.12.93.1 dev qg-68351942-28 proto static 9.12.93.0/24 dev qg-68351942-28 proto kernel scope link src 9.12.93.201 10.0.0.0/24 dev qr-5ba4fd5b-b5 proto kernel scope link src 10.0.0.1 - Trying to reach the floating IP from the controller node [root@packstack ~(keystone_admin)]# ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. From icmp_seq=1 Destination Host Unreachable ...9.12.93.2039.12.93.8 - Check floating ip status [root@packstack ~(keystone_admin)]# openstack port list --network private1 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | 5ba4fd5b-b5e1-4f56-b681-963cf2c9ac8a | | fa:16:3e:14:0b:4a | ip_address='10.0.0.1', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | 9eefea66-897b-4682-883c-b61e22c3c09a | | fa:16:3e:c6:5f:5c | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | a73cd16c-8480-4461-99f2-7837e10f27b1 | | fa:16:3e:91:5f:98 | ip_address='10.0.0.200', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 9eefea66-897b-4682-883c-b61e22c3c09a +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | packstack.example.local | | binding_profile | | | binding_vif_details | bound_drivers.0='openvswitch', bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2024-11-15T11:38:07Z | | data_plane_status | None | | description | | | device_id | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | | hardware_offload_type | None | | hints | | | id | 9eefea66-897b-4682-883c-b61e22c3c09a | | ip_allocation | None | | mac_address | fa:16:3e:c6:5f:5c | | name | | | network_id | 2a02741e-35f0-4a61-81b0-abd4b5a09f36 | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | bbba2507099c4209a529acd64352939e | | propagate_uplink_status | None | | resource_request | None | | revision_number | 10 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | 2a206a8d-fb54-4fa7-9db7-e5f197dcc107 | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2024-11-18T08:15:49Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ [root@packstack ~(keystone_admin)]# openstack port list --network public1 +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | 2b215f41-edf8-4c61-8969-383143340444 | | fa:16:3e:30:7e:08 | ip_address='9.12.93.200', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | fa:16:3e:28:39:a9 | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | N/A | | 68351942-28a1-4df3-8661-bf157fcd5982 | | fa:16:3e:bf:66:56 | ip_address='9.12.93.201', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 31d7b194-50a0-4a25-b102-542210e5f3f3 +-------------------------+----------------------------------------------------------------------------+ | Field | Value | +-------------------------+----------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2024-11-15T15:57:42Z | | data_plane_status | None | | description | | | device_id | 3dc9d9c3-28eb-4dfb-a41b-9bbfac9f96da | | device_owner | network:floatingip | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | | hardware_offload_type | None | | hints | | | id | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | ip_allocation | None | | mac_address | fa:16:3e:28:39:a9 | | name | | | network_id | bcc1c132-074f-45d5-a715-a2d371cdb5be | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | | | propagate_uplink_status | None | | resource_request | None | | revision_number | 2 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | | | status | N/A | | tags | | | trunk_details | None | | updated_at | 2024-11-15T15:57:43Z | +-------------------------+----------------------------------------------------------------------------+ - I couldn't find something related to the binding port in these logs file. - tail -f /var/log/neutron/openvswitch-agent.log - tail -f /var/log/neutron/server.log - tail -f /var/log/neutron/l3-agent.log [image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…> Any advise will be welcome

2 2

Live Migration problems
by Gilles Mocellin 22 Nov '24

22 Nov '24

Hi Stackers, I have 2 recurring problems with live migration. If someone already face that and have recommendations... My env : - OpenStack 2023.1, deployed by OpenStack-Ansible 27.5.1 - Network : ML2/LinuxBridges - Some shared Networks on a VLAN provider - Most private networks on VXLAN - OS : Ubuntu 20.04 - libvirt-daemon-system: 8.0.0-1ubuntu7.10~cloud0 - qemu-system-x86: 1:4.2-3ubuntu6.30 Each time we have maintenance to do on compute nodes, we live-migrate the instances, with `nova host-evacuate-live` (still no openstack CLI command to do that). 1. Never ending / failed migrations In general, it works, but each time we have 1 or 2 (on 40 instances) that don't want to complete. It's some memory active instances (Kubernetes controller nodes with etcd...). Even if I had configured live_migration_permit_auto_converge to True. Now, if we try to force complete (openstack server migration force complete), migration fails, instance return to ACTIVE status on source. And I can see libvirt errors : [...] libvirt.libvirtError: operation failed: migration out job: unexpectedly failed Not really helpful. I tried removing live_migration_permit_auto_converge, reduce live_migration_completion_timeout to 60 and set live_migration_timeout_action to force_complete. Same problems. 2. Instance unreachable More problematic, as it's not easy to see : some instances are not reachable after migration. And after a while, DHCP leases expire. The only way to recover is to restart neutron-linuxbridge-agent on the instance compute node. In fact, on all compute nodes, because instances evacuation spread them on all the compute nodes. Then, if DHCP has not expired, the instance is joinable again. We made a script to find those instances, it lists all ports bound to a nova instance. Then connects to each control/network nodes, enter qdhcp network namespaces (found with the MAC adresse in DHCP leases files), and try to ping the instance private IP. The only log I see, but don't know if it has something to do with that is : Device tapxxx-yy has no active binding in host But I can see that message even since, without unjoinable instances... Here it is, if something ring a bell, for an existing bug or configuration error, please let us know ! Thanks. -- Gilles

3 4

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 21 Nov '24

21 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading, the hypervisor is not showing correct numbers of VMs on compute nodes from browser. But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

3 5

[keystone][magnum] Policy doesn't allow certificate:get to be performed
by Satish Patel 21 Nov '24

21 Nov '24

Folks, I have configured keystone with LDAP authentication and it works but when I deploy magnum + magnum-cluster-api. But when I am trying to create a cluster and obtain credentials I am getting policy errors. Few month back I had similar issue [1] and I have resolved withadding [assignment] driver = sql But somehow It's back and started getting errors again. This time I have tried everything but still getting the same error and no idea where to find and how to find a workaround here. My users in LDAP only authenticate and the rest of the things work from SQL. I have assigned role readers to users also. But that didn't resolve the issue. (venv-openstack) root@os-eng-ctrl-01:~# openstack coe cluster config ssr-dev3 identity version 3% is not in supported versions: 2, 2.0, 3 Policy doesn't allow certificate:get to be performed (HTTP 403) (Request-ID: req-d49586c0-fa88-4bbc-9a12-5d07c2b06c0e) My openstack release version is 2023.1 1. https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

1 1

[ironic][networking] Interest in starting a baremetal networking working group?
by Julia Kreger 20 Nov '24

20 Nov '24

Greetings folks, During the PTG this past week, the Ironic community including developers and operators reached a conclusion that we need to start back up a networking focused working group to try and move the current state of capabilities forward when bare metal machines are involved. Specifically, management of physical switch connections to bare metal *and* programming DPU/IPU devices for networking attachments. As we're all quite aware, it is a complex topic with many different threads. Given resurgent interest and a mutual desire to find some better solutions, it just makes sense to have regular discussions. That being said we need to identify an ideal day of week, frequency, and preferred time. I'd recommend sometime between 1500 UTC and 1800 UTC. Preferably a Tuesday or Wednesday. Would every two weeks work? Or would folks prefer to meet weekly to try and build the most possible velocity through regular interaction? Thoughts? -Julia

10 16

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 20 Nov '24

20 Nov '24

Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:image001.png@01DB39FB.69BBA000] Thanks in advance. Regards, Ishan

2 1

[kolla] DHCP not accessible on compute nodes and VM IP Assignment Issue in Compute Nodes - Kolla Ansible 2024.1
by Jaganathan, Keerthana (SCC) 20 Nov '24

20 Nov '24

Hello Kolla Ansible community, We are experiencing difficulties with VM IP address assignment in our OpenStack deployment using Kolla Ansible 2024.1 on Rocky Linux 9.2. Here are the details of the issue: 1. Problem description: IP addresses are successfully assigned to VMs created on control nodes, but VMs created on compute-only nodes fail to receive IP addresses. This suggests that DHCP requests originating from the compute nodes are not reaching the DHCP agents on the control nodes. 2. Error messages/logs: Console log of a Cirros VM on a compute node shows: checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 0.88. request failed ... failed 20/20: up 54.89. request failed failed to read iid from metadata. tried 20 failed to get instance-id of datasource 3. Configuration: - 7 nodes in total: 5 control/compute/network nodes, 2 compute-only nodes - Single bonded interface (bond0) with 3 VLANs (bond0.10, bond0.20, bond0.30) - network_interface: bond0.10 - api_interface: bond0.20 - tunnel_interface: same as network_interface - neutron_external_interface: bond0 - Provider network: VLAN 41 - ml2_conf.ini [ml2_type_vlan] network_vlan_ranges = physnet1:41:41 - 5 neutron DHCP agents, one on each control node 4. Troubleshooting steps taken: - Verified br-tun OpenFlow rules are dropping packets - Confirmed 5 DHCP agents are running, one on each control node - Openvswitch containers: agent and Here are the openvswitch open flow rules (dump) from the compute node (node16) for br-ex, br-in and br-tun (openvswitch-vswitchd)# ovs-ofctl dump-flows br-ex cookie=0xa9774500bf4a6912, duration=175366.856s, table=0, n_packets=3093, n_bytes=945104, priority=4,in_port="phy-br-ex",dl_vlan=1 actions=mod_vlan_vid:41,NORMAL cookie=0xa9774500bf4a6912, duration=177050.619s, table=0, n_packets=2, n_bytes=176, priority=2,in_port="phy-br-ex" actions=drop cookie=0xa9774500bf4a6912, duration=177050.621s, table=0, n_packets=8178, n_bytes=2363856, priority=0 actions=NORMAL (openvswitch-vswitchd)# ovs-ofctl dump-flows br-int cookie=0x98223299c10a59d3, duration=177059.641s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop cookie=0x98223299c10a59d3, duration=175373.945s, table=0, n_packets=123, n_bytes=9594, priority=10,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136 actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=175373.944s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="qvoab4fc3cb-a2" actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=177059.625s, table=0, n_packets=20, n_bytes=1668, priority=2,in_port="int-br-ex" actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=0, n_packets=3028, n_bytes=939610, priority=9,in_port="qvoab4fc3cb-a2" actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=175375.861s, table=0, n_packets=2256, n_bytes=219762, priority=3,in_port="int-br-ex",dl_vlan=41 actions=mod_vlan_vid:1,resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.643s, table=0, n_packets=2, n_bytes=176, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.644s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=24, n_packets=123, n_bytes=9594, priority=2,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136,nd_target=fe80::f816:3eff:fe4f:8020 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=175373.945s, table=24, n_packets=0, n_bytes=0, priority=2,arp,in_port="qvoab4fc3cb-a2",arp_spa=141.52.56.110 actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=177059.641s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.947s, table=25, n_packets=2970, n_bytes=935510, priority=2,in_port="qvoab4fc3cb-a2",dl_src=fa:16:3e:4f:80:20 actions=resubmit(,30) cookie=0x98223299c10a59d3, duration=177059.640s, table=30, n_packets=2970, n_bytes=935510, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.639s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.642s, table=58, n_packets=5351, n_bytes=1165042, priority=0 actions=resubmit(,60) cookie=0x98223299c10a59d3, duration=177059.642s, table=60, n_packets=5351, n_bytes=1165042, priority=1 actions=NORMAL cookie=0x98223299c10a59d3, duration=177059.640s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL (openvswitch-vswitchd)[root@os16 /]# ovs-ofctl dump-flows br-tun cookie=0xad1cb3762184465d, duration=178072.704s, table=0, n_packets=4815, n_bytes=1114796, priority=1,in_port="patch-int" actions=resubmit(,2) cookie=0xad1cb3762184465d, duration=178072.703s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.703s, table=2, n_packets=1164, n_bytes=69852, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=7, n_bytes=528, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=3644, n_bytes=1044416, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.701s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.700s, table=10, n_packets=0, n_bytes=0, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xad1cb3762184465d,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int" cookie=0xad1cb3762184465d, duration=178072.699s, table=20, n_packets=7, n_bytes=528, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=21, n_packets=1164, n_bytes=69852, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=22, n_packets=4815, n_bytes=1114796, priority=0 actions=drop I would greatly appreciate any guidance on resolving this. Best regards, le3188

1 0

[kolla] IRC meeting cancelled - 20.11.2024
by Bartosz Bezak 20 Nov '24

20 Nov '24

Hello, Unfortunately, this week’s Kolla IRC meeting is cancelled. There are multiple conferences happening this week. Apologies for the late notice. Best regards, Bartosz Bezak

1 0

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 20 Nov '24

20 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading the hypervisor is not showing correct numbers of VMs on compute nodes: [image: image.png] But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

1 0

[Cinder] Restore volume snaptshots while volume is in use
by Jan 19 Nov '24

19 Nov '24

Hi, for providing a more enterprise like (VMware, HyperV, Proxmox) snapshot feature, it would be great to be able to revert snapshots in use. With the "revert-to-snapshot" functions everything is there to support this feature. IMO if the storage backend (driver) does not support the operation, it should raise an error (ex. resetting a ceph rbd volume while a vm is running is not possible). But if it supports the operation, we should not stop the user doing this, even if the volume is "in-use". One might introduce a force switch, to prevent triggering it unintended. A frontend component like skyline can also show additional information like the status of an attached instance. Since the "Detach and attach boot volumes" spec [1] was never fully implemented, there is no way to revert an instance to a snapshot state without completely rebuilding it. Like described in the spec draft, this is not always suitable and at least always cumbersome. -- Jan [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/detac… --- suggested patch ----- https://github.com/jmussmann/cinder/commit/4f971d77a94806e9ebdccab45392058c… From 4f971d77a94806e9ebdccab45392058c25005761 Mon Sep 17 00:00:00 2001 From: jmussmann <jmussmann+github(a)posteo.de> Date: Thu, 31 Oct 2024 09:13:45 +0100 Subject: [PATCH] add: allow reverting snapshots of volumes in use --- cinder/volume/api.py | 5 +++-- cinder/volume/manager.py | 4 ++-- cinder/volume/rpcapi.py | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/cinder/volume/api.py b/cinder/volume/api.py index 0e20baff0..9aa160740 100644 --- a/cinder/volume/api.py +++ b/cinder/volume/api.py @@ -390,8 +390,9 @@ class API(base.Base): """revert a volume to a snapshot""" context.authorize(vol_action_policy.REVERT_POLICY, target_obj=volume) + volume_state = volume.status v_res = volume.update_single_status_where( - 'reverting', 'available') + 'reverting', ['available', 'in-use']) if not v_res: msg = (_("Can't revert volume %(vol_id)s to its latest snapshot " "%(snap_id)s. Volume's status must be 'available'.") @@ -408,7 +409,7 @@ class API(base.Base): "snap_id": snapshot.id}) raise exception.InvalidSnapshot(reason=msg) - self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot) + self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot, volume_state) def delete(self, context: context.RequestContext, diff --git a/cinder/volume/manager.py b/cinder/volume/manager.py index 30de5d8c7..18914c073 100644 --- a/cinder/volume/manager.py +++ b/cinder/volume/manager.py @@ -1150,7 +1150,7 @@ class VolumeManager(manager.CleanableManager, self.create_snapshot(context, snapshot) return snapshot - def revert_to_snapshot(self, context, volume, snapshot) -> None: + def revert_to_snapshot(self, context, volume, snapshot, original_volume_state) -> None: """Revert a volume to a snapshot. The process of reverting to snapshot consists of several steps: @@ -1207,7 +1207,7 @@ class VolumeManager(manager.CleanableManager, LOG.exception(msg, msg_args) v_res = volume.update_single_status_where( - 'available', 'reverting') + original_volume_state, 'reverting') if not v_res: msg_args = {"id": volume.id, "status": 'available'} diff --git a/cinder/volume/rpcapi.py b/cinder/volume/rpcapi.py index 58de69c1c..0b36cebdd 100644 --- a/cinder/volume/rpcapi.py +++ b/cinder/volume/rpcapi.py @@ -180,11 +180,11 @@ class VolumeAPI(rpc.RPCAPI): volume=volume) @rpc.assert_min_rpc_version('3.15') - def revert_to_snapshot(self, ctxt, volume, snapshot): + def revert_to_snapshot(self, ctxt, volume, snapshot, volume_state): version = self._compat_ver('3.15') cctxt = self._get_cctxt(volume.service_topic_queue, version) cctxt.cast(ctxt, 'revert_to_snapshot', volume=volume, - snapshot=snapshot) + snapshot=snapshot, original_volume_state=volume_state) def delete_volume(self, ctxt: context.RequestContext, -- 2.47.0

3 2

#eventlet-removal - HTTP SGI working group call for discussion
by Herve Beraud 19 Nov '24

19 Nov '24

tl;dr: We should have a cross discussion concerning the replacement of Eventlet.wsgi and we want to invite you to join the discussion and hence help us to define a standardized approach in a collaborative way. The Eventlet WSGI module can be used for 2 things: 1. as a WSGI server to serve any kind of web framework (flask, routes, etc...) [1] 2. and in some cases as a framework for web applications [2] Some OpenStack deliverables are using it in one way or another. Some deliverables like Glance already have credible migrations opportunities. But some of you may still be wondering what is your option to replace the Eventlet WSGI features? That's a good question. Now that the specs defining the removal of Eventlet from oslo.service are merged we have to define the way we want to proceed for the WSGI features. Indeed, we decided to drop the WSGI support from oslo.service. That means that we have to define, at least, a "default" way to replace it. If we want to avoid having multiple ways to manage things, we have to provide some guidance and standardizations. For this reason, after the PTG, we encouraged the creation of the HTTP SGI Working Group [4]. This working group aim to answer which (W|A)SGI servers are needed or credible alternatives. Which framework we have/want/need to use in parallel with the SGI server. The outcomes created by this group will serve as "a default" to guide the community through the replacement of the WSGI features of Eventlet. We prepared comparisons of the available alternatives for each layer: - https://wiki.openstack.org/wiki/SGI-servers-comparison - https://wiki.openstack.org/wiki/web-frameworks-comparison - https://wiki.openstack.org/wiki/Http-libraries-comparison Feel free to comment and to put your pros and cons on the given related etherpads (see the links in the comparison). This topic will be discussed at each irc meeting on the #openstack-eventlet-removal channel [5], until we find a default official option. Next meeting date is: 2024 December 3rd [6]. It would be good to reach an official consensus before the Milestone 3 of Epoxy. The goal is simply to ensure that our community looks in the same direction and that everyone is on the same page. We want to ensure that nobody is left behind. Feel free to reply to this thread or to put your comments directly on the various etherpads [6][7][8][9]. Thanks for reading. [1] https://eventlet.readthedocs.io/en/latest/reference/api/eventlet.html#modul… [2] https://eventlet.readthedocs.io/en/latest/modules/wsgi.html [3] https://review.opendev.org/c/openstack/oslo-specs/+/927503 [4] https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group [5] https://meetings.opendev.org/meetings/openstack_eventlet_removal/ [6] https://etherpad.opendev.org/p/epoxy-eventlet-tracking [7] https://etherpad.opendev.org/p/eventlet-removal-SGI-servers-comparison [8] https://etherpad.opendev.org/p/eventlet-removal-web-frameworks-comparison [9] https://etherpad.opendev.org/p/eventlet-removal-http-libraries-comparison -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

Kolla TLS
by devispain＠mail.com 19 Nov '24

19 Nov '24

Reading the Kolla docs it says: The back-end TLS cert/key can be the same certificate that is used for the VIP, as long as those certificates are configured to allow requests from both the VIP and internal networks. But there is no advice on how to do this, can someone explain how to use letsencrypt vip certs for backend tls, i have found nowhere that explains it at all. Trying to use a self signed cert for backend tls seems to break everything when using proper certs for vips

1 0

python3-dns 2.6.1 breaks Nova
by Francesco Di Nucci 19 Nov '24

19 Nov '24

Hi all, maybe this is already known (https://bugs.launchpad.net/glance/+bug/2054732) but also python3-dns-2.6.1 seems to break Nova and other services, with the error reported in the issue, that the DNS "answered udp() got an unexpected keyword argument 'ignore_errors';" It seems that Almalinux has the new version in the repos, while Rocky Linux still has 2.3.0 Best regards Francesco Di Nucci

2 2

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 19 Nov '24

19 Nov '24

Hello Stackers, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base The image includes several packages (such as Apache) that don’t appear necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though all containers don’t use them. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

Oslo Spec Review, Eventlet Removal Proposal.
by Daniel Bengtsson 19 Nov '24

19 Nov '24

Hi everyone, I hope you’re all doing well. During yesterday’s Oslo IRC meeting, the main topic was the removal of eventlet in oslo.service[1]. It seems that everyone is in agreement with this proposal, and no objections were raised. I would appreciate it if you could take another look and review the code. If we don’t receive any further objections by Monday, I propose we proceed with merging the spec. Thanks in advance for your help and feedback! [1] https://review.opendev.org/c/openstack/oslo-specs/+/927503?usp=dashboard -- Daniel Bengtsson Software engineer

2 1

[publiccloud-sig] Reminder: next meeting November 20th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 19 Nov '24

19 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 20th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> As I’ll be traveling, I’ll not be able to participate myself. felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 19 Nov '24

19 Nov '24

Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:image001.png@01DB39FF.4A36CA30] Thanks in advance, Ishan

4 3

[octavia] [lbaas] CPU pinning & multiqueue
by yardalgedal＠gmail.com 19 Nov '24

19 Nov '24

Hi, openstack team, is someone tested CPU pinning together with enabled multiqueue for VM? I'm seeing huge performance degradation (750k RPS => 200k RPS) on flavor with 16 vCPUs and 16 gb ram when using image built with CPU pinning vs image without it, we're using zed (I've backported commits to check, but I think newer versions can also be affected), haproxy 2.4, ubuntu 22.04.

4 4

[Glance] Status of image clone feature
by Masahito Muroi 19 Nov '24

19 Nov '24

Hello Glance folks, I'm wondering the development status of the image clone feature[1]. We provide multi region OpenStack cluster and Glance API is deployed in each region individually. We as a cloud operator want to clone same glance image attributes and images across all the regions using the cloning feature. I have checked the image interoperable feature[2] is available, but the feature looks for cloud users, not for cloud operators. If the development is suspended by some reason, we're happy to start talking the cloning feature. 1. https://wiki.openstack.org/wiki/Glance-image-cloning 2. https://docs.openstack.org/glance/latest/admin/interoperable-image-import.h… thank you, Masahito

2 2

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-19)
by Goutham Pacha Ravi 19 Nov '24

19 Nov '24

Hello Stackers, We're now past milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. As a contributor/maintainer, you have probably made plans to adjust your goals and project deliverables in light of the likely drop in capacity for the year-end holiday season that's ahead. In the past week, the OpenStack Technical Committee didn't commit any new governance changes; however, we're actively working on our action items from the PTG [2]. === Weekly Meeting === The TC's weekly meeting last week was well attended [3]. Clark Boylan (clarkb) noted that contributors were reaching out to OpenDev Infrastructure Maintainers via IRC and the Zuul email list seeking help to set up and configure "third-party" CI systems to meet the requirements set by several projects (Nova, Neutron, Cinder, Manila, ...). Contributors often lack clear documentation and support for setting up these systems, leading to misplaced queries on Zuul or OpenDev channels. We agreed that there is a lack of centralized, up-to-date documentation regarding CI systems put together to test things outside of the purview of the community's Zuul CI. We have lost mindshare regarding third-party CI setup and maintenance, and this hurts the participation of vendor driver maintainers. The TC would love to see interested parties come together to re-form the "Third-Party CI SIG." If we're unable to meaningfully help contributors with this, we must evaluate whether third-party CI should remain a mandatory requirement for service projects. It was highlighted that Nova maintains a dedicated weekly topic to discuss specific third-party CI issues, a model other projects might emulate. We also discussed kicking off the election planning for the 2025.2 cycle. There will be 5 seats available on the OpenStack TC, and every OpenStack project team will elect a PTL or liaison [4]. Election officials will convene over the next few weeks to propose election dates and other details. There's still time to volunteer to be an election official. Please contact me off-list if you're interested! The next meeting of the OpenStack Technical Committee is today (11/18/2024) at 1800 UTC. This meeting will be held via IRC on OFTC's #openstack-tc channel. Please find the meeting agenda on the meeting wiki [5]. I hope you're able to attend. Please remember, you can always propose new topics for this meeting, and it's a great way to synchronously engage with the TC every week. === Governance Proposals === ==== Open for Review ==== rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 ==== Merged ==== Add ubuntu noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] OpenStack TC's Epoxy PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] TC Meeting IRC Log 2024-11-12: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.log.html [4] OpenStack Election website: https://governance.openstack.org/election/ [5] TC Meeting Agenda, 2024-11-19: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

Re: [Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 19 Nov '24

19 Nov '24

Hey Martin, Artemio, Thanks for responding. The VM is sufficiently large for development with about 60 GB of root storage. I was able to fix the keystone failure, by changing the OVS_RUNDIR like this OVS_RUNDIR=$OVS_PREFIX/var/run/ovn. However the script hangs here, and doesn’t move ahead. +functions:wait_for_compute:505 echo 'Didn'\''t find service registered by hostname after 60 seconds' Didn't find service registered by hostname after 60 seconds +functions:wait_for_compute:506 openstack --os-cloud devstack-admin --os-region RegionOne compute service list I am trying to setup a development environment for the nova service. Thanks Ishan From: Martin Kopec <mkopec(a)redhat.com> Date: Tuesday, 19 November 2024 at 4:05 AM To: Artemio Mendoza <artemiom681(a)gmail.com> Cc: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [Devstack] Trying to setup development environment on VM Hi there, the VM should be big enough, the snip from the logs doesn't suggest any reason why keystone didn't start. There is one job that runs devstack on debian every day - https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bookwor… You can try to compare your local.conf with the one used in the jobs. On Mon, 18 Nov 2024 at 21:08, Artemio Mendoza <artemiom681(a)gmail.com<mailto:artemiom681@gmail.com>> wrote: Hi Ishan, In my experience, I was able to set up devstack in the Ubuntu 22.04 (Jammy) VM. I never tried Debian. Hopefully, it can help you. Thank you, Artemio On Mon, Nov 18, 2024 at 12:46 PM Ishan Shanware (ishanwar) <ishanwar(a)cisco.com<mailto:ishanwar@cisco.com>> wrote: Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:ii_19340db3c084cff311] Thanks in advance, Ishan -- Martin Kopec IRC: kopecmartin

1 0

[tc][ptg] 2025.1 "Epoxy" Technical Committee PTG Summary
by Goutham Pacha Ravi 18 Nov '24

18 Nov '24

Hello Stackers, The following is a summary of the Technical Committee's virtual PTG discussions during the last week. For longer notes / recordings, please consult the Etherpad [1] where we kept minutes, or recordings within a YouTube playlist [2]. === Reviving the Inclusive language discussion === The TC brainstormed with the Diversity and Inclusion Working Group [3] about the renaming of the "master" development branch in repositories. We evaluated whether a TC resolution is necessary to provide a clear, standardized approach. Much of OpenDev tooling allows projects the ability to customize their default branch names, with an understanding that this could sometimes be confusing for tools and end-users. Some open-source communities have attempted similar renaming efforts but faced pushback or incomplete adoption, as highlighted by Kubernetes, Ceph, and GNOME. Additionally, there remain concerns over the feasibility, workload, and support available in the OpenStack community to undertake a coordinated renaming effort. The discussion did not yield a consensus and we will not be pursuing an OpenStack wide effort to renaming all the development branches in OpenStack. This doesn't prevent a future change of course, however. If we do undertake this in the future, we will begin by assessing the community's bandwidth for the effort and we'll identify contributors who will lead and assist before proceeding. The conversation also considered the broader scope of inclusive language. The D&I WG does not strongly recommend renaming the "master" branch of git repositories as it is not linked to problematic contexts like slavery in this case. More concerning terminology, like "slave," should be prioritized: https://wiki.openstack.org/wiki/Diversity/Inclusivity Action Items: - Evaluate and Draft a TC Resolution (gouthamr): TC resolution will propose formalizing OpenStack's stance on inclusive language, which would include specific recommendations about branch naming. - Encourage a timeline to deal with non-Inclusive language in code and documentation (tc) === Community Leaders meeting === Subtopic 1: Translating OpenStack documentation and i18n SIG's challenges The OpenStack i18n team discussed challenges and potential improvements for translating OpenStack documentation, particularly due to a slow shift from Zanata to Weblate. The translation team currently has limited activity and faces infrastructure issues, such as Weblate integration problems after a recent cloud upgrade. Key goals are to engage Asian communities (e.g., Korea, Vietnam, Indonesia) to support documentation translations, focusing initially on a subset of projects like Nova, Cinder, Neutron, and Glance. To streamline the process, the i18n team proposed starting with machine translations, which would be refined by regional language moderators. They also discussed AI-generated translations and compliance with OpenInfra's AI policy, requiring clear indication of AI involvement in commits. The i18n team needs infrastructure support, including Zuul automation for translation updates, and guidance for project core teams on reviewing translations. The discussion continued in the i18n SIG meetings during the rest of the week: https://etherpad.opendev.org/p/oct2024-ptg-i18n Action Items - I18n SIG will work on weblate migration with priority, and that will need resolving infrastructure issues as well as figuring out Zuul integration. - Sylvain Bauza (bauzas) takes on the role of the i18n TC liaison, and will highlight the SIGs issues to the TC - Begin translation efforts for a subset of projects (starting with Nova) by translating content in the “docs” folders. - Apply machine translation as a first step, with periodic updates pushed to each project until Zuul automation is set up. - Clearly mark AI-assisted translations in commit messages and add a notice on documentation pages to indicate machine translation usage, in line with OpenInfra’s AI policy. - Define and document processes for core teams to review translated content, considering creating specialized review teams for translation directories. Subtopic 2: Vulnerability Management and your project We discussed the processes within the OpenStack Vulnerability Management Team (VMT) and the need for consistency/coherence in the face of evolving regulation across the world, besides our interest in maintaining good security hygiene. A key concern expressed was the need to keep core security contacts up to date, and the potential for automatic inclusion of all OpenStack projects under the VMT's purview. Adding all project teams and deliverables under the VMT will provide consistency in our security bug management process. It will also allow the VMT to formally help smaller project teams to maintain a good security hygiene. It will allow for cross-project bugs to be resolved quickly while still maintaining the tenets of embargoed disclosures. We also discussed the possibility of a Common Vulnerabilities and Exposures (CVE) authority within OpenStack. Jeremy Stanley (fungi) emphasised that as a community we must be prioritizing alignment and avoiding premature optimizations while regulations are still being drafted. There were suggestions that the VMT wanted the TC to act as a path of escalation to projects where the core security contacts were unresponsive. If we include all TC governed projects under the VMT, there were also concerns about the VMT's workload increasing. fungi clarified factors that drain the VMT's time (mainly chasing project security contacts to triage issues in a timely manner), and didn't think that adding all OpenStack projects would be a burden on the VMT. Having consistent processes may in fact reduce the VMT's workload. There is a strong desire however to grow the VMT with dedicated volunteers. So if you're reading this and are interested, please hop into OFTC's #openstack-security channel and join us. Action Items: - Project PTLs (or DPLs) must update their core-sec teams on Launchpad/Storyboard, removing inactive members where necessary (e.g., Nova to review its `nova-coresec` members) - PTLs of each project must confirm or designate a security liaison for each project, or confirm that the PTL will take on this role. - gouthamr will propose a TC resolution to include all governed projects automatically included in VMT oversight - the security-sig will continue engaging with the Open Regulatory Compliance Working Group to monitor and influence relevant regulatory developments (without immediate process changes on our end at this time) - We will also create a path of escalation for projects not actively following security processes, which may include notifying TC without detailed specifics or, as a last resort, considering project removal from OpenStack. - The VMT and OpenStack Security Team will evaluate the need and feasibility of a common CVE authority for OpenStack and assess community feedback on its potential value. Subtopic 3: Remove Postgres CI jobs from the projects We discussed whether to discontinue Postgres CI jobs across projects, with Ironic and Neutron teams already planning to do so. This stems from a lack of testing coverage and recurring errors in some projects, particularly Neutron. The group raised concerns about the need for a consistent, community-wide approach to database support, especially given the challenges of switching database backends. Current user survey data indicate that about 5% of deployments use Postgres, but maintaining its support is not feasible without dedicated resources. Action Items - Update OpenStack documentation to clearly outline the level of support and testing for each database backend, helping users understand support status before deploying. - Projects (like Ironic, Neutron) must drop their postgres CI jobs, earlier in this cycle so we can prevent unexpected CI failures in interconnected projects - Reviving postgres support will need volunteers. The TC last published a resolution in 2017 [4] explaining the state of support for postgresql. There must be a follow up to state that non-MySQL backends are not tested within the community. Subtopic 4: Migrate CI testing from Ubuntu 22.04 to Ubuntu 24.04 (this implies python3.12 support/usage) Ghanshyam Mann (gmann) is championing the goal to ensure that the advertised runtimes for 2025.1 are enforced in the community CI jobs. Base CI job changes are already underway Action Items: - gmann will share the goal progress with the community as this progresses - project teams must triage integration job failures with Python3.12/Ubuntu Noble and prioritize fixes. They may temporarily pin selective test jobs to python3.9/Ubuntu Jammy however, we expect these issues to be resolved sooner within this release cycle. === Eventlet Removal === This was a "kick-off" cross project discussion on the proposed goal to drop the usage of the "eventlet" library across OpenStack repositories [5]. The migration plan includes identifying suitable alternatives (e.g., native threads, asyncio, awaitlet) to meet different services' needs while addressing challenges around async compatibility with existing infrastructure (e.g., WSGI). Hervé Beraud (hberaud) led this discussion, and we discussed the pop-up team that has been formed to guide this transition. I'd refer you to the summary that hberaud shared on the openstack-discuss ML earlier for a more in-depth summary of further discussions [6] Action Items: - The TC and the community must review the proposal [7] for outcomes, timeline and select the "Eventlet Removal" goal as a cross-community goal - hberaud will set up working groups focused on specific migration needs: background tasks, networking/HTTP, and database interactions. Details are outlined on the Eventlet Removal Wiki [4] - Mike Bayer (zzzeek) and hberaud will work with project teams to assess where asyncio may be beneficial or problematic, and provide guidelines on selecting alternative solutions - We need volunteers from project teams to join the pop-up team and evaluate eventlet usages in the projects they maintain. Please head to OFTC's #openstack-eventlet-removal channel to participate in the discussion === Bridging the gap between community and contributing organizations === Ildiko Vancsa (ildikov) and Jeremy Stanley (fungi) brought the fourth of a series of community discussions to the TC and the community. We brainstormed ways to increase contributions from OpenInfra member organizations, improve contributor experience, and address blockers to community growth and engagement. The goals outlined are to bridge gaps between casual contributors and experienced community members, optimize review processes, and enhance visibility of contribution pathways and expectations. The challenges discussed included visibility of documentation, contributor engagement with maintainers, review delays, and difficulty navigating IRC. The brainstorming that followed proposed some ideas to try, such as contributor spotlights for maintainers (e.g., Superuser articles or video interviews). Please review the "bridging the gap" PTG etherpad for further discussions around the topic [8] Action Items: - Project PTLs must ensure we have clear documentation on patch review and escalation paths. This includes clearly highlighting the core maintainers, liaisons and PTL and ways to contact them. Teams must also list any code review rules that projects have evolved in the documentation. - Project maintainers must consider “review days” to prioritize and expedite small patches. - Develop a Matrix/Element connection guide for OpenStack rooms as an alternative to IRC promoting the guide across "so you want to contribute" pages. - Formalize guidelines around code review pitfalls === OceanBase Database === Members of the OceanBase project community had a cross-project discussion with OpenStack maintainers through this session. OceanBase was stated to be a compatible alternative to MySQL within OpenStack, without necessitating code changes in OpenStack services. The OceanBase team aims to support broader community involvement by contributing integration code for devstack and deployment tools like Kolla, OpenStack-Helm, and OpenStack-Ansible (OSA). The team learned about resource constraints in the community CI jobs, and we brainstormed approaches to support integration testing. Action Items: - OceanBase contributors will implement OceanBase support in devstack, potentially as a plugin if setup is complex. - The team will explore an OpenStack’s CI job by working with the OpenStack QA team, targeting tempest testing - The team will initiate feature requests within deployment projects (e.g., Kolla, OpenStack-Helm, OSA), and potentially collaborate with OpenStack's Large Scale SIG === Updating the OpenStack tenant policy on CI === As a community, we have been discouraging indiscriminate "recheck"s on failed CI jobs; however, our recent experiences landing interdependent CVE fixes within several repositories necessitated fighting repeated unrelated failures. The discussion was around some desire to avoid this in the future and several approaches were brainstormed. Clark Boylan (clarkb) helped the TC understand OpenStack's policy regarding "clean check" practices in the Zuul CI [9]. This policy is intended to reduce repeated gate failures and encourage debugging. It would be an anti-pattern to explore a way to bypass this. The idea of dropping this policy was discussed and the attendees overwhelmingly agreed that there were significant downsides to doing this, including the infeasibility of re-orienting each project team towards how to responsibly craft their CI to avoid pitfalls. There was a suggestion to allocate a limited “infra budget” per project, allowing them to assess re-check frequency and test requirements based on specific project stability and job complexity. There was a tangential discussion on why Zuul's philosophy of testing does not allow re-running single failed jobs, and several contributors shared their experience with CI elsewhere where this convenience came at the expense of introducing serious bugs. Currently, the community can reach out in OFTC's #opendev channel to get help merging a change despite Zuul's disagreements, de-queing a change, or re-enqueuing it on Zuul. These bypass mechanisms were deemed sufficient for the problem at hand. Action Items: - The need to recheck stems from unstable test jobs. Teams must try to move flaky tests to experimental or periodic test queues and attempt to isolate and fix issues with them. - If you spot common failure patterns (e.g., timeouts, mirror issues, OOM errors), please raise awareness to the project teams or to the Infra administrators via the openstack-discuss mailing list or the #opendev channel on OFTC - Zuul CI could use documentation that clarifies why re-running individual jobs in a failed buildset is not a supported action to manage contributor expectations. === Testing and shipping non-OSI compatible software within OpenStack binary artifacts === This discussion was an opportunity to clarify the community's stance on adding, testing, maintaining, supporting, documenting and shipping OpenStack software that pertains to components that are not licensed with an OSI-approved license [10]. A specific recent example was Masakari's Consul integration. Consul ships with a non-OSI-compliant BSL 1.1 license, and is as such incompatible with OpenStack's license policy. However, the general rule has been that the community would strive to support and test only open source solutions. While there may be integrations to proprietary or non-OSI licensed components, they will not be tested with Community Infrastructure, and in each case, a free and open source alternative must be available. In addition, if an OpenStack service supports a feature, there must be a OSI compatible implementation that the community will support and test with CI, and there can be any number of implementations for the feature concerning non-OSI components, but these must be tested by vendors or users of such integrations. Some projects (like Neutron and Cinder) have documented guidelines for this, other projects (like Nova and Manila) enforce this as an unwritten rule. This creates ambiguity around licensing and integration expectations. At the tail end, there was a suggestion to explore the feasibility of hosting an OpenStack container registry (e.g., registry.openstack.org) This could help us better manage binary artifacts and maintain container consistency across services, though permanent registry storage remains a concern. The discussion around this item was tabled. Action Items: - The TC will formally define and document a policy requiring open-source alternatives for non-OSI software integrations. We must ensure that it is applied consistently across all OpenStack projects. - In continuation of this topic, we will review and clarify how to handle existing integrations with dependencies that have changed licenses (e.g., Redis). Determine whether to seek alternatives, discontinue first-party CI testing, or require third-party CI for these integrations. That's a wrap! It was great fun seeing you all at the PTG. I look forward to working on these AIs with you! Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] PTG Etherpad for TC discussions: https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b [2] Recordings of the TC PTG: https://www.youtube.com/playlist?list=PLhwOhbQKWT7XGjIwT0mtPpixpuY-tKYoh [3] PTG Etherpad of the D&I working group: https://etherpad.opendev.org/p/r.4f257833bbd7e0284c26f34e2bbc87c6 [4] TC Resolution on the state of testing of database systems: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.… [5] Eventlet removal wiki: https://wiki.openstack.org/wiki/Eventlet-removal [6] Eventlet removal PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [7] Eventlet removal Goal Proposal: https://review.opendev.org/c/openstack/governance/+/931254 [8] "Bridging the Gap" PTG Etherpad: https://etherpad.opendev.org/p/r.522712847f6a5a1d7bd2031566cde4e9 [9] "Clean Check" CI policy: https://docs.openstack.org/contributors/common/zuul-status.html#why-do-chan… [10] OSI Approved Licenses: https://opensource.org/licenses

2 1

[watcher] re-starting a weekly irc meeting
by Marios Andreou 18 Nov '24

18 Nov '24

Hello OpenStack o/, As discussed at PTG [1] and since there are now a few of us looking to contribute to watcher, it seems like a good time to re-start a regular irc meeting in #openstack-watcher for plans, reviews or any other relevant topics. It seems that Tuesday or Thursday work well for most of the current/new contributors. I started a doodle poll with these two options there https://doodle.com/meeting/participate/id/aMoRKp5d to see if there is any clear preference between the two. I provisionally propose 1200 UTC to (try) and accommodate different time zones (obviously there is no perfect solution for this). If you are interested but the time and/or days do not work for you please reply here so we can try and work it out (If there is enough interest we may consider something like alternate times but first lets see if there is a preference or problem with the current proposal), thanks! marios [1] https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L1

2 1

[Glance] Survey: Property protection file
by Cyril Roelandt 18 Nov '24

18 Nov '24

Hello, Glance allows admins to define custom rules regarding CRUD (create/read/update/delete) operations on custom image properties[1]. This is done by setting the property_protection_file option in glance-api.conf to the path to an INI file defining such permissions. We believe this feature has not seen a lot of use, and are thinking about removing it. We would like to know what you think about this. We set up an online form[2] that you can fill to help us understand real-world use cases for this feature. 1) Are you using a property protection file? 2) Are you using roles, polices, or both for property protection? (Or "None" if you do not use this feature) 3) Can you give examples of config files you use? 4) Would it be an issue for you if Glance were to remove the feature? Only the role-base protection? Only the policy-based protection? On behalf of the Glance team, Cyril [1] https://docs.openstack.org/glance/latest/admin/property-protections.html [2] https://framaforms.org/survey-glance-property-protection-file-1731944193

1 0

[neutron][requirements] Lock failures within Neutron with Eventlet 0.37.0
by Herve Beraud 18 Nov '24

18 Nov '24

Hey Neutrinos, Mid-September we released Eventlet 0.37.0 [1]. Then Jens diligently proposed a patch [2] to bump our global requirements to this newest version of Eventlet. Unfortunately this requirement upgrade is stalled by the Neutron cross job [3]. There are plenty of tests failing with lock errors in threading context (`RuntimeError: cannot notify on un-acquired lock`, please see the logs [3]). A new patch [4] changed the way we manage RLocks and threading at runtime. This patch Upgrade RLocks as the last thing we do. Green existing locks after patching modules, since patching modules might involve imports that create new locks. Neutron tests might potentially need some adjustment to properly work with this new Eventlet version. The support of Python 3.13 is coming and it includes this behavioural change, so if we want the Python 3.13 support of Eventlet in OpenStack, we need first to adjust our OpenStack deliverables to prepare the ground. Can you please check this issue with your Neutron expert eyes. Feel free to also take a look at the changelog of the version 0.37.0 of Eventlet [5], but I think the other patches are irrelevant to this context and can be safely ignored as they are unrelated to locks or threading. To finish Masakari and Mistral also failed but the errors seem a bit different than the Neutron ones. In all the cases Masakri and Mistral cross jobs are non-voting jobs. Thanks for reading. [1] https://github.com/eventlet/eventlet/issues/979 [2] https://review.opendev.org/c/openstack/requirements/+/933257 [3] https://zuul.opendev.org/t/openstack/build/e399c76765f34bfda62e2125b80caf84… [4] https://github.com/eventlet/eventlet/commit/06ec82896ebb9a26edaf6e1ad4d6339… [5] https://github.com/eventlet/eventlet/issues/979 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

2 2

[neutron] Bug deputy - week of Nov 11th
by Sławek Kapłoński 18 Nov '24

18 Nov '24

Hi, I was Neutron bug deputy last week. Here's summary of what new bugs were opened during that period: ## Critical - https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2088285 - [neutron-tempest-plugin] neutron-dynamic-routing using os-ken Quagga are failing - assignent to Lajos, fix already proposed, we have https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935366 and https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935368 ## High - https://bugs.launchpad.net/neutron/+bug/2088166 - os-ken ``DockerImage`` class failing during the image creation - assigned to Rodolfo and patch is proposed already, - https://bugs.launchpad.net/neutron/+bug/2088286 - external port created by neutron-vpnaas with ovn driver is not fully configured - needs assignment, this seems to be bug related to the `ovn-bgp-agent` and `vpnaas` ## Medium - https://bugs.launchpad.net/neutron/+bug/2088154 - ``ProcessManager.disable`` failing if the process doesn't exist - assigned to Rodolfo and patch is proposed already, ## Low - https://bugs.launchpad.net/neutron/+bug/2087909 - When network log exists, neutron-ovn-db-sync-util has a wrong log message - needs assignment, ## Whishlist - https://bugs.launchpad.net/neutron/+bug/2087863 - fullstack job doesn't collect keepalived and other backend logs - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087939 - [eventlet-deprecation] Remove the usage of eventlet in the OVS agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087941 - [eventlet-deprecation] Remove the usage of eventlet in the OVN metadata agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087942 - [eventlet-deprecation] Remove the usage of eventlet in the OVN agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087943 - [eventlet-deprecation] Remove the usage of eventlet in the L3 agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087944 - [eventlet-deprecation] Remove the usage of eventlet in the DHCP agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087946 - [eventlet-deprecation] Remove the usage of eventlet in the SR-IOV agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087947 - [eventlet-deprecation] Remove the usage of eventlet in the Linux Bridge agent - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2087953 - [eventlet-deprecation] Remove the usage of eventlet in the Neutron API - needs assignment There is bunch of the "eventlet-deprecation" bugs reported this week. I marked all of them with the tag "eventlet-deprecation" so that all can be easily found: https://bugs.launchpad.net/neutron/+bugs?field.tag=eventlet-deprecation -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[neutron] external port created by neutron-vpnaas with ovn driver is not fully configured
by przemyslaw.kuczynski＠ochk.pl 18 Nov '24

18 Nov '24

Guys can You help with https://bugs.launchpad.net/neutron/+bug/2088286 Hello We are deploying OpenStack/Neturon with OVN, BGP (ovn-bgp-agent) and VPNaaS and we encountered a problem - because of incomplete configuration of VPN services external port bgp agent isn't announcing it's address. Summary: external port created by neutron-vpnaas with ovn plugin/driver is not fully/properly configured and because of this ovn-bgp-agent won't announce/publish/configure routing path for this address. $ /var/lib/kolla/venv/bin/neutron-ovn-vpn-agent --version neutron-ovn-vpn-agent 24.0.2.dev42 $ openstack network show public1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2024-09-06T15:48:18Z | | description | | | dns_domain | None | | id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1500 | | name | public1 | | port_security_enabled | True | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 2 | | qos_policy_id | None | | revision_number | 2 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | tags | | | tenant_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | updated_at | 2024-09-06T15:48:19Z | +---------------------------+--------------------------------------+ $ openstack subnet show 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 203.0.113.226-203.0.113.254 | | cidr | 203.0.113.224/27 | | created_at | 2024-09-06T15:48:19Z | | description | | | dns_nameservers | | | dns_publish_fixed_ip | None | | enable_dhcp | False | | gateway_ip | 203.0.113.225 | | host_routes | | | id | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | public1 | | network_id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2024-09-06T15:48:19Z | +----------------------+--------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ $ openstack vpn service create --router router_vpn test_vpn_service +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | Description | | | Flavor | None | | ID | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | Name | test_vpn_service | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Router | 30e448d5-078e-40fe-9418-901f8195b6cb | | State | True | | Status | PENDING_CREATE | | Subnet | None | | external_v4_ip | 203.0.113.246 | | external_v6_ip | None | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +----------------+--------------------------------------+ $ openstack vpn ipsec site connection create --peer-id 1.2.3.4 --peer-address 1.2.3.4 --psk 1234 --vpnservice test_vpn_service --ikepolicy ikepolicy --ipsecpolicy ipsecpolicy test_vpn_tunnel --peer-endpoint-group west-peer-epg --local-endpoint-group local_network +--------------------------+----------------------------------------------------+ | Field | Value | +--------------------------+----------------------------------------------------+ | Authentication Algorithm | psk | | Description | | | ID | ca7f63dc-685f-4f9d-bc23-47c7b5b1c577 | | IKE Policy | 1457dc12-c1ec-4574-8985-9e93dcf06f56 | | IPSec Policy | c68d172f-220f-455c-b46c-b9ff8b9e46e4 | | Initiator | bi-directional | | Local Endpoint Group ID | 0f7f645c-8ae9-4dc8-a286-40afabbc2dd7 | | Local ID | | | MTU | 1500 | | Name | test_vpn_tunnel | | Peer Address | 1.2.3.4 | | Peer CIDRs | | | Peer Endpoint Group ID | 842cc6e8-d124-4258-b9d8-8b901d06cd97 | | Peer ID | 1.2.3.4 | | Pre-shared Key | 1234 | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Route Mode | static | | State | True | | Status | PENDING_CREATE | | VPN Service | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | dpd | {'action': 'hold', 'interval': 30, 'timeout': 120} | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +--------------------------+----------------------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 | vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb | fa:16:3e:e5:e0:69 | ip_address='203.0.113.246', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:vpn_router_gateway | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ routing table on gateway/network node: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) unknown (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 (vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb) stands out from other ports, has no addresses $ ovn-nbctl lsp-set-addresses 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 "fa:16:3e:e5:e0:69 203.0.113.246/27" $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) fa:16:3e:e5:e0:69 203.0.113.246/27 (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown routing table after: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.246 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link And now it's possible for tunnel to connect. I tried to identify code of ovn driver/plugin in neutron-vpnaas responsible for external port allocation, but I wasn't able to find it. --

1 0

[ironic] Having issue with deploying ISO image to node
by James Leong 18 Nov '24

18 Nov '24

Hi all, I am trying to play around with ironic, and I am having an issue with deploying an Ubuntu 20.04 ISO image onto my bare metal node. Since I am quite new to bare metal, I am not entirely sure what went wrong as I couldn't find any error messages shown on my log file (ironic-conductor, ironic-api, ironic_neutron_agent, and nova-compute-ironic). This might be my guess; the ironic inspection is hitting timeout, which is causing the deployment to timeout. However, I have no idea why the inspection is failing at timeout. Any help would be appreciated. Best, James

6 7

[openstack-helm] [Cinder] Cinder-Scheduler oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
by daniel890723＠gmail.com 18 Nov '24

18 Nov '24

My Enviroment: | OS | Ubuntu 22.04 | | Kernel | 5.15.0-87-generic | | kubernetes | v1.30.3 | (tested with both v1.28.14 and v1.30.3) Hello I'm testing with Openstack-helm the installation works fine but when i trying to create volume from image and when creating Cinder-Scheduler shows error and the log output: Cinder-Scheduler log: 2024-11-18 01:13:21.651 1 ERROR cinder.scheduler.filter_scheduler [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Error scheduling e838be3b-80a2-4c6c-98cc-53cc84527236 from last vol-service: cinder-volume-worker@rbd1#rbd1 : ['Traceback (most recent call last):\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task\n result = task.execute(**arguments)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1241, in execute\n model_update = self._create_from_image(context,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/utils.py", line 669, in _wrapper\n return r(f, *args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 379, in __call__\n do = self.iter(retry_state=retry_state)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 314, in iter\n return fut.result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result\n return self.__get_result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result\n raise self._exception\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 382, in __call__\n result = fn(*args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image\n model_update, cloned = self._prepare_image_cache_entry(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/decorator.py", line 232, in fun\n return caller(func, *(extras + args), **kw)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/coordination.py", line 239, in _synchronized\n return f(*a, **k)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry\n model_update = self._create_from_image_cache_or_download(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_from_image_cache_or_download\n with image_utils.TemporaryImages.fetch(\n', ' File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__\n return next(self.gen)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 1375, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 861, in fetch_verify_image\n data = get_qemu_data(image_id, has_meta, format_raw,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 700, in get_qemu_data\n data = qemu_img_info(dest,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 169, in qemu_img_info\n format_name = cinder.privsep.format_inspector.get_format_if_safe(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap\n self.start()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 283, in start\n channel = daemon.RootwrapClientChannel(context=self)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 374, in __init__\n super(RootwrapClientChannel, self).__init__(sock, context)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 187, in __init__\n self.exchange_ping()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping\n raise FailedToDropPrivileges(msg)\n', 'oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start\n'] 2024-11-18 01:13:21.652 1 INFO cinder.message.api [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Creating message record for request_id = req-d4455f4e-a374-4251-ab04-7c3523809b6f 2024-11-18 01:13:21.667 1 ERROR cinder.scheduler.flows.create_volume [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236 I also tried several way to test by set SYS_ADMIN on Cinder_scheduler pod change the version to 2023.2 but non of the above method works Is this because the pod permission or there is something i need to set when deploy using openstack-helm ? P.S : i can create Empty Volume and mount to instance

1 0

[FOSDEM] Reminder about call for participation - Virtualization and Cloud infrastructure Room 2025
by Piotr Kliczewski 17 Nov '24

17 Nov '24

All, This yeah submission deadline is enforced for all devrooms so this year our due date is on 1st of December not as all previous years on 8th. This is our gentle reminder to submit [0] your proposals. See you soon at FOSDEM! [0] https://fosdem.org/submit

1 0

[ironic] PTG Summary and Workitems for 2025.1
by Jay Faulkner 16 Nov '24

16 Nov '24

Hi all, The Ironic team, as usual, has whittled our PTG notes down into a document describing the items we'll be focusing on this cycle. The usual disclaimer applies: this is open source, and we're talking about what we /want/ to do, not making any promises about what /will/ be done. The document can be found at https://specs.openstack.org/openstack/ironic-specs/priorities/2025-1-workit…. I hesitate to include this in the same email; because that ^ document is semi-official and this is not; but I also made a short video describing these priorities if you prefer that medium; https://www.youtube.com/watch?v=oy25mATgbrY Thanks to everyone in the Ironic community for an awesome PTG, and a special thanks to our PTL Riccardo for doing an excellent job organizing and moderating it. Thanks, Jay Faulkner

1 0

[ec2-api][kuryr][murano][sahara][senlin][solum][winstackers] Move 2023.1 Antelope and older branches to End of Life
by Elõd Illés 15 Nov '24

15 Nov '24

Hi, The projects in $SUBJECT are retired projects without any activity or maintenance for some time now. That is why instead of moving their stable/2023.1 (antelope) branch to Unmaintained we plan to directly move them to End of Life. To be able to do that we also have to move older branches to End of Life and delete them too. We don't expect any objection, but if someone wants to step up and bring these projects back to life, then this is the last chance before we move them to End of Life. Thanks, Előd Illés irc: elodilles @ #openstack-release

2 1

[release] Release countdown for week R-19, Nov 18 - 22
by Elõd Illés 15 Nov '24

15 Nov '24

Development Focus ----------------- We are now past the Epoxy-1 milestone. Teams should now be focused on feature development and completion of goals [0]. [0] https://governance.openstack.org/tc/goals/selected/index.html General Information ------------------- Our next milestone in this development cycle will be Epoxy-2, on January 9th, 2025. This milestone is when we freeze the list of deliverables that will be included in the 2025.1 Epoxy final release, so if you plan to introduce new deliverables in this release, please propose a change to add an empty deliverable file in the deliverables/epoxy directory of the openstack/releases repository. Now is also generally a good time to look at bugfixes that were introduced in the master branch that might make sense to be backported and released in a stable release. If you have any question around the OpenStack release process, feel free to ask on this mailing-list or on the #openstack-release channel on IRC. Upcoming Deadlines & Dates -------------------------- Epoxy-2 Milestone: January 9th, 2025 2025.1 Epoxy final release: April 2nd, 2025 Előd Illés irc: elodilles @ #openstack-release

1 0

CLI or Steps to find installed OpenStack version
by Manohar Kundapura (makundap) 15 Nov '24

15 Nov '24

Hi All, Looking for the steps to find out which release/version of OpenStack from the below link, is installed on a Server. https://releases.openstack.org/ Thanks, Manohar

3 2

[horizon] SCSS compilers
by Radomir Dopieralski 15 Nov '24

15 Nov '24

The openstack dashboard uses scss for styling, and so far we have used a python scss library for compiling those files to css for use in the browser. However, that library (pyscss) had in recent years some problems, and is not developed or supported by the authors anymore. So it's time to switch to something with a better support. The django-compressor plugin that we use for handling the compilation of files allows us to use any command-line tool to do the compilation, so we can switch to an scss compiler that is supported and used by the wider community. However, there are several options we can choose from: sassc python-libsass pysassc ruby-sass ruby-sassc node-node-sass Each of those would introduce a non-python dependency on the dashboard. The question for the packagers and distro maintainer is which of those tools would be preferable in terms of support, packaging, and maintenance for the long term? Do you have opinions? Maybe there is another tool that would be even better that I didn't find?

2 3

[neutron] Drivers meeting agenda for Friday, November 15th
by Brian Haley 15 Nov '24

15 Nov '24

Hello Neutrinos: Sorry for the late notice but this is the agenda for tomorrow's drivers meeting: * Configurable Agent Termination on OVS Restart * https://bugs.launchpad.net/neutron/+bug/2086776 * There has been good discussion in this bug from Liu * L3 Agent Readiness Status for HA Routers * https://bugs.launchpad.net/neutron/+bug/2086794 * OVS Agent Synchronization Status for Container Environments * https://bugs.launchpad.net/neutron/+bug/2086799 - related to 2086794 * Add OVN Ganesha Agent to setup path for VM connectivity to NFS-Ganesha * https://bugs.launchpad.net/neutron/+bug/2087541 Unfortunately I will not be able to attend due to a last-minute conflict, so if someone else could lead the meeting I would appreciate it, otherwise we will postpone until next Friday. -Brian

1 0

[watcher] unmaintained branch cleanup
by Sean Mooney 14 Nov '24

14 Nov '24

Hi everyone, As some may now we are currently trying to revive the maintenance of the watcher project, i have volunteered to be the release liaison for the project. At the ptg a few weeks ago we briefly discussed the state of the unmaintained and stable branches watcher currently had 5 unmaintained branches from the oldest unmaintained/victoria <https://opendev.org/openstack/watcher/src/branch/unmaintained/victoria> to the newly created unmaintained/2023.1 <https://opendev.org/openstack/watcher/src/branch/unmaintained/2023.1> https://opendev.org/openstack/watcher/branches since the unmaintained branches have been create, they have not received any meaningful patches, including in some cases the auto generated patches from our openstack patch bots. during the ptg i proposed moving all unmaintained branches older than 2023.1 to end of life. recently https://review.opendev.org/c/openstack/releases/+/935041 has been proposed to "Transition zed-eom branches to EOL" which aligned to my proposed plan for watcher. Per the ptg discussion i committed to reaching out via the mailing list to see if there was any objection to this plan https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L79 please respond here or on https://review.opendev.org/c/openstack/releases/+/935041 if you have an objection to the unmaintained branch transition. regards sean

1 0

[nova] mdev management for vgpu
by Mickael Razzouk 14 Nov '24

14 Nov '24

Hello, I am tying a few things with vGPU on nova and I struggle with the mdev management Nova does. My environment: - Nvidia A2 pGPU - openstack caracal What I did : - installed the nvidia grid host driver on the compute - enabled the virtual function of the pGPU - added in nova.conf the mdev type wanted and the pci address of the virtual functions - restarted nova - created a flavor with the property resources:VGPU=1 (basicaly followed this documentation : https://docs.openstack.org/nova/latest/admin/virtual-gpu.html) the only diference is that I did not put the pGPU pci address in the nova.conf but the address of its virtual functions, so in my case 0000:41:00.4 and not 0000:41:00.0 (the RP were not detected otherwise) From my understanding of this specs sheet (https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/a…) and some experimentation, When a vgpu is requested by the end user : - Nova looks at the mdevs already running and use them if available. - If there is no mdev available, Nova looks at the RP tree to find mdev capable pci devices that have the specific mdev type available. - If there is such pci device available, Nova create the mdev with a UUID and create the domain XML with this UUID inside. - If no such device is available, Nova returns no host available as an error. My questions are : - Mdev are not persistent across reboot, but when it happens, nova crash at boot because the mdevs are missing, just recreating mdev does not fix the issue as they need to have the same UUID. One could fix the issue by making mdev persistent using a tool like mdevctl, but by creating the mdev the first time I seem to me that Nova tries to manage mdev, is it normal for it to require manual intervention afterward ? Or am I missing something ? - In the nova configuration, is it normal to diverge from the documentation and enter the VF PCI addresses to detect the RF ? Thank you in advance for your help.

4 4

[nova] Spec review day this Thursday 14 Nov 2024
by Sylvain Bauza 14 Nov '24

14 Nov '24

Hey folks, As agreed at the PTG, we'll have a spec review day this week on Thursday. Please mark your calendars for that day as we'll do a first round of spec reviews. Thanks all, -Sylvain

1 1