- openstack-discuss - lists.openstack.org

[kolla] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 26 Nov '24

26 Nov '24

Hi Kolla Community, I would like to understand the current guidelines for adding or removing a controller node from an OpenStack cluster. With the introduction of quorum queues, we’ve observed that adding a new controller node causes a loss of quorum within the cluster. Consequently, during the reinstallation of a controller node, Masakari disables all the nova-compute services. Is this behaviour expected with quorum queues, or could this be addressed differently? Additionally, I’ve noticed that as soon as the Kolla containers are restarted in the cluster, clients are unable to connect, causing the cluster to go down. Could you please share insights or recommendations on how to handle this scenario effectively? Looking forward to hearing your thoughts. Thanks, Ishan Cisco Systems

2 2

[keystone] Weekly meeting canceled 27-Nov-2024
by Dave Wilde 26 Nov '24

26 Nov '24

Hello, Due to the Thanksgiving holiday in the US I'm canceling the weekly meeting and reviewathon this week. Please reach out here or on IRC if you need anything. Thanks! /Dave

1 0

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[openstack helm] [Ansible] Openstack-helm Ansible inventory.yaml missing some value
by daniel890723＠gmail.com 25 Nov '24

25 Nov '24

Hello today i'm testing installation with completely new enviroment i was following the guide from Openstack-helm document and install k8s cluster using ansible and deploy-env playbook but i encounter this error when setup ssh cluster TASK [deploy-env : Generate ssh key pair] ****************************************************************************************************************** skipping: [node-1] fatal: [primary]: FAILED! => {"msg": "Failed to change ownership of the temporary files Ansible (via chmod nor setfacl) needs to create despite connecting as a privileged user. Unprivileged become user would be unable to read the file."} and the log with -vvv output: debug2: mux_client_hello_exchange: master version 4 debug3: mux_client_forwards: request forwardings: 0 local, 0 remote debug3: mux_client_request_session: entering debug3: mux_client_request_alive: entering debug3: mux_client_request_alive: done pid = 114038 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 chown: invalid user: ‘zuul’ debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 And by checking the playbook we can see the playbook is using "{{ client_ssh_user }}" on primary node and "{{ cluster_ssh_user }}" with cluster node which is not defined in inventory.yaml Causing it using default user zuul by adding client_ssh_user & cluster_ssh_user in ansible inventory it works normally Is this meant to be like this? or the document missing some variables in inventory.yaml

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-18)
by Goutham Pacha Ravi 25 Nov '24

25 Nov '24

Hello Stackers, We're 18 weeks away from the coordinated release for 2025.1 ("Epoxy") [1]. We're beginning a period of low activity due to year-end holidays across the world. Last week, the TC didn't merge any new governance proposals; however, several are currently in progress. === Weekly Meeting === The TC met in OFTC's #openstack-tc IRC channel on 2024-11-19 [2]. We discussed a series of global requirements upper constraint bumps that were committed and the ensuing CI flakiness. We appreciated the effort to proactively update dependencies to their latest versions earlier in the release cycle so we can have more time to test and stabilize OpenStack software. Sometimes, these failures may interrupt other project priorities. The best way we can help is if project teams identify a relatively quick-running and stable integration job to run against changes to the global requirements repository. The TC and OpenDev administrators worked with Launchpad administrators to regain control of trackers belonging to the Watcher project team. It was noted that Launchpad trackers for any project should not be owned/administered by individuals. Instead, there must be a Launchpad team driving all trackers associated with an OpenStack initiative, and "openstack-admins" must own this team. This approach ensures the sustainability of project teams and protects projects from the impact of changing individual priorities. We also briefly discussed the "eventlet-removal" goal proposal. The goal document is being rewritten [3] to simplify what is being done and the timelines we're aiming for. I'd love for the community to take a look at this change and review it. The next meeting of the OpenStack Technical Committee is today, 2024-11-25, at 1800 UTC in OFTC's #openstack-tc channel. The agenda for this meeting is in our meeting wiki [4]. I hope you're able to join us. === Governance Proposals === ==== Open for Review ==== Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 ==== Merged ==== Fix doc job for pillow 11.0.0 | https://review.opendev.org/c/openstack/governance/+/935967 Add Ubuntu Noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meetings: Join us at our weekly meeting. The Technical Committee meets every week on Tuesdays at 1800 UTC [4]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting IRC Log 2024-11-19: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.log.html [3] Eventlet removal Community goal: https://review.opendev.org/c/openstack/governance/+/931254 [4] TC Meeting Agenda, 2024-11-26: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[docs][dev] replace_global_vars not working in project-deploy-guide
by Jakub Darmach 25 Nov '24

25 Nov '24

Hello everyone, I've been updating documentation in kolla-ansible and noticed that while replacing variable content works in https://docs.openstack.org/kolla-ansible it fails to replace the variable with proper content in https://docs.openstack.org/project-deploy-guide/kolla-ansible. If I understand correctly, it's being substituted with var having proper content in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… Later on used in https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… And now for the example - |KOLLA_BRANCH_NAME| variable: Working substitution: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#t… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… Failed to render content: https://docs.openstack.org/project-deploy-guide/kolla-ansible/latest/quicks… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… The var seems to be used the same in both examples, I suspect slightly different configuration on project end of the things. Can anyone advise where should I start looking? Best regards, Jakub

2 1

[neutron] Bug deputy report for week of November 18th
by Brian Haley 25 Nov '24

25 Nov '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - https://review.opendev.org/c/openstack/neutron/+/935804 High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2088453 - [UT] Neutron tests failing with eventlet 0.37.0 - https://review.opendev.org/c/openstack/neutron/+/935524 * https://bugs.launchpad.net/neutron/+bug/2089157 - [neutron-specs] CI job "openstack-tox-docs" broken - https://review.opendev.org/c/openstack/neutron-specs/+/935836 * https://bugs.launchpad.net/neutron/+bug/2089250 - neutron-vpnaas: Automatic VPN Agent failover not working with OVN driver - needs owner * https://bugs.launchpad.net/neutron/+bug/2089252 - neutron-vpnaas: LibreSwan 4 connections not listed as ACTIVE - Submitter said he would look this week Medium ------ * https://bugs.launchpad.net/neutron/+bug/2089169 - [OVN] Test ``TestCreateNeutronPgDrop.test_non_existing`` is randomly failing - https://review.opendev.org/c/openstack/neutron/+/935802

1 0

[cinder] Meeting canceled this week, 27 Nov
by Jon Bernard 25 Nov '24

25 Nov '24

Hello all, schools are closed this week so I won't be able to host the weekly meeting due to the holidays. I'll see you all the following week. Take care, -- Jon

1 0

[horizon][ceph] object storage container storage policy missing
by Satish Patel 24 Nov '24

24 Nov '24

Folks, Openstack Release: Zed Recently I have integrated ceph RGW service with openstack/keystone and testing with command line tool "openstack container create foo1" and it works but when I go to Horizon UI I am seeing it doesn't have storage policy dropdown list and because of that I am not able to create container or upload file from Horizon GUI https://ibb.co/KhYW6Rd It works with commands like but not horizon. $ openstack container create foo1 +---------------------------------------+-----------+------------------------------------------------+ | account | container | x-trans-id | +---------------------------------------+-----------+------------------------------------------------+ | AUTH_d4d82849885e4126aef402c408ad2c5c | foo1 | tx00000c057454841b13497-0065b558b4-42ff2eb-ahm | +---------------------------------------+-----------+------------------------------------------------+ $ openstack container list +------+ | Name | +------+ | foo1 | +------+

2 2

[Neutron] - Unable to ping my instance from my Openstack controller node
by ABATCHA OLLOH 23 Nov '24

23 Nov '24

Hello Everyone, Could you give me any advice and help me to better understand neutron. On my VirtualBox VM I properly installed Openstack using packstack (all-in-one installation). I have access to the horizon dashboard. I'm able to launch an instance and associate floating IP but from the controller node I cannot reach my instance. - Here are my interfaces config: - enp0s3: for mgmt - enp0s8: for openstack services - enp0s9: for neutron_external_interface (used as a public interface) - enp0s10: interface on my controller to be connected to the neutron_external_interface - VirtualBox Promiscuous mode : Allow All [root@packstack ~(keystone_admin)]# ip -br -c a lo UNKNOWN 127.0.0.1/8 ::1/128 enp0s3 UP 9.10.93.8/24 fe80::a00:27ff:fe2e:150a/64 enp0s8 UP 9.11.93.8/24 fe80::a00:27ff:fec7:56ab/64 enp0s9 UP fe80::a00:27ff:fef9:3cc7/64 enp0s10 UP 9.12.93.15/24 fe80::a00:27ff:feff:3641/64 ovs-system DOWN br-tun DOWN br-int DOWN br-ex UNKNOWN 9.12.93.8/24 fe80::b021:85ff:fe8a:9d44/64 qbr9eefea66-89 UP qvo9eefea66-89@qvb9eefea66-89 UP fe80::1409:2aff:feb4:e37d/64 qvb9eefea66-89@qvo9eefea66-89 UP fe80::8c84:15ff:fe7d:8896/64 tap9eefea66-89 UNKNOWN fe80::fc16:3eff:fec6:5f5c/64 - Security Group Rules associated to my instance - Egress IPv4 Any Any 0.0.0.0/0 - Ingress IPv4 ICMP Any 0.0.0.0/0 - Ingress IPv4 TCP 22 (SSH) 0.0.0.0/0 - Ingress IPv4 TCP 80 (HTTP) 0.0.0.0/0 - Instances status [root@packstack ~(keystone_admin)]# openstack server list +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | testrocky | ACTIVE | private1=10.0.0.202, 9.12.93.203 | N/A (booted from volume) | m1.devops | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ - From the router inside my openstack I can ping the floating ip [root@packstack ~(keystone_admin)]# ip netns list qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. 64 bytes from 9.12.93.203: icmp_seq=1 ttl=64 time=6.67 ms 64 bytes from 9.12.93.203: icmp_seq=2 ttl=64 time=1.51 ms9.12.93.203 [root@packstack ~(keystone_admin)]# ip netns qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ip route default via 9.12.93.1 dev qg-68351942-28 proto static 9.12.93.0/24 dev qg-68351942-28 proto kernel scope link src 9.12.93.201 10.0.0.0/24 dev qr-5ba4fd5b-b5 proto kernel scope link src 10.0.0.1 - Trying to reach the floating IP from the controller node [root@packstack ~(keystone_admin)]# ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. From icmp_seq=1 Destination Host Unreachable ...9.12.93.2039.12.93.8 - Check floating ip status [root@packstack ~(keystone_admin)]# openstack port list --network private1 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | 5ba4fd5b-b5e1-4f56-b681-963cf2c9ac8a | | fa:16:3e:14:0b:4a | ip_address='10.0.0.1', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | 9eefea66-897b-4682-883c-b61e22c3c09a | | fa:16:3e:c6:5f:5c | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | a73cd16c-8480-4461-99f2-7837e10f27b1 | | fa:16:3e:91:5f:98 | ip_address='10.0.0.200', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 9eefea66-897b-4682-883c-b61e22c3c09a +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | packstack.example.local | | binding_profile | | | binding_vif_details | bound_drivers.0='openvswitch', bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2024-11-15T11:38:07Z | | data_plane_status | None | | description | | | device_id | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | | hardware_offload_type | None | | hints | | | id | 9eefea66-897b-4682-883c-b61e22c3c09a | | ip_allocation | None | | mac_address | fa:16:3e:c6:5f:5c | | name | | | network_id | 2a02741e-35f0-4a61-81b0-abd4b5a09f36 | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | bbba2507099c4209a529acd64352939e | | propagate_uplink_status | None | | resource_request | None | | revision_number | 10 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | 2a206a8d-fb54-4fa7-9db7-e5f197dcc107 | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2024-11-18T08:15:49Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ [root@packstack ~(keystone_admin)]# openstack port list --network public1 +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | 2b215f41-edf8-4c61-8969-383143340444 | | fa:16:3e:30:7e:08 | ip_address='9.12.93.200', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | fa:16:3e:28:39:a9 | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | N/A | | 68351942-28a1-4df3-8661-bf157fcd5982 | | fa:16:3e:bf:66:56 | ip_address='9.12.93.201', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 31d7b194-50a0-4a25-b102-542210e5f3f3 +-------------------------+----------------------------------------------------------------------------+ | Field | Value | +-------------------------+----------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2024-11-15T15:57:42Z | | data_plane_status | None | | description | | | device_id | 3dc9d9c3-28eb-4dfb-a41b-9bbfac9f96da | | device_owner | network:floatingip | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | | hardware_offload_type | None | | hints | | | id | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | ip_allocation | None | | mac_address | fa:16:3e:28:39:a9 | | name | | | network_id | bcc1c132-074f-45d5-a715-a2d371cdb5be | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | | | propagate_uplink_status | None | | resource_request | None | | revision_number | 2 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | | | status | N/A | | tags | | | trunk_details | None | | updated_at | 2024-11-15T15:57:43Z | +-------------------------+----------------------------------------------------------------------------+ - I couldn't find something related to the binding port in these logs file. - tail -f /var/log/neutron/openvswitch-agent.log - tail -f /var/log/neutron/server.log - tail -f /var/log/neutron/l3-agent.log [image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…> Any advise will be welcome

2 2

Live Migration problems
by Gilles Mocellin 22 Nov '24

22 Nov '24

Hi Stackers, I have 2 recurring problems with live migration. If someone already face that and have recommendations... My env : - OpenStack 2023.1, deployed by OpenStack-Ansible 27.5.1 - Network : ML2/LinuxBridges - Some shared Networks on a VLAN provider - Most private networks on VXLAN - OS : Ubuntu 20.04 - libvirt-daemon-system: 8.0.0-1ubuntu7.10~cloud0 - qemu-system-x86: 1:4.2-3ubuntu6.30 Each time we have maintenance to do on compute nodes, we live-migrate the instances, with `nova host-evacuate-live` (still no openstack CLI command to do that). 1. Never ending / failed migrations In general, it works, but each time we have 1 or 2 (on 40 instances) that don't want to complete. It's some memory active instances (Kubernetes controller nodes with etcd...). Even if I had configured live_migration_permit_auto_converge to True. Now, if we try to force complete (openstack server migration force complete), migration fails, instance return to ACTIVE status on source. And I can see libvirt errors : [...] libvirt.libvirtError: operation failed: migration out job: unexpectedly failed Not really helpful. I tried removing live_migration_permit_auto_converge, reduce live_migration_completion_timeout to 60 and set live_migration_timeout_action to force_complete. Same problems. 2. Instance unreachable More problematic, as it's not easy to see : some instances are not reachable after migration. And after a while, DHCP leases expire. The only way to recover is to restart neutron-linuxbridge-agent on the instance compute node. In fact, on all compute nodes, because instances evacuation spread them on all the compute nodes. Then, if DHCP has not expired, the instance is joinable again. We made a script to find those instances, it lists all ports bound to a nova instance. Then connects to each control/network nodes, enter qdhcp network namespaces (found with the MAC adresse in DHCP leases files), and try to ping the instance private IP. The only log I see, but don't know if it has something to do with that is : Device tapxxx-yy has no active binding in host But I can see that message even since, without unjoinable instances... Here it is, if something ring a bell, for an existing bug or configuration error, please let us know ! Thanks. -- Gilles

3 4

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 21 Nov '24

21 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading, the hypervisor is not showing correct numbers of VMs on compute nodes from browser. But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

3 5

[keystone][magnum] Policy doesn't allow certificate:get to be performed
by Satish Patel 21 Nov '24

21 Nov '24

Folks, I have configured keystone with LDAP authentication and it works but when I deploy magnum + magnum-cluster-api. But when I am trying to create a cluster and obtain credentials I am getting policy errors. Few month back I had similar issue [1] and I have resolved withadding [assignment] driver = sql But somehow It's back and started getting errors again. This time I have tried everything but still getting the same error and no idea where to find and how to find a workaround here. My users in LDAP only authenticate and the rest of the things work from SQL. I have assigned role readers to users also. But that didn't resolve the issue. (venv-openstack) root@os-eng-ctrl-01:~# openstack coe cluster config ssr-dev3 identity version 3% is not in supported versions: 2, 2.0, 3 Policy doesn't allow certificate:get to be performed (HTTP 403) (Request-ID: req-d49586c0-fa88-4bbc-9a12-5d07c2b06c0e) My openstack release version is 2023.1 1. https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

1 1

[ironic][networking] Interest in starting a baremetal networking working group?
by Julia Kreger 20 Nov '24

20 Nov '24

Greetings folks, During the PTG this past week, the Ironic community including developers and operators reached a conclusion that we need to start back up a networking focused working group to try and move the current state of capabilities forward when bare metal machines are involved. Specifically, management of physical switch connections to bare metal *and* programming DPU/IPU devices for networking attachments. As we're all quite aware, it is a complex topic with many different threads. Given resurgent interest and a mutual desire to find some better solutions, it just makes sense to have regular discussions. That being said we need to identify an ideal day of week, frequency, and preferred time. I'd recommend sometime between 1500 UTC and 1800 UTC. Preferably a Tuesday or Wednesday. Would every two weeks work? Or would folks prefer to meet weekly to try and build the most possible velocity through regular interaction? Thoughts? -Julia

10 16

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 20 Nov '24

20 Nov '24

Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:image001.png@01DB39FB.69BBA000] Thanks in advance. Regards, Ishan

2 1

[kolla] DHCP not accessible on compute nodes and VM IP Assignment Issue in Compute Nodes - Kolla Ansible 2024.1
by Jaganathan, Keerthana (SCC) 20 Nov '24

20 Nov '24

Hello Kolla Ansible community, We are experiencing difficulties with VM IP address assignment in our OpenStack deployment using Kolla Ansible 2024.1 on Rocky Linux 9.2. Here are the details of the issue: 1. Problem description: IP addresses are successfully assigned to VMs created on control nodes, but VMs created on compute-only nodes fail to receive IP addresses. This suggests that DHCP requests originating from the compute nodes are not reaching the DHCP agents on the control nodes. 2. Error messages/logs: Console log of a Cirros VM on a compute node shows: checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 0.88. request failed ... failed 20/20: up 54.89. request failed failed to read iid from metadata. tried 20 failed to get instance-id of datasource 3. Configuration: - 7 nodes in total: 5 control/compute/network nodes, 2 compute-only nodes - Single bonded interface (bond0) with 3 VLANs (bond0.10, bond0.20, bond0.30) - network_interface: bond0.10 - api_interface: bond0.20 - tunnel_interface: same as network_interface - neutron_external_interface: bond0 - Provider network: VLAN 41 - ml2_conf.ini [ml2_type_vlan] network_vlan_ranges = physnet1:41:41 - 5 neutron DHCP agents, one on each control node 4. Troubleshooting steps taken: - Verified br-tun OpenFlow rules are dropping packets - Confirmed 5 DHCP agents are running, one on each control node - Openvswitch containers: agent and Here are the openvswitch open flow rules (dump) from the compute node (node16) for br-ex, br-in and br-tun (openvswitch-vswitchd)# ovs-ofctl dump-flows br-ex cookie=0xa9774500bf4a6912, duration=175366.856s, table=0, n_packets=3093, n_bytes=945104, priority=4,in_port="phy-br-ex",dl_vlan=1 actions=mod_vlan_vid:41,NORMAL cookie=0xa9774500bf4a6912, duration=177050.619s, table=0, n_packets=2, n_bytes=176, priority=2,in_port="phy-br-ex" actions=drop cookie=0xa9774500bf4a6912, duration=177050.621s, table=0, n_packets=8178, n_bytes=2363856, priority=0 actions=NORMAL (openvswitch-vswitchd)# ovs-ofctl dump-flows br-int cookie=0x98223299c10a59d3, duration=177059.641s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop cookie=0x98223299c10a59d3, duration=175373.945s, table=0, n_packets=123, n_bytes=9594, priority=10,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136 actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=175373.944s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="qvoab4fc3cb-a2" actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=177059.625s, table=0, n_packets=20, n_bytes=1668, priority=2,in_port="int-br-ex" actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=0, n_packets=3028, n_bytes=939610, priority=9,in_port="qvoab4fc3cb-a2" actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=175375.861s, table=0, n_packets=2256, n_bytes=219762, priority=3,in_port="int-br-ex",dl_vlan=41 actions=mod_vlan_vid:1,resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.643s, table=0, n_packets=2, n_bytes=176, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.644s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=24, n_packets=123, n_bytes=9594, priority=2,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136,nd_target=fe80::f816:3eff:fe4f:8020 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=175373.945s, table=24, n_packets=0, n_bytes=0, priority=2,arp,in_port="qvoab4fc3cb-a2",arp_spa=141.52.56.110 actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=177059.641s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.947s, table=25, n_packets=2970, n_bytes=935510, priority=2,in_port="qvoab4fc3cb-a2",dl_src=fa:16:3e:4f:80:20 actions=resubmit(,30) cookie=0x98223299c10a59d3, duration=177059.640s, table=30, n_packets=2970, n_bytes=935510, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.639s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.642s, table=58, n_packets=5351, n_bytes=1165042, priority=0 actions=resubmit(,60) cookie=0x98223299c10a59d3, duration=177059.642s, table=60, n_packets=5351, n_bytes=1165042, priority=1 actions=NORMAL cookie=0x98223299c10a59d3, duration=177059.640s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL (openvswitch-vswitchd)[root@os16 /]# ovs-ofctl dump-flows br-tun cookie=0xad1cb3762184465d, duration=178072.704s, table=0, n_packets=4815, n_bytes=1114796, priority=1,in_port="patch-int" actions=resubmit(,2) cookie=0xad1cb3762184465d, duration=178072.703s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.703s, table=2, n_packets=1164, n_bytes=69852, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=7, n_bytes=528, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=3644, n_bytes=1044416, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.701s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.700s, table=10, n_packets=0, n_bytes=0, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xad1cb3762184465d,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int" cookie=0xad1cb3762184465d, duration=178072.699s, table=20, n_packets=7, n_bytes=528, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=21, n_packets=1164, n_bytes=69852, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=22, n_packets=4815, n_bytes=1114796, priority=0 actions=drop I would greatly appreciate any guidance on resolving this. Best regards, le3188

1 0

[kolla] IRC meeting cancelled - 20.11.2024
by Bartosz Bezak 20 Nov '24

20 Nov '24

Hello, Unfortunately, this week’s Kolla IRC meeting is cancelled. There are multiple conferences happening this week. Apologies for the late notice. Best regards, Bartosz Bezak

1 0

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 19 Nov '24

19 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading the hypervisor is not showing correct numbers of VMs on compute nodes: [image: image.png] But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

1 0

[Cinder] Restore volume snaptshots while volume is in use
by Jan 19 Nov '24

19 Nov '24

Hi, for providing a more enterprise like (VMware, HyperV, Proxmox) snapshot feature, it would be great to be able to revert snapshots in use. With the "revert-to-snapshot" functions everything is there to support this feature. IMO if the storage backend (driver) does not support the operation, it should raise an error (ex. resetting a ceph rbd volume while a vm is running is not possible). But if it supports the operation, we should not stop the user doing this, even if the volume is "in-use". One might introduce a force switch, to prevent triggering it unintended. A frontend component like skyline can also show additional information like the status of an attached instance. Since the "Detach and attach boot volumes" spec [1] was never fully implemented, there is no way to revert an instance to a snapshot state without completely rebuilding it. Like described in the spec draft, this is not always suitable and at least always cumbersome. -- Jan [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/detac… --- suggested patch ----- https://github.com/jmussmann/cinder/commit/4f971d77a94806e9ebdccab45392058c… From 4f971d77a94806e9ebdccab45392058c25005761 Mon Sep 17 00:00:00 2001 From: jmussmann <jmussmann+github(a)posteo.de> Date: Thu, 31 Oct 2024 09:13:45 +0100 Subject: [PATCH] add: allow reverting snapshots of volumes in use --- cinder/volume/api.py | 5 +++-- cinder/volume/manager.py | 4 ++-- cinder/volume/rpcapi.py | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/cinder/volume/api.py b/cinder/volume/api.py index 0e20baff0..9aa160740 100644 --- a/cinder/volume/api.py +++ b/cinder/volume/api.py @@ -390,8 +390,9 @@ class API(base.Base): """revert a volume to a snapshot""" context.authorize(vol_action_policy.REVERT_POLICY, target_obj=volume) + volume_state = volume.status v_res = volume.update_single_status_where( - 'reverting', 'available') + 'reverting', ['available', 'in-use']) if not v_res: msg = (_("Can't revert volume %(vol_id)s to its latest snapshot " "%(snap_id)s. Volume's status must be 'available'.") @@ -408,7 +409,7 @@ class API(base.Base): "snap_id": snapshot.id}) raise exception.InvalidSnapshot(reason=msg) - self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot) + self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot, volume_state) def delete(self, context: context.RequestContext, diff --git a/cinder/volume/manager.py b/cinder/volume/manager.py index 30de5d8c7..18914c073 100644 --- a/cinder/volume/manager.py +++ b/cinder/volume/manager.py @@ -1150,7 +1150,7 @@ class VolumeManager(manager.CleanableManager, self.create_snapshot(context, snapshot) return snapshot - def revert_to_snapshot(self, context, volume, snapshot) -> None: + def revert_to_snapshot(self, context, volume, snapshot, original_volume_state) -> None: """Revert a volume to a snapshot. The process of reverting to snapshot consists of several steps: @@ -1207,7 +1207,7 @@ class VolumeManager(manager.CleanableManager, LOG.exception(msg, msg_args) v_res = volume.update_single_status_where( - 'available', 'reverting') + original_volume_state, 'reverting') if not v_res: msg_args = {"id": volume.id, "status": 'available'} diff --git a/cinder/volume/rpcapi.py b/cinder/volume/rpcapi.py index 58de69c1c..0b36cebdd 100644 --- a/cinder/volume/rpcapi.py +++ b/cinder/volume/rpcapi.py @@ -180,11 +180,11 @@ class VolumeAPI(rpc.RPCAPI): volume=volume) @rpc.assert_min_rpc_version('3.15') - def revert_to_snapshot(self, ctxt, volume, snapshot): + def revert_to_snapshot(self, ctxt, volume, snapshot, volume_state): version = self._compat_ver('3.15') cctxt = self._get_cctxt(volume.service_topic_queue, version) cctxt.cast(ctxt, 'revert_to_snapshot', volume=volume, - snapshot=snapshot) + snapshot=snapshot, original_volume_state=volume_state) def delete_volume(self, ctxt: context.RequestContext, -- 2.47.0

3 2

#eventlet-removal - HTTP SGI working group call for discussion
by Herve Beraud 19 Nov '24

19 Nov '24

tl;dr: We should have a cross discussion concerning the replacement of Eventlet.wsgi and we want to invite you to join the discussion and hence help us to define a standardized approach in a collaborative way. The Eventlet WSGI module can be used for 2 things: 1. as a WSGI server to serve any kind of web framework (flask, routes, etc...) [1] 2. and in some cases as a framework for web applications [2] Some OpenStack deliverables are using it in one way or another. Some deliverables like Glance already have credible migrations opportunities. But some of you may still be wondering what is your option to replace the Eventlet WSGI features? That's a good question. Now that the specs defining the removal of Eventlet from oslo.service are merged we have to define the way we want to proceed for the WSGI features. Indeed, we decided to drop the WSGI support from oslo.service. That means that we have to define, at least, a "default" way to replace it. If we want to avoid having multiple ways to manage things, we have to provide some guidance and standardizations. For this reason, after the PTG, we encouraged the creation of the HTTP SGI Working Group [4]. This working group aim to answer which (W|A)SGI servers are needed or credible alternatives. Which framework we have/want/need to use in parallel with the SGI server. The outcomes created by this group will serve as "a default" to guide the community through the replacement of the WSGI features of Eventlet. We prepared comparisons of the available alternatives for each layer: - https://wiki.openstack.org/wiki/SGI-servers-comparison - https://wiki.openstack.org/wiki/web-frameworks-comparison - https://wiki.openstack.org/wiki/Http-libraries-comparison Feel free to comment and to put your pros and cons on the given related etherpads (see the links in the comparison). This topic will be discussed at each irc meeting on the #openstack-eventlet-removal channel [5], until we find a default official option. Next meeting date is: 2024 December 3rd [6]. It would be good to reach an official consensus before the Milestone 3 of Epoxy. The goal is simply to ensure that our community looks in the same direction and that everyone is on the same page. We want to ensure that nobody is left behind. Feel free to reply to this thread or to put your comments directly on the various etherpads [6][7][8][9]. Thanks for reading. [1] https://eventlet.readthedocs.io/en/latest/reference/api/eventlet.html#modul… [2] https://eventlet.readthedocs.io/en/latest/modules/wsgi.html [3] https://review.opendev.org/c/openstack/oslo-specs/+/927503 [4] https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group [5] https://meetings.opendev.org/meetings/openstack_eventlet_removal/ [6] https://etherpad.opendev.org/p/epoxy-eventlet-tracking [7] https://etherpad.opendev.org/p/eventlet-removal-SGI-servers-comparison [8] https://etherpad.opendev.org/p/eventlet-removal-web-frameworks-comparison [9] https://etherpad.opendev.org/p/eventlet-removal-http-libraries-comparison -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

Kolla TLS
by devispain＠mail.com 19 Nov '24

19 Nov '24

Reading the Kolla docs it says: The back-end TLS cert/key can be the same certificate that is used for the VIP, as long as those certificates are configured to allow requests from both the VIP and internal networks. But there is no advice on how to do this, can someone explain how to use letsencrypt vip certs for backend tls, i have found nowhere that explains it at all. Trying to use a self signed cert for backend tls seems to break everything when using proper certs for vips

1 0

python3-dns 2.6.1 breaks Nova
by Francesco Di Nucci 19 Nov '24

19 Nov '24

Hi all, maybe this is already known (https://bugs.launchpad.net/glance/+bug/2054732) but also python3-dns-2.6.1 seems to break Nova and other services, with the error reported in the issue, that the DNS "answered udp() got an unexpected keyword argument 'ignore_errors';" It seems that Almalinux has the new version in the repos, while Rocky Linux still has 2.3.0 Best regards Francesco Di Nucci

2 2

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 19 Nov '24

19 Nov '24

Hello Stackers, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base The image includes several packages (such as Apache) that don’t appear necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though all containers don’t use them. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

Oslo Spec Review, Eventlet Removal Proposal.
by Daniel Bengtsson 19 Nov '24

19 Nov '24

Hi everyone, I hope you’re all doing well. During yesterday’s Oslo IRC meeting, the main topic was the removal of eventlet in oslo.service[1]. It seems that everyone is in agreement with this proposal, and no objections were raised. I would appreciate it if you could take another look and review the code. If we don’t receive any further objections by Monday, I propose we proceed with merging the spec. Thanks in advance for your help and feedback! [1] https://review.opendev.org/c/openstack/oslo-specs/+/927503?usp=dashboard -- Daniel Bengtsson Software engineer

2 1

[publiccloud-sig] Reminder: next meeting November 20th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 19 Nov '24

19 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 20th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> As I’ll be traveling, I’ll not be able to participate myself. felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0