November 2024 - openstack-discuss

[keystone] Weekly meeting canceled 27-Nov-2024
by Dave Wilde 26 Nov '24

26 Nov '24

Hello, Due to the Thanksgiving holiday in the US I'm canceling the weekly meeting and reviewathon this week. Please reach out here or on IRC if you need anything. Thanks! /Dave

1 0

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[openstack helm] [Ansible] Openstack-helm Ansible inventory.yaml missing some value
by daniel890723＠gmail.com 26 Nov '24

26 Nov '24

Hello today i'm testing installation with completely new enviroment i was following the guide from Openstack-helm document and install k8s cluster using ansible and deploy-env playbook but i encounter this error when setup ssh cluster TASK [deploy-env : Generate ssh key pair] ****************************************************************************************************************** skipping: [node-1] fatal: [primary]: FAILED! => {"msg": "Failed to change ownership of the temporary files Ansible (via chmod nor setfacl) needs to create despite connecting as a privileged user. Unprivileged become user would be unable to read the file."} and the log with -vvv output: debug2: mux_client_hello_exchange: master version 4 debug3: mux_client_forwards: request forwardings: 0 local, 0 remote debug3: mux_client_request_session: entering debug3: mux_client_request_alive: entering debug3: mux_client_request_alive: done pid = 114038 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 chown: invalid user: ‘zuul’ debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 And by checking the playbook we can see the playbook is using "{{ client_ssh_user }}" on primary node and "{{ cluster_ssh_user }}" with cluster node which is not defined in inventory.yaml Causing it using default user zuul by adding client_ssh_user & cluster_ssh_user in ansible inventory it works normally Is this meant to be like this? or the document missing some variables in inventory.yaml

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-18)
by Goutham Pacha Ravi 26 Nov '24

26 Nov '24

Hello Stackers, We're 18 weeks away from the coordinated release for 2025.1 ("Epoxy") [1]. We're beginning a period of low activity due to year-end holidays across the world. Last week, the TC didn't merge any new governance proposals; however, several are currently in progress. === Weekly Meeting === The TC met in OFTC's #openstack-tc IRC channel on 2024-11-19 [2]. We discussed a series of global requirements upper constraint bumps that were committed and the ensuing CI flakiness. We appreciated the effort to proactively update dependencies to their latest versions earlier in the release cycle so we can have more time to test and stabilize OpenStack software. Sometimes, these failures may interrupt other project priorities. The best way we can help is if project teams identify a relatively quick-running and stable integration job to run against changes to the global requirements repository. The TC and OpenDev administrators worked with Launchpad administrators to regain control of trackers belonging to the Watcher project team. It was noted that Launchpad trackers for any project should not be owned/administered by individuals. Instead, there must be a Launchpad team driving all trackers associated with an OpenStack initiative, and "openstack-admins" must own this team. This approach ensures the sustainability of project teams and protects projects from the impact of changing individual priorities. We also briefly discussed the "eventlet-removal" goal proposal. The goal document is being rewritten [3] to simplify what is being done and the timelines we're aiming for. I'd love for the community to take a look at this change and review it. The next meeting of the OpenStack Technical Committee is today, 2024-11-25, at 1800 UTC in OFTC's #openstack-tc channel. The agenda for this meeting is in our meeting wiki [4]. I hope you're able to join us. === Governance Proposals === ==== Open for Review ==== Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 ==== Merged ==== Fix doc job for pillow 11.0.0 | https://review.opendev.org/c/openstack/governance/+/935967 Add Ubuntu Noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meetings: Join us at our weekly meeting. The Technical Committee meets every week on Tuesdays at 1800 UTC [4]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting IRC Log 2024-11-19: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.log.html [3] Eventlet removal Community goal: https://review.opendev.org/c/openstack/governance/+/931254 [4] TC Meeting Agenda, 2024-11-26: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[docs][dev] replace_global_vars not working in project-deploy-guide
by Jakub Darmach 26 Nov '24

26 Nov '24

Hello everyone, I've been updating documentation in kolla-ansible and noticed that while replacing variable content works in https://docs.openstack.org/kolla-ansible it fails to replace the variable with proper content in https://docs.openstack.org/project-deploy-guide/kolla-ansible. If I understand correctly, it's being substituted with var having proper content in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… Later on used in https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… And now for the example - |KOLLA_BRANCH_NAME| variable: Working substitution: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#t… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… Failed to render content: https://docs.openstack.org/project-deploy-guide/kolla-ansible/latest/quicks… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… The var seems to be used the same in both examples, I suspect slightly different configuration on project end of the things. Can anyone advise where should I start looking? Best regards, Jakub

2 1

[neutron] Bug deputy report for week of November 18th
by Brian Haley 26 Nov '24

26 Nov '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - https://review.opendev.org/c/openstack/neutron/+/935804 High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2088453 - [UT] Neutron tests failing with eventlet 0.37.0 - https://review.opendev.org/c/openstack/neutron/+/935524 * https://bugs.launchpad.net/neutron/+bug/2089157 - [neutron-specs] CI job "openstack-tox-docs" broken - https://review.opendev.org/c/openstack/neutron-specs/+/935836 * https://bugs.launchpad.net/neutron/+bug/2089250 - neutron-vpnaas: Automatic VPN Agent failover not working with OVN driver - needs owner * https://bugs.launchpad.net/neutron/+bug/2089252 - neutron-vpnaas: LibreSwan 4 connections not listed as ACTIVE - Submitter said he would look this week Medium ------ * https://bugs.launchpad.net/neutron/+bug/2089169 - [OVN] Test ``TestCreateNeutronPgDrop.test_non_existing`` is randomly failing - https://review.opendev.org/c/openstack/neutron/+/935802

1 0

[cinder] Meeting canceled this week, 27 Nov
by Jon Bernard 26 Nov '24

26 Nov '24

Hello all, schools are closed this week so I won't be able to host the weekly meeting due to the holidays. I'll see you all the following week. Take care, -- Jon

1 0

[horizon][ceph] object storage container storage policy missing
by Satish Patel 25 Nov '24

25 Nov '24

Folks, Openstack Release: Zed Recently I have integrated ceph RGW service with openstack/keystone and testing with command line tool "openstack container create foo1" and it works but when I go to Horizon UI I am seeing it doesn't have storage policy dropdown list and because of that I am not able to create container or upload file from Horizon GUI https://ibb.co/KhYW6Rd It works with commands like but not horizon. $ openstack container create foo1 +---------------------------------------+-----------+------------------------------------------------+ | account | container | x-trans-id | +---------------------------------------+-----------+------------------------------------------------+ | AUTH_d4d82849885e4126aef402c408ad2c5c | foo1 | tx00000c057454841b13497-0065b558b4-42ff2eb-ahm | +---------------------------------------+-----------+------------------------------------------------+ $ openstack container list +------+ | Name | +------+ | foo1 | +------+

2 2

[Neutron] - Unable to ping my instance from my Openstack controller node
by ABATCHA OLLOH 23 Nov '24

23 Nov '24

Hello Everyone, Could you give me any advice and help me to better understand neutron. On my VirtualBox VM I properly installed Openstack using packstack (all-in-one installation). I have access to the horizon dashboard. I'm able to launch an instance and associate floating IP but from the controller node I cannot reach my instance. - Here are my interfaces config: - enp0s3: for mgmt - enp0s8: for openstack services - enp0s9: for neutron_external_interface (used as a public interface) - enp0s10: interface on my controller to be connected to the neutron_external_interface - VirtualBox Promiscuous mode : Allow All [root@packstack ~(keystone_admin)]# ip -br -c a lo UNKNOWN 127.0.0.1/8 ::1/128 enp0s3 UP 9.10.93.8/24 fe80::a00:27ff:fe2e:150a/64 enp0s8 UP 9.11.93.8/24 fe80::a00:27ff:fec7:56ab/64 enp0s9 UP fe80::a00:27ff:fef9:3cc7/64 enp0s10 UP 9.12.93.15/24 fe80::a00:27ff:feff:3641/64 ovs-system DOWN br-tun DOWN br-int DOWN br-ex UNKNOWN 9.12.93.8/24 fe80::b021:85ff:fe8a:9d44/64 qbr9eefea66-89 UP qvo9eefea66-89@qvb9eefea66-89 UP fe80::1409:2aff:feb4:e37d/64 qvb9eefea66-89@qvo9eefea66-89 UP fe80::8c84:15ff:fe7d:8896/64 tap9eefea66-89 UNKNOWN fe80::fc16:3eff:fec6:5f5c/64 - Security Group Rules associated to my instance - Egress IPv4 Any Any 0.0.0.0/0 - Ingress IPv4 ICMP Any 0.0.0.0/0 - Ingress IPv4 TCP 22 (SSH) 0.0.0.0/0 - Ingress IPv4 TCP 80 (HTTP) 0.0.0.0/0 - Instances status [root@packstack ~(keystone_admin)]# openstack server list +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | testrocky | ACTIVE | private1=10.0.0.202, 9.12.93.203 | N/A (booted from volume) | m1.devops | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ - From the router inside my openstack I can ping the floating ip [root@packstack ~(keystone_admin)]# ip netns list qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. 64 bytes from 9.12.93.203: icmp_seq=1 ttl=64 time=6.67 ms 64 bytes from 9.12.93.203: icmp_seq=2 ttl=64 time=1.51 ms9.12.93.203 [root@packstack ~(keystone_admin)]# ip netns qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ip route default via 9.12.93.1 dev qg-68351942-28 proto static 9.12.93.0/24 dev qg-68351942-28 proto kernel scope link src 9.12.93.201 10.0.0.0/24 dev qr-5ba4fd5b-b5 proto kernel scope link src 10.0.0.1 - Trying to reach the floating IP from the controller node [root@packstack ~(keystone_admin)]# ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. From icmp_seq=1 Destination Host Unreachable ...9.12.93.2039.12.93.8 - Check floating ip status [root@packstack ~(keystone_admin)]# openstack port list --network private1 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | 5ba4fd5b-b5e1-4f56-b681-963cf2c9ac8a | | fa:16:3e:14:0b:4a | ip_address='10.0.0.1', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | 9eefea66-897b-4682-883c-b61e22c3c09a | | fa:16:3e:c6:5f:5c | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | a73cd16c-8480-4461-99f2-7837e10f27b1 | | fa:16:3e:91:5f:98 | ip_address='10.0.0.200', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 9eefea66-897b-4682-883c-b61e22c3c09a +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | packstack.example.local | | binding_profile | | | binding_vif_details | bound_drivers.0='openvswitch', bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2024-11-15T11:38:07Z | | data_plane_status | None | | description | | | device_id | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | | hardware_offload_type | None | | hints | | | id | 9eefea66-897b-4682-883c-b61e22c3c09a | | ip_allocation | None | | mac_address | fa:16:3e:c6:5f:5c | | name | | | network_id | 2a02741e-35f0-4a61-81b0-abd4b5a09f36 | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | bbba2507099c4209a529acd64352939e | | propagate_uplink_status | None | | resource_request | None | | revision_number | 10 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | 2a206a8d-fb54-4fa7-9db7-e5f197dcc107 | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2024-11-18T08:15:49Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ [root@packstack ~(keystone_admin)]# openstack port list --network public1 +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | 2b215f41-edf8-4c61-8969-383143340444 | | fa:16:3e:30:7e:08 | ip_address='9.12.93.200', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | fa:16:3e:28:39:a9 | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | N/A | | 68351942-28a1-4df3-8661-bf157fcd5982 | | fa:16:3e:bf:66:56 | ip_address='9.12.93.201', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 31d7b194-50a0-4a25-b102-542210e5f3f3 +-------------------------+----------------------------------------------------------------------------+ | Field | Value | +-------------------------+----------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2024-11-15T15:57:42Z | | data_plane_status | None | | description | | | device_id | 3dc9d9c3-28eb-4dfb-a41b-9bbfac9f96da | | device_owner | network:floatingip | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | | hardware_offload_type | None | | hints | | | id | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | ip_allocation | None | | mac_address | fa:16:3e:28:39:a9 | | name | | | network_id | bcc1c132-074f-45d5-a715-a2d371cdb5be | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | | | propagate_uplink_status | None | | resource_request | None | | revision_number | 2 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | | | status | N/A | | tags | | | trunk_details | None | | updated_at | 2024-11-15T15:57:43Z | +-------------------------+----------------------------------------------------------------------------+ - I couldn't find something related to the binding port in these logs file. - tail -f /var/log/neutron/openvswitch-agent.log - tail -f /var/log/neutron/server.log - tail -f /var/log/neutron/l3-agent.log [image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…> Any advise will be welcome

2 2

Live Migration problems
by Gilles Mocellin 23 Nov '24

23 Nov '24

Hi Stackers, I have 2 recurring problems with live migration. If someone already face that and have recommendations... My env : - OpenStack 2023.1, deployed by OpenStack-Ansible 27.5.1 - Network : ML2/LinuxBridges - Some shared Networks on a VLAN provider - Most private networks on VXLAN - OS : Ubuntu 20.04 - libvirt-daemon-system: 8.0.0-1ubuntu7.10~cloud0 - qemu-system-x86: 1:4.2-3ubuntu6.30 Each time we have maintenance to do on compute nodes, we live-migrate the instances, with `nova host-evacuate-live` (still no openstack CLI command to do that). 1. Never ending / failed migrations In general, it works, but each time we have 1 or 2 (on 40 instances) that don't want to complete. It's some memory active instances (Kubernetes controller nodes with etcd...). Even if I had configured live_migration_permit_auto_converge to True. Now, if we try to force complete (openstack server migration force complete), migration fails, instance return to ACTIVE status on source. And I can see libvirt errors : [...] libvirt.libvirtError: operation failed: migration out job: unexpectedly failed Not really helpful. I tried removing live_migration_permit_auto_converge, reduce live_migration_completion_timeout to 60 and set live_migration_timeout_action to force_complete. Same problems. 2. Instance unreachable More problematic, as it's not easy to see : some instances are not reachable after migration. And after a while, DHCP leases expire. The only way to recover is to restart neutron-linuxbridge-agent on the instance compute node. In fact, on all compute nodes, because instances evacuation spread them on all the compute nodes. Then, if DHCP has not expired, the instance is joinable again. We made a script to find those instances, it lists all ports bound to a nova instance. Then connects to each control/network nodes, enter qdhcp network namespaces (found with the MAC adresse in DHCP leases files), and try to ping the instance private IP. The only log I see, but don't know if it has something to do with that is : Device tapxxx-yy has no active binding in host But I can see that message even since, without unjoinable instances... Here it is, if something ring a bell, for an existing bug or configuration error, please let us know ! Thanks. -- Gilles

3 4