April 2025 - openstack-discuss - lists.openstack.org

[kolla-ansible][rabbitmq] RabbitMQ cluster creation took too long
by Nguyễn Hữu Khôi 28 Apr '25

28 Apr '25

Hello. I am deploying OpenStack 2025.1 with Kolla-Ansible 2025.1 (master branch) using the latest images. However, when I created a RabbitMQ(4.0.x) cluster with 3 nodes, it took too long—around 12 minutes(it stucked at DB: virgin node -> run peer discovery). I then changed the images to RabbitMQ version 3.13.7, and it worked well. My OS: Ubuntu 24.01 Nguyen Huu Khoi

1 1

[nova] Did something change about interface detach/attach from Yoga to Caracal?
by Nell Jerram 28 Apr '25

28 Apr '25

I'm not sure why it was originally written, but I have an ancient system test that: - creates a VM attached to one network - finds the VM's port and calls interface_detach for it - checks that the VM is now unpingable - calls interface_attach with the relevant network_id - hard reboots the VM - checks that the VM is now pingable again. With Yoga, the test passes, and at the end of the test the VM has one NIC, eth0, with an IP address. With Caracal, the test also passes, but at the end of the test the VM has two NICs: - eth0 with no IP address - eth1 with an IP address. Diffing the Nova code, it looks like interface_detach and interface_attach might actually have been no-ops in Yoga - although possibly only for a "vdpa" vnic-type, and I'm unsure whether that vnic-type would be relevant to my test setup. Any ideas about this? Many thanks - Nell

2 3

[openstack-ansible] lxc-container upgrade from ubuntu 18 to ubuntu 20 for USSURI
by Danish Khan 27 Apr '25

27 Apr '25

Dear Team, I am trying to upgrade lxc-container from ubuntu 18 to ubuntu 20 and I doing it by destroying and recreating lxc-containers. I have successfully upgraded cinder, glance, heat, horizon,memcached, neutron, nova and placement containers. But now I am unable to create galera, utility containers. I am getting below error for both: failed: [host5_galera_container-ac20] (item={'url': ' http://repo.vexxhost.net/mariadb/key.gpg'}) => {"ansible_loop_var": "key", "attempts": 5, "changed": false, "key": {"url": " http://repo.vexxhost.net/mariadb/key.gpg"}, "msg": "Failed to download key at http://repo.vexxhost.net/mariadb/key.gpg: HTTP Error 404: Not Found"} And it seems the URL http://repo.vexxhost.net/mariadb/key.gpg doesn't exist now. Do we have any workaround for this? Regards, Danish

2 5

[neutron] Today's CI meeting cancelled
by Yatin Karel 27 Apr '25

27 Apr '25

Hello Neotron Team !! I will not be around today so cancelling CI meeting. Sorry for the last minute change. -- Thanks and Regards Yatin Karel

1 0

[neutron][ptg] 2025.2 Flamingo PTG summary
by Brian Haley 24 Apr '25

24 Apr '25

Hi all, Thanks for all that attended the meetings, we had a good turnout for Neutron, the Nova/Cinder cross-project and the eventlet removal discussions that were important to the team. For complete notes see the etherpad [0] but a summary is below. And please comment if you feel I missed anything. Thank you, -Brian ### Epoxy retrospective ### Like any other PTG, it started with a retrospective of the past cycle, with the highlights and improvement points. Good: - Eventlet removal is going well. Even the transition period (for the Neutron API with ML2/OVN) finished with good results. - Permanent "open doors" status to welcome newcomers. Our meetings (team meeting, drivers meeting) are a good forum for them. - Removed experimental code (Linuxbridge, IPv6 PD). - Migration from ML2/Linux Bridge to ML2/OVN discussions have started in Operators community (https://etherpad.opendev.org/p/neutron-lb-ovn-migration) Bad: - The loss of 2 important core developers. - No new specs merged in Epoxy cycle. ### Gate Stability ### Team has been very good at addressing gate issues quickly, and meets on a regular basis. We have noticed a few issues related to eventlet changes and are reverting them until we can address them. ### Spring cleanup / Code base modernization ### At the beginning of every cycle, we always try and cleanup the code base, for example, removing deprecated and dead code, doc updates, job consolidation. We have made progress on those already. - During Epoxy, several changes were made to match the expected output of pyupgrade/autopep8 for py39+. We will make a similar change in Flamingo for py310+ as that is now the lower bounds (haleyb). - It was also discussed to include "ruff" to our checklist (mlavalle). - Now that the OVN agent is in place and the default in gate, it was discussed about removing the OVN metadata agent as it is unnecessary to have both. - CI job reduction: move the ML2/OVS iptables-hybrid jobs to periodic (and experimental) and determine if iptables-hybrid driver can be moved to experimental config space. - TODOs: spend some time addressing the current TODO notes in the code, fixing what is proposed, already started. - Remove the unneeded OVN maintenance tasks, according to the comments in the methods. - Will make sure beginning/end of cycle docs are up-to-date on things like CI jobs, etc. (haleyb) ### Eventlet removal ### The "oslo.service" library will implement two backends (eventlet or threading), that will allow us to test both implementations during the removal. It could be useful to have temporary CI jobs, using both implementations. The testing CI (unit test, functional and fullstack) should start the migration during this cycle, but only once the Neutron service code is migrated. Please see [1] and [2] for more information and meetings notes from the eventlet-specific agenda. ### Migration to SDK (neutronclient removal) ### Etherpad link: https://etherpad.opendev.org/p/python-neutronclient_deprecation During the last cycle a number of Horizon patches merged (thanks Lajos!). Heat patches are still under review (and not attended). Nova patch is WIP: https://review.opendev.org/c/openstack/nova/+/928022 Neutron team will add deprecation warnings in the neutronclient code as there are still a number of places in both in and out-of tree code that is using the neutronclient. The fullstack tests will migrate to SDK during this cycle. ### Migrate the OVN L3 scheduler to use HA_Chassis_Group ### Link: https://bugs.launchpad.net/neutron/+bug/2092271 As recommended by the OVN core team, the usage of ``Gateway_Chassis`` to bind a ``Logical_Router_Port`` is deprecated. Instead of this, ``HA_Chassis_Group`` should be used. This bug tracks this effort that should be done during the next cycle. ### An OVN L3 router tool ### Link: https://bugs.launchpad.net/neutron/+bug/2103521 The goal of this tool is: - To be able to list the current GW LRP assignment, according to the ``HA_Chassis_Group``. It will show the GW chassis assignation level (according to the priority) and the number of routers per chassis. - To be able to reschedule the current assignments. In case of unbalance (chassis deletion, router deletion), this tool will allow to reschedule the GW chassis across the existing LRPs. This is something that is useful to cloud operators that currently must manually try and re-balance LRPs. Status: Approved ### Alembic idempotency ### Link: https://bugs.launchpad.net/neutron/+bug/2100770 This request comes from an issue seen in RHOSO18 with a DB migration back port. It is not possible to change the cycle milestones (last DB migrations of each cycle), thus the D/S migrations (in U/S are forbidden) are done before these milestones. The problem happens when a user, that is already in the last DB migration step, needs to re-execute the DB migration tool to receive the new back ported migration. That will fail because it is not possible to execute a DB migration script twice. This proposal includes: - Implementing a new test class that checks any new DB migration in order to ensure it works, as we found issues recently. - Enforce any new DB migration (code developers and reviewers) to be idempotent. - This would be supported on all existing migrations as well, not just from today forward. - Provide a usage guide on the alembic methods if necessary Status: Approved ### Stop DB drop migrations, with field drop exceptions ### Link: https://bugs.launchpad.net/neutron/+bug/2104160 In a DB migration, developers should add a TODO note to drop anything not used in the next SLURP release. That will prevent the case described in the bug, where a user executed the "expand" phase migration code with the old Neutron API code (as specified by the upgrade guide), resulting in an error due to a missing DB column, which was dropped by a DB migration script. The initial output of this bug is a new policy in the development guide. ### Create a restricted default policy for undefined API calls ### Link: https://bugs.launchpad.net/neutron/+bug/2098737 The goal of this RFE is to define a default restrictive rule for every non-defined policy in the Neutron API, as "oslo.policy" allows us to create a "default" policy that will be used in case the API call is not defined. The "default" Neutron policy is RULE_ADMIN_OR_OWNER, we propose to change this to RULE_ADMIN_ONLY. Any new API not defined in the policies would be executable only by the admin user. This change cannot be done now, due to the high number of missing API rules not defined that are using the default rule, see https://review.opendev.org/c/openstack/neutron/+/945687 - but this rule migration should be a background tasks for any team member that will improve the quality of the RBAC system in Neutron. Status: Approved ### Nova/Cinder cross-project ### We discussed two topics at the Nova/Cinder cross-project meeting. Project-Specific QoS Controls for Granular Resource Management. Driven by Bloomberg. Link: https://bugs.launchpad.net/neutron/+bug/2102184 Despite many questions still existing about the implementation, the Neutron team agreed to accept a spec describing it. The Neutron team agreed to have a per-project QoS policy that will affect to a non-defined set of resources (ports, networks, routers, FIPs). Bloomberg agreed to work on this later this cycle. For live-migration with OVN, Nova currently has a vif-plugged event timer to work-around any missed (or not sent) notifications. Link: https://bugs.launchpad.net/nova/+bug/2073254 The code for the multiple binding is in core OVN (23.09) and Neutron. Nova team will propose a patch to remove the condition in Nova and wait for the vif-plugged event when using the ML2/OVN backend as the condition should not happen any more. Patch link: https://review.opendev.org/c/openstack/nova/+/946950 ### Service role permissions ### The Octavia allowed address pair driver requires new service role permissions. Link: https://bugs.launchpad.net/neutron/+bug/2105502 Three patches proposed: - Add new default policy for device_id field on ports: https://review.opendev.org/c/openstack/neutron/+/861169 - Allow service role to create/update port device_id: https://review.opendev.org/c/openstack/neutron/+/947003 - Allow service role more RBAC access for Octavia: https://review.opendev.org/c/openstack/neutron/+/945329 ### L3/DHCP thread pool resizing ### Link: https://review.opendev.org/c/openstack/neutron/+/938411 This proposal is related to the eventlet removal topic. There is no direct replacement for ``eventlet.GreenPool``. The resizable mechanism does not provide a resource improvement in speed or memory and there is no kernel library that provides this functionality. The default and static maximum number of threads will be 32 (the current maximum). ### VXLAN physical networks ### Link: https://bugs.launchpad.net/neutron/+bug/2105855 The goal is to be able to schedule a VXLAN port depending on the physical mappings as the VTEP interfaces in each compute node could be connected to different physical networks. This would help improve the "networking-generic-switch" back end as it could use this new information. An alternative to this could be to implement a new type driver: l2vni, used for tunneled drivers (VXLAN) with a physical back end. A spec is required to study this proposal further. ### vlan_transparent ### Link: https://bugs.launchpad.net/neutron/+bug/2092174 Proposal to deprecate and remove the vlan_transparent config option. By default, the loaded drivers will determine if the functionality is supported or not. Status: Approved ### OVN "Add-Only" Sync Mode ### Link: https://bugs.launchpad.net/neutron/+bug/2099818 Proposal to add a new OVN sync mode option that will only add things to the OVN database and never remove from either OVN or Neutron. There were a few worries, as turning-off delete might just make things more out of sync, and not address any issues being seen in ovn-controller or ovn-northd. It was also pointed out that we did spend some time over past cycles tagging neutron-owned objects in OVN explicitly, and there might just be some additional edge cases we need to fix. This might address the issue where other objects change unexpectedly. Need to discuss further at future Drivers meeting as submitter did not attend. ### RFE discussions ### On-demand topic to discuss any in-progress or proposed RFE changes, please see etherpad for more information. [0] https://etherpad.opendev.org/p/apr2025-ptg-neutron [1] https://etherpad.opendev.org/p/neutron-eventlet-deprecation [2] https://etherpad.opendev.org/p/apr2025-ptg-eventlet

1 0

[watcher] Cancelling May 1st weekly meeting
by Douglas Viroel 24 Apr '25

24 Apr '25

Hi all, Since most participants will be out on May 1st, due to local holidays, we agree[1] to cancel next week's meeting. We will meet again on May 8th. [1] https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-04-24-12.00… BR, -- Douglas Viroel - dviroel

1 0

[openstack-ansible]: Operating system upgrade procedure Ubuntu 20.04 to 22.04
by Rambo Rambo 24 Apr '25

24 Apr '25

Hi Team We are currently on below release of Openstack: Openstack Release: Zed OS Release: Ubuntu 20.04.2 LTS I am looking for the tested procedure/suggestions/experience for doing the operating system upgrade from 20.04 to 22.04 on this openstack release. Regards Rambo

2 5

Deploying a DC - DR solution
by KK CHN 23 Apr '25

23 Apr '25

Folks , I am exploring the possibility of OpenStack to set up an On Prem DC and Near line DR using OpenStack . I have exposure to OpenStack from the Ussuri version onwards but am not a master of OpenStack. Objective is to achieve near zero data loss of the growing VMs at the DC replicated in real time to the Near Line DR. At times of a DC crash or for scheduled maintenance activity to ensure business continuity (RPO near zero and RTO 30 minutes - 1 Hour) to operate from the DR site. Heard about similar solutions exists for VMWare or any proprietary OEM like vSphere replication,vSAN and SRM to orchestrate failover and fallback with RPO near zero and RTO 30 minutes to 1 Hr. But not an option due to budget constraints at the moment. What ways do folks at OpenStack achieve this kind of requirement? The underlying tools and techniques I have chosen to deploy and the best possible design of DC and DR to achieve the objectives. Any hints and guidance to start, much appreciated. Thank you, Krishane

3 2

[horizon] Heat template generator not working on Chrome
by Jorge Merlino 23 Apr '25

23 Apr '25

Hi folks, There is an issue with Horizon's heat template generator when using chrome or a chrome based browser. It can be easily reproduced by going to the Project / Orchestration / Template Generator menu. There are icons on the top of the screen that can be dragged into the canvas below. That works fine in Firefox but does not work in Chrome. When the icon is dragged over the canvas in Chrome it shows a little green plus sign next to it which does not appear in Firefox and dropping the icon does nothing. I could not see any errors show in the browser console or the debugger. Tested on horizon 22.1.1 (openstack Yoga) Thanks for your help Jorge

2 2

cinder.volume.manager: RDB permission error (error creating image)
by usridzero＠gmail.com 23 Apr '25

23 Apr '25

Running Yoga on an isolated system. Everything seemed to have installed properly with openstack-ansible, but I'm getting the error: cinder.volume.manager: RDB permission error (error creating image). I can't get any logs from the isolated systems. Any help will be greatly appreciated.

1 0