January 2025 - openstack-discuss - lists.openstack.org

[publiccloud-sig] Public Cloud SIG Meeting this week
by Felix Kronlage-Dammers 28 Jan '25

28 Jan '25

Hi everyone, This upcoming Wednesday (January 29th) the next meeting of the publiccloud-sig is going to happen or at least would happen: However I’ll not be around tomorrow to facilitate and I did not get any feedback from anyone else on it. Usually we would be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig - so give it a shot ;) See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG felix -- Felix Kronlage-Dammers Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[ops][neutron] Neutron-db-manage upgrade from Antelope to Bobcat fails with "foreign key constraint 'porthints_ibfk_1' are incompatible" error
by federica fanzago 28 Jan '25

28 Jan '25

Dear all, we are running an OpenStack deployment Yoga on AlmaLinux 9.5 and we are attempting the upgrade Openstack up to Caracal release, one release at time (Zed--Antelope--Bobcat--Caracal) The procedure we are following is to shut down all openstack services, to remove the "old" openstack release and to install the new one, upgrading all service databases. This procedure works well for Zed and Antelope, but during the installation of the Bobcat release, the Neutron database upgrade fails when creating the new "porthints" table. The command we are using is su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron and the error message we receive is ... File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (3780, "Referencing column 'port_id' and referenced column 'id' in foreign key constraint 'porthints_ibfk_1' are incompatible.") [SQL: CREATE TABLE porthints ( port_id VARCHAR(36) NOT NULL, hints VARCHAR(4095) NOT NULL, PRIMARY KEY (port_id), FOREIGN KEY(port_id) REFERENCES ports (id) ON DELETE CASCADE )ENGINE=InnoDB ... Our "ports" table uses the collation utf8mb3_general_ci mysql> SHOW FULL COLUMNS FROM ports; +------------------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+ | Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment | +------------------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+ | project_id | varchar(255) | utf8mb3_general_ci | YES | MUL | NULL | | select,insert,update,references | | | id | varchar(36) | utf8mb3_general_ci | NO | PRI | NULL | | select,insert,update,references | | | name | varchar(255) | utf8mb3_general_ci | YES | | NULL | | select,insert,update,references | | | network_id | varchar(36) | utf8mb3_general_ci | NO | MUL | NULL | | select,insert,update,references | | | mac_address | varchar(32) | utf8mb3_general_ci | NO | | NULL | | select,insert,update,references | | | admin_state_up | tinyint(1) | NULL | NO | | NULL | | select,insert,update,references | | | status | varchar(16) | utf8mb3_general_ci | NO | | NULL | | select,insert,update,references | | | device_id | varchar(255) | utf8mb3_general_ci | NO | MUL | NULL | | select,insert,update,references | | | device_owner | varchar(255) | utf8mb3_general_ci | NO | | NULL | | select,insert,update,references | | | standard_attr_id | bigint | NULL | NO | UNI | NULL | | select,insert,update,references | | | ip_allocation | varchar(16) | utf8mb3_general_ci | YES | | NULL | | select,insert,update,references | | +------------------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+ in Bobcat the collation required for the new "porthints" table is latin1_swedish_ci MariaDB [neutron]> show create table porthints; +-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | porthints | CREATE TABLE `porthints` ( `port_id` varchar(36) NOT NULL, `hints` varchar(4095) NOT NULL, PRIMARY KEY (`port_id`), CONSTRAINT `porthints_ibfk_1` FOREIGN KEY (`port_id`) REFERENCES `ports` (`id`) ON DELETE CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci | So, we have tried to create the porthints table in mysql, forcing the collation to utf8mb3_general_ci mysql> CREATE TABLE porthints ( port_id VARCHAR(36) NOT NULL, hints VARCHAR(4095) NOT NULL, PRIMARY KEY (port_id), FOREIGN KEY(port_id) REFERENCES ports (id) ON DELETE CASCADE )ENGINE=InnoDB DEFAULT CHARSET=utf8 DEFAULT COLLATE utf8mb3_general_ci; and commenting out the creation of the new table in the neutron code /usr/lib/python3.9/site-packages/neutron/db/migration/alembic_migrations/versions/2023.2/expand/6f1145bff34c_port_hints.py. In this way, the upgrade works, but we encounter the same kind of problem during the upgrade to Caracal, specifically with the creation of the new table porthardwareoffloadtype. Have you already encountered this issue during the Neutron upgrade? Do you have any suggestions on how to upgrade the neutron DB, without manually creating new tables? Thanks a lot, cheers, Federica -- Federica Fanzago INFN Sezione di Padova Via Marzolo, 8 35131 Padova - Italy Tel: +39 049.967.7367 --

2 3

[nova] Performance Issues with OpenStack Server List on os-compute-api-version 2.96
by 黒羽正徳 [Kuroha Masanori] 28 Jan '25

28 Jan '25

Hello, We have recently encountered a situation where the response time for the openstack server list command is significantly slower when specifying os-compute-api-version 2.96 for certain projects. In comparison with version 2.95, some projects experience a response time that is 2-3 times longer, while other projects show little to no difference. We understand that in version 2.96, there was a change to obtain the availability zone from the request_spec [1]. However, upon checking the instance records in request_spec for the projects, we have not identified any differences that would account for this slow response behavior. Additionally, we have observed that when certain ERROR VM is present, the response time for the server list can become considerably slower for specific projects. In one case, we resolved the slow response issue by deleting instances that did not have corresponding fault records in the database. However, the exact reason why this action resolved the issue remains unclear. If anyone has more detailed information or potential solutions regarding these issues, or if you have encountered similar situations, we would greatly appreciate your insights. [1] https://git.linecorp.com/Flava-IaaS/nova/commit/c98c8d84eec6e65f1b67ae1ab82… Best regards, Masanori Kuroha

1 1

[largescale-sig] Next meeting: January 15, 9utc
by Thierry Carrez 28 Jan '25

28 Jan '25

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20250115T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-9)
by Goutham Pacha Ravi 28 Jan '25

28 Jan '25

Hello Stackers, We're 9 weeks away from OpenStack's 2025.1 coordinated "Epoxy" release [1]. These weeks leading up to feature freeze (R-5/Feb 24-28) will be about "getting things done" for project teams. As Slawek Kaplonski (slaweq) wrote to you [2], we'll be kicking off combined elections for Project Team Leads for the 2025.2 "Flamingo" release cycle and five seats on the OpenStack Technical Committee next week. In light of this, please confirm your eligibility to vote in these elections by navigating to the CIVS website and entering your Gerrit email address: https://civs1.civs.us/cgi-bin/opt_in.pl In each of the past three elections, we've had roughly a fifth of eligible voters cast their votes. It would make a great difference to our community if we could improve this statistic with your cooperation! In the past week, the OpenStack Technical Committee approved a resolution to welcome the "Freezer" (Disaster-Recovery-as-a-Service) project back into the fold of "active" projects [3]. I'd like to extend a shout-out to Dmitriy Rabotyagov (noonedeadpunk) and Álvaro Soto (khyr0n) for their work to revive the project team. The team is looking to pursue the release of several project deliverables during this release. === Weekly Meeting === The TC weekly IRC meeting on 2025-01-21 [4] was well attended. We were in "getting things done" mode too, as much of the meeting followed up on action items from past meetings. We discussed getting more active review participation for changes to the "openstackdocstheme" repository by working with the Oslo team. We also deliberated on election nomination windows. We are in favor of early nominations from folks who are unable to self-nominate due to holidays or other personal reasons to be away from their computers during the nomination window. However, there were concerns expressed that the election officials may be burdened by the work necessary to shepherd such nominations. Early nominations would fail several CI checks, require re-checks, and potentially create some confusion. We'll attempt to watch this closely for the next election cycle and recalibrate. Our goal is to avoid leaderless projects due to holiday cycles during elections. The TC also discussed the next steps for the newly active Freezer project to release artifacts during the current release cycle. The next meeting of the OpenStack TC is today, 2025-01-28, at 1800 UTC in the #openstack-tc IRC channel on OFTC. Please find the meeting's agenda on the meeting wiki [5]. I hope you'll be able to join us. === Governance Proposals === ==== Merged ==== - Retire Freezer DR | https://review.opendev.org/c/openstack/governance/+/938183 - Remove Freezer from inactive state | https://review.opendev.org/c/openstack/governance/+/938938 ==== Open for Review ==== - Retire qdrouterd role | https://review.opendev.org/c/openstack/governance/+/938193 - Reset the DPL model for Oslo project | https://review.opendev.org/c/openstack/governance/+/939485 - Reset the DPL model for Release project | https://review.opendev.org/c/openstack/governance/+/939486 - Add devstack-plugin-prometheus repo to Quality Assurance repos | https://review.opendev.org/c/openstack/governance/+/939953 - Reset the DPL model for Watcher project | https://review.opendev.org/c/openstack/governance/+/939488 - Add Watcher project update liaison | https://review.opendev.org/c/openstack/governance/+/940168 - Adopt DPL model for Watcher 2025.2 | https://review.opendev.org/c/openstack/governance/+/940167 - Define 2025 upstream investment opportunities | https://review.opendev.org/c/openstack/governance/+/939507 - Add resolution for late Freezer release | https://review.opendev.org/c/openstack/governance/+/939792 - Deprecate ironic-lib | https://review.opendev.org/c/openstack/governance/+/939278 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 === Upcoming Events === - 2025-01-28: OpenInfra Board Monthly Meeting: https://board.openinfra.org/ - 2025-02-01: FOSDEM 2025 (https://fosdem.org/2025/) OpenStack's 15th Birthday Celebration - 2025-02-05: Nominations open for 2025.2 TC/PTL Elections: https://governance.openstack.org/election/ - 2025-02-28: 2025.1 ("Epoxy") Feature Freeze and release milestone 3 [1] - 2025-03-06: SCALE 2025 + OpenInfra Days NA (https://www.socallinuxexpo.org/scale/22x) Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.2 Election Kickoff: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] Remove Freezer from inactive state: https://review.opendev.org/c/openstack/governance/+/938938 [4] TC Meeting IRC Log 2025-01-21: https://meetings.opendev.org/meetings/tc/2025/tc.2025-01-21-18.00.log.html [5] TC Meeting Agenda, 2025-01-28: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[mistral] meetings during next PTG
by Arnaud Morin 27 Jan '25

27 Jan '25

Hey all, Next PTG will take place between April 7 to 10. Is there anyone willing to talk about mistral developement during this PTG? So far, mistral has not evolved that much, but maybe someone is having ideas about it? Also, there is this eventlet-removal topic that could be discussed. Let me know, Cheers, Arnaud.

1 0

[neutron] Bug Deputy Report Jan 20 - Jan 26
by Yatin Karel 27 Jan '25

27 Jan '25

Hi Neutrinos, Below is the bug deputy report for last week:- High ------ - https://bugs.launchpad.net/neutron/+bug/2095807 - OVN loadbalancer via FIP does not work for DVR + VLAN networks Unassigned OVN octavia provider, Asked Fernando to have a look as could be known one - https://bugs.launchpad.net/neutron/+bug/2095598 - test_async_process_respawns functional test failing in fips functional job Unassigned We thought it would get fixed with https://review.opendev.org/c/openstack/neutron/+/939627 but it's not, so have to be checked again - https://bugs.launchpad.net/neutron/+bug/2095595 - The neutron-dhcp-agent resync throttling mechanism doesn't work Assigned to Anton Kurbatov - https://bugs.launchpad.net/neutron/+bug/2095590 - [OVN] Neutron API stops refreshing the hash ring node Assigned to Rodolfo - https://bugs.launchpad.net/neutron/+bug/2095566 - [OVN] Router creation timeouts randomly Unassigned Rodolfo mentioned he has some fixes in flight to cover ovn wsgi related issue, so to confirm if this is also covered. Medium ---------- - https://bugs.launchpad.net/neutron/+bug/2095576 - [OVS] False ERROR log message every time when creating VM Unassigned - https://bugs.launchpad.net/neutron/+bug/2095322 - ADMIN_OR_NET_OWNER_{MEMBER|READER} rules don't work for subnets Unassigned Incomplete --------------- - https://bugs.launchpad.net/neutron/+bug/2096704 - [RFE] Allow rebalancing of OVN LRPs Asked for more context -- Thanks and Regards Yatin Karel

1 0

[all][elections][ptl][tc] Combined PTL/TC 2025.2 cycle Election Season
by Sławek Kapłoński 27 Jan '25

27 Jan '25

Hi, Election details: https://governance.openstack.org/election/ The nomination period officially begins Feb 05, 2025 23:45 UTC. Please read the stipulations and timelines for candidates and electorate contained in these governance documents[1][2][3]. Due to circumstances of timing, PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. Similarly, there will only be a TC poll if there are more than 5 candidates stepping forward the available TC positions. All members of the community are encouraged to confirm their email address as described at: https://governance.openstack.org/election/#electorate There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you wish to discuss in private please email any of the election officials[4] so that we may address your concerns. Thank you, [1] https://governance.openstack.org/tc/reference/charter.html [2] https://governance.openstack.org/tc/reference/role-of-the-tc.html [3] https://docs.openstack.org/project-team-guide/ptl.html [4] https://governance.openstack.org/election/#election-officials -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[openstack-helm] Use Loci images by default
by Vladimir Kozhukalov 26 Jan '25

26 Jan '25

Hi, This is to announce multiple things: 1) OpenStack/Loci (a project for building OpenStack images) is currently equipped with periodic jobs that are run nightly. These jobs build images for 3 most recent OpenStack releases: - 2023.2 Bobcat - 2024.1 Caracal - 2024.2 Dalmatian Images are published to these two destinations: - docker.io/loci/* (which is known for its strict request rate limits) - quay.io/airshipit/* (which is much more tolerant to the number of requests) Images are tested before publishing with two basic deployment configurations: - compute-kit (Deploy minimal set of OpenStack components to run VMs) - cinder (Deploy Cinder with Ceph backend) 2) By default we are going to use Loci images published to the Quay registry [1]. All test jobs will be run with these images and this is also a recommendation for users to consume images from quay to mitigate possible rate limit issues. 3) OpenStack/OpenStack-Helm-Images project will build only non-OpenStack images. Primarily the motivation is to deduplicate the code base. Consequently, OpenStack images published nightly to docker.io/openstackhelm/* will not be updated after 2024.2. [1] https://review.opendev.org/c/openstack/openstack-helm/+/939426 -- Best regards, Kozhukalov Vladimir

2 2

Guidance Required for OVN-BGP-Agent Configuration with EVPN Mode
by AkashMadan Talebailkar 26 Jan '25

26 Jan '25

Dear Community, I hope this email finds you well. I came across this insightful discussion on the OpenStack forum regarding the OVN-BGP-Agent Discussio<https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030080.…>n. We are doing research on openstack capability and would greatly appreciate your guidance. We are utilizing the OVN mechanism driver along with the OVN-BGP-Agent<https://github.com/openstack/ovn-bgp-agent/tree/master> in EVPN mode. Our objective is to establish BGP-EVPN between our leaf switch and the OpenStack controller. The OVN-BGP-Agent should advertise the IP and MAC addresses of VMs to its BGP peers. Please find attached the following details for your reference: bgp-agent.conf Agent configuration OVN_DB configuration Result We installed the OVN-BGP-Agent and FRR on the controller. Below are the steps we followed: Network Setup: Created a tenant network (NET) with a subnet of 121.168.0.0/24. Created a provider network (EXNET) with a subnet of 192.27.5.0/24. Router Configuration: Set up a router (newrouter) with an external network (EXNET) and unchecked SNAT. Added the NET interface to the router. OVN Database Configuration: Configured the OVN Northbound DB (detailed in the attachment). VM Creation: Deployed a VM in NET with the VM's IP 121.168.0.202 PFA for the agent configuration and FRR configuration done once agent VM has been created. Despite following the steps above, the VM IP is not visible in FRR, PFA for the result. Consequently, we are unable to advertise its route via BGP-EVPN. We kindly request your guidance to identify any potential misconfigurations or missing steps. Thank you for your time and support. Best regards, Akash Talebailkar Palc Networks akashmadan.t(a)palcnetworks.com +91 9035376555

2 1

[manila][cephfs][kollla-ansible]cannot create access rule on share
by Nguyễn Hữu Khôi 26 Jan '25

26 Jan '25

Hello. I setup manila with cephfs driver-CephFS NFS shares and I use NFS-Ganesha based “ceph nfs” service. This is my configuration: [DEFAULT] debug = true enabled_share_protocols = NFS,CIFS,CEPHFS [cephfsnfs1] driver_handles_share_servers = False share_backend_name = CEPHFSNFS1 share_driver = manila.share.drivers.cephfs.driver.CephFSDriver cephfs_protocol_helper_type = NFS cephfs_conf_path = /etc/ceph/ceph.conf cephfs_auth_id = manila cephfs_cluster_name = az1 cephfs_filesystem_name = cephfs I can create a share but cannot set access rule for it and I encountered this error in manila-share.log: 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/ganesha/manager.py", line 472, in add_export 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server self._dbus_send_ganesha("AddExport", "string:" + path, 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/ganesha/manager.py", line 410, in _dbus_send_ganesha 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server self.execute("dbus-send", "--print-reply", "--system", 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/ganesha/manager.py", line 240, in _execute 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server raise exception.GaneshaCommandFailure( 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed. 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-5d49f7c6-27a0-43c3-931d-56486b8f3d51.conf string:EXPORT(Export_Id=106) 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server Exit code: 96 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server Stdout: '' 2025-01-25 22:00:11.895 774 ERROR oslo_messaging.rpc.server Stderr: '/var/lib/kolla/venv/bin/manila-rootwrap: Executable not found: dbus-send (filter match = dbus-addexport)\n' One more thing, when I run openstack share export location list myshare I see lab-control01:/volumes/_nogroup/5d49f7c6-27a0-43c3-931d-56486b8f3d51/fb094ad8-9200-4c93-b800-304178f51136 lab-control01 is my openstack controller, is it right because nfs service is on ceph cluster. Thank you. Nguyen Huu Khoi

1 1

[all][PTLs][release] OpenStack 2025.2 Flamingo release schedule plan
by Elõd Illés 24 Jan '25

24 Jan '25

Hi everyone, Let me inform you that 2025.2 Flamingo release schedule plan can be reviewed: https://review.opendev.org/c/openstack/releases/+/939528 (or for better reading you can take a look at the generated preview for Patch Set 1 [1] but note that the link **only** works for Patch Set 1!) [1] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/… Cheers, Előd Illés irc: elodilles @ #openstack-release

1 0

[kolla-ansible] Multipath unsuccesfull in nova
by Kamil Madac 24 Jan '25

24 Jan '25

Hello, I have a openstack deployed with kolla-ansible 2024.2. I'm trying to configure cinder and nova to use multipath when attaching volumes from the Hitachi SAN array. Without multipathing everything works good. When I enabled multipath in kolla-ansible (enable_multipathd: "yes"), deployment is without errors, volume can be created, but attachment of volume ends up with error in nova_compute container: ... 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] raise exc_type(*result[2]) 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Command: multipath -C dm-0 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Exit code: 1 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stdout: '' 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stderr: '3601817.750747 | /etc/multipath.conf does not exist, blacklisting all devices.\n3601817.750753 | You can run "/sbin/mpathconf --enable" to create\n3601817.750754 | /etc/multipath.conf. See man mpathconf(8) for more details\n3601817.751393 | sdd: error 2 in pathinfo, discarding path\n3601817.751501 | sde: error 2 in pathinfo, discarding path\n3601817.751504 | 360060e8012b142005040b14200000ee2: removing empty pathgroup 0\n3601817.751505 | dm-0: no usable paths found\n' When I manually added /etc/multipath.conf to nova_compute container, the attachment passed without any issues. I do not want to build special image only because of that single config file. My question is, how to make kolla-ansible to copy/add /etc/multipath.conf file into nova_compute service? Is there any mechanism for that? Thanks. Kamil Madac

3 4

[nova][neutron][ops] disabling the heal instance info cache periodic
by Sean Mooney 24 Jan '25

24 Jan '25

hi folks o/ This is a short PSA and request for comments The nova and neutron teams have discussed the existence and importantly the default behavior of nova's "heal instance info cache periodic" on an off for the better part of the last 4+ years but we never had time to take any action on it until now. Context for those who are not aware of it, nova has an instance info cache which we use to cache networking information such as ip address related to ports associated with an instance. This was introduced in the Essex release when nova networks was being replaced with quantum/neutron. Its rater fitting but over due that in the Epoxy release we are now turning this off. for more context i have captured some of the important point in the release note of the proposed change https://review.opendev.org/c/openstack/nova/+/939476/2/releasenotes/notes/d… But ill provide a TL;DR here. By default, as of essex, [compute]heal_instance_info_cache_interval defaulted to 60. it has been possible to disable this by setting the value to <=0 (recommended to use -1 by convention). when the periodic is enabled, every nova compute agent loop over the instance on the current host, retrieves all there ports form neutron and fully rebuilds the cache(as of stein). on a small cloud (less then 50 computes) this has minimal impact on the performance of the neutron server, on moderate to large clouds this creates 10s of MB of constant db read load. This has several side-effects: - first it creates unnecessary load on the neutron api and neutron db/network backend in creasing latency and reducing overall performance. - second, it create unnecessary load on rabbitmq as each compute service need to do an RPC to update the info cache(every 60 seconds). - third both of the first two waste power doing extra work that is not needed. - finally as of Icehouse this periodic has been optional because we added the os-server-external-events api so that neutron can tell nova if there has been a change to refresh the cache as needed. With that in mind hte nova team is proposing that we disable this going forward, for this release i have not deprecated the periodic so operators can re-enable it if they find its required for any reason. in the 2026.1 release we will likely revisit this based on operator feedback/bug reports. In this weeks team meeting we agree to wait 2 more weeks to get feedback from OPS if they had any concerns with this change hence this email. If anyone knows of a reason why we should not proceed with this default change feel free to let us know via this email thread or the gerrit review https://review.opendev.org/c/openstack/nova/+/939476 regards sean

2 1

April 2025 PTG Dates & Team Signup Kickoff
by Kendall Nelson 24 Jan '25

24 Jan '25

Hello Everyone, I am pleased to announce the next OpenInfra PTG[1] will be taking place April 7-11, 2025! Registration will open soon! To sign up your team, you must complete the survey[2] by February 21st at 7:00 UTC. We NEED accurate contact information (email and and IRC nickname ideally) for the moderator of your team’s sessions. This is because the survey information will be used to organize the schedule signups which will be done via the PTGBot. If you are not on IRC, please get setup[3] on the OFTC network and join #openinfra-events. You are also encouraged to familiarize yourself with the PTGBot documentation[4] as well. If you have any questions, please reach out to ptg(a)openinfra.dev! Information about signing up for time slots will be sent to moderators after the team signup deadline. Continue to visit openinfra.org/ptg for updates. -Kendall (diablo_rojo) [1] PTG Site: https://openinfra.org/ptg/ [2] Team Survey: https://openinfrafoundation.formstack.com/forms/ptg_april_2025_survey [3] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [4] PTGBot README: https://opendev.org/openstack/ptgbot/src/branch/master/README.rst

1 0

[neutron] Drivers meeting canceled Friday, January 24th
by Brian Haley 24 Jan '25

24 Jan '25

Hello Neutrinos: I'm going to cancel tomorrow's drivers meeting as there is no agenda. Just a reminder there are still open specs that need review, list is at [0]. Have a nice weekend! -Brian [0] https://review.opendev.org/q/status:open+project:openstack/neutron-specs

1 0

[cinder] 2025.1 Midcycle reminder
by Jon Bernard 23 Jan '25

23 Jan '25

Hello all, this is a reminder that our midcycle meeting will take the place of next week's upstream meeting on 29 Jan. I've allocated 2 hours (as opposed to our usual 1) in case additional time is needed. The etherpad we'll be using is here [1]. See you all then! [1]: https://etherpad.opendev.org/p/cinder-epoxy-midcycles -- Jon

1 0

[Neutron][operators] LinuxBridge to OVN Outreach
by Daniel Failing 23 Jan '25

23 Jan '25

Dear operators of Neutron deployments using LinuxBridge, This mail is to target operators who plan to move away from neutron LinuxBridge as well. In our setup we are currently planing to move away from LinuxBridge and use OVN instead. Based on the last PTGs survey with quite a few deployments using LinuxBridge and it being marked experimental for some time, we feel there could be a common ground and pain points. We were wondering how many other operators have the same problems and want to share knowledge and if necessary consolidate code written for a potential migration. Knowing there are quite some different deployments, we most likely will not be able to find the golden solution that works for everyone, but we could strive for not making the same mistakes multiple times in different deployments. We are located in European timezones, if there are others interested in a zoom/jitsi call, please let me know your timezones/time preferences, so we find a good slot. Otherwise I can propose one of the following dates already: Friday, Jan 24st 2 or 3pm UTC Tuesday, Jan 28st 10am UTC Tuesday, Jan 28st 3:30pm UTC Wednesday, Jan 29st 10am UTC Friday, Jan 31st 2 or 3pm UTC Best, Daniel CERN Cloud Infrastructure [1] original thread: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

3 3

[openstack-ansible]:Wallaby to Xena Upgrade
by Rambo Rambo 23 Jan '25

23 Jan '25

Hi Team I am planning openstack-ansible wallaby to Xena upgrade. Distribution: openstack-ansible Release: Wallaby (current) ==> Xena (Planned) I was checking the release notes of Xena and could see that there is introduction of SSL in several services with nova for live migrations one of them. https://docs.openstack.org/releasenotes/openstack-ansible/xena.html As per the release notes: - Nova now defaults to to using the “QEMU-native TLS” feature for live migrations, rather than the deprecated SSH method. A pre-existing PKI (Public Key Infrastructure) setup is required. QEMU-native TLS requires all compute hosts to accept TCP connections on port 16514 and port range 49152 to 49261. More information can be found here: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qem… Can you please confirm what steps are required to setup this PKI before doing the upgrade or it will be taken care by internal-pki using openstack-ansible upgrade method? Is there any option to skip this QEMU-native TLS for live-migrations and keep the orginal ssh based live-migrations? Regards

2 5

[all][oslo] revisiting the proposal to deprecate eventlet from oslo.service
by Mike Bayer 23 Jan '25

23 Jan '25

Hi - We've come to an impasse regarding our efforts to create a new backend selection system for oslo.service which will allow applications based on eventlet to have an incremental migration path where they can select for threads instead of eventlet [1] [2]. Briefly, the change organizes oslo.service into two different "backends", known as "eventlet" and "threaded". A calling application may choose to use either backend exclusively. Applications that use oslo.service right now are essentially using the "eventlet" backend. As these applications seek to migrate away from eventlet, they'd select the "threaded" backend and tune their applications to work under this new mode, where eventually the "eventlet" backend would be removed from oslo.service and ultimately from all of openstack entirely. All of that falls under things that have been agreed upon. The current impasse is among three options of how this "backend" system in oslo.service may be informed of which backend should be used: 1. being selected at runtime via CONF from the .cfg file , which is what was originally agreed upon 2. at start up time via environment variable 3. as a fixed setting in source code. Personally, as someone who has worked for decades on multiple switchable backend-oriented systems, I'm fully in the camp of choice 3, which is that switching the backend is just the beginning of a whole series of changes and tuning that a codebase would need in order to accommodate this backend change, and having the ability to "switch the backend back to eventlet" in CONF or runtime is simply not something anyone would want to do, unless a considerable amount of additional effort were made to support the codebase being reliable under both idioms. Which I would argue is a waste of effort since eventlet is going away. More concretely, if you took nova, which were using oslo.service with eventlet and decided to try the "threaded" backend, that's naturally going to have a lot of implications for other parts of the code. For example, if your code is setting eventlet monkey patching, that has to be turned off when this backend selection is made. If your code has other explicit use of eventlet, this also needs to be changed. Finally, threaded concurrency is naturally going to "feel" different than coroutine-based concurrency - context switches happen at different places and with different frequencies. Typically in a large codebase, the places that are prone to race conditions can shift; areas that had no races under coroutines now have threaded concurrency problems, areas that were sensitive to coroutine issues like being CPU bound are suddenly smooth with threading. The overall number of "working routines" can change, whereas eventlet can spin up 1000 coroutines like popcorn, under threads we now have to work with pools of threads typically a few dozen or so threads large. Overall, if I had my whole application passing under tempest under "threading", I would have had to make changes to get there, which may very well be mutually exclusive vs. test resiliency under eventlet. More generally, getting my application to run under threads represents a change in implementation that I expect to be permanent. If there are continued issues running under threaded mode, those are bugs that have to be fixed in any case. In the discussion at [1], Takashi Kajinami writes that a PTG discussion (for which I dont have any link or background) explicitly decided against the above idea: > In the initial design we considered that developer may make the decision to migrate from eventlet to the new mechanism, and didn't intend to provide any options to switch back to the original eventlet model. > However there was some strong push back during the previous PTG and people were concerned with having no mechanism to switch back to "previously worked well" eventlet model and to address the concern we agreed to introduce the option for operators to switch the backend. > If we don't allow users to configure the option then IMO we should not expose the option and we should use an internal flag instead (though that's basically what was disagreed with during the PTG, AFAIR) So first off, we have already realized that going with option 1. , use CONF, is not tenable, because the CONF collection is not typically in its final populated state when imports occur, and since the "backends" are an import time selection, there's a chicken/egg problem that cannot be resolved unless applications either highly modify the way in which their applications consume CONF vs. do their basic imports, or the way we are writing our backends in [1] needs to be modified so that imports can proceed using proxy objects which then change their loaded implementation when CONF is set up. Both of these options seem deeply complex and overarchitected in order to support the feature of "I want to change the backend in CONF" which I maintain is not going to be a real world use case. As a compromise, I suggested option 2. using an environment variable. This allows the backend selection to remain something that can be theoretically changed from the outside without modifying source code. However, current deployment practices suggest this is not really useful, as again per Takashi the way in which we deploy mod_wsgi does not allow environment variables to be local per virtual host; per [3] it looks like you need a hardcoded python script anyway that sets the env the way you want which makes it not too useful. As we were proceeding on option 2. we received some more pushback on the "env" idea from Arnaud Morin which I don't disagree with, and still more pushback and arguments leading in favor of 3, here from Jay Faulkner: > Using environment variables to set backends means that operators can change what backend is used. This is an easy thing for them to screw up and will end up creating a large amount of operator pain and bug churn. I am -1 in the strongest terms to the use of operator-adjustable values to enable/disable backends. The above was surprising considering that this was supposed to be a decision that had already been made at PTG, yet there still seems to be disagreement as though the decision were not actually made in any final way. So as I am tasked along with Daniel Bengtsson and Herve Beraud with getting [1] merged and moving onto building out the threaded backend, I would like to ask the group here to give me some background on the concerns raised at PTG and if we can all just here revisit the whole issue and hopefully decide that at least to start, let's get this merged without any CONF/env variable process (option 3); an application that's gone through the effort to transition to threads with the help of this backend selector should be assumed to be moving forward with that implementation, and if it has problems, that's just an ordinary bug like any other. thanks for reading! [1] https://review.opendev.org/c/openstack/oslo.service/+/935783 [2] https://review.opendev.org/c/openstack/oslo-specs/+/927503 [3] https://gist.github.com/GrahamDumpleton/b380652b768e81a7f60c

10 11

[kolla] Deprecating swift support
by Michał Nasiadka 23 Jan '25

23 Jan '25

Hello, Kolla and Kolla-Ansible is deprecating Swift deployment support, due to absence of active maintainers in the community for swift Kolla images and swift Kolla-Ansible role. If there are any volunteers to pick up that - please reach out either on #openstack-kolla or in the patch [1]. [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/939968 Best regards, Michal

1 0

[watcher] Bug Triage Meeting - January 14, 2025, 12pm UTC
by Ronelle Landy 23 Jan '25

23 Jan '25

Hello All,We will be holding a IRC/Google Meet meeting on Tuesday, January 14, 2025 at 12pm UTC to start the triage process for the list of current Watcher bugs: <https://bugs.launchpad.net/watcher> <https://bugs.launchpad.net/watcher> - <https://bugs.launchpad.net/watcher>https://bugs.launchpad.net/watcher - https://bugs.launchpad.net/watcher-dashboard - https://bugs.launchpad.net/watcher-tempest-plugin Anyone interested in the Watcher project is invited to join the meeting on: - IRC channel #openstack-watcher - Google Meet link: meet.google.com/drk-bgbx-uyz Notes from the meeting will be captured on the IRC channel.Thank you

1 3

[manila] Removal of the Standalone CephFS NFS protocol helper in Epoxy
by Ashley Rodriguez 23 Jan '25

23 Jan '25

Hi zorillas! As per subject line, we will be deprecating the standalone CephFS NFS protocol helper in this cycle. This is following our deprecation warning we put out in 2024.1 / Caracal [1] [2]. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the ceph-orch-deployed clustered NFS service; this code was backported to stable/2023.1 (Antelope). Although the upgrade path is disruptive, there is a UX improvement in Manila to ease this migration through surfacing “preferred” export paths. Moving forward, deployments must have “cephfs_nfs_cluster_id” in your configuration, otherwise the driver will fail to start. Manila will not make any export changes to NFS-Ganesha, so upgrading manila without an NFS cluster will mean that only the control plane is not going to work; but, data plane will remain unaffected. To assist migration, admins can also configure "cephfs_ganesha_export_ips" (or alternatively, "cephfs_ganesha_server_ip") alongside "cephfs_nfs_cluster_id". Setting these options will allow the CephFS driver to report additional export paths. These additional export paths will have the "preferred" metadata key set to False. Best, Manila team [1] https://review.opendev.org/c/openstack/manila/+/912648 [2] https://docs.openstack.org/manila/2024.2/configuration/shared-file-systems/…

5 5

[Glance]Cancelling Today's Weekly Meeting
by Mridula Joshi 23 Jan '25

23 Jan '25

Hello, As PTL and I are not around today , hence skipping today's weekly meeting . If someone has anything to discuss, ping Abhishek or Cyril on the IRC channel . Thanks, Mridula Joshi

1 0

[security-sig][ops][neutron] OVN Security issue can impact OpenStack users
by Jay Faulkner 22 Jan '25

22 Jan '25

Hi all, A recent security issue with OVN recently came to light and was fixed in a recent release. Full details are here: https://www.openwall.com/lists/oss-security/2025/01/22/5 . OpenStack deployers running OVN are strongly recommended to upgrade to the new fixed version. No CVE number has been issued yet for this resolved vulnerability. As a note, this was originally reported to the VMT as a neutron bug, but after triage it was responsibly reported to OVN and has been held under embargo. Thank you to the folks who participated in this triage process, including the OVN community for fixing the issue. Please see https://bugs.launchpad.net/ossa/+bug/2088280 for full details on OpenStack handling of this security report. Thanks, Jay Faulkner OpenStack VMT

1 0