March 2025 - openstack-discuss - lists.openstack.org

[oslo] python-consul2 is unmaintained
by Takashi Kajinami 21 Apr '25

21 Apr '25

Hello, I recently noticed that python-consul2 library[1], which is used by the consul coordination driver in tooz, hasn't been updated for 3+ years. Looking at issues in the project, there is an indication that the library is not compatible with ACL API in consul >= 1.11[2]. There is another fork created[3] and this includes fixes to support new API endpoints. Although the driver in tooz does not directly use the feature, I think it's better to discuss our plan to mitigate risks caused by the unmaintained library. As far as I can think of, there are three options now 1. Deprecate consul driver and remove it 2. Keep using python-consul2 now. At least basic functionality is proven to work by our functional tests (Note that the tests do not cover all configuration patterns) 3. Replace python-consul2 by py-consul Before we start the overall discussions I'd like to ask a few questions to the wider audience, as our inputs. - Is anyone using the consul coordination driver now ? - Does anyone have experience with python-consul2 and/or py-consul ? Note that this discussion affects the http messaging driver[4] currently proposed, because it also requires client library to communicate with Consul. Actually I noticed the situation when I noticed that the requirement check job is failing because the library is not part of global requirements list... Thank you, Takashi Kajinami [1] https://github.com/poppyred/python-consul2 [2] https://github.com/poppyred/python-consul2/issues/42 [3] https://github.com/criteo-forks/py-consul [4] https://review.opendev.org/c/openstack/oslo.messaging/+/912499 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 3

[octavia] Purging ERROR'ed amphora and load balancers stuck in transitional states
by Paul Browne 11 Apr '25

11 Apr '25

Hello Octavia users, For a while I've been trying to find a systematic way to remove Octavia LBs+Amphora that have become stuck in ERROR or transitional PENDING_* states ; $ openstack loadbalancer amphora list -f value | grep ERROR a2323f14-3aaa-418a-8249-111aaa9c21fe 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR MASTER 10.8.0.242 192.168.3.247 e5b236ba-e7ee-4ed7-9f58-57ce7a408489 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR BACKUP 10.8.0.190 192.168.3.247 6b556f28-93c9-49dd-b6ee-4379288e7957 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR MASTER 10.8.0.39 192.168.3.126 c669db5d-8686-4d5c-9e95-e02030b34301 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR BACKUP 10.8.0.174 192.168.3.126 $ openstack loadbalancer list -f value | grep -vi active 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe k8s-clusterapi-cluster-default-ci-6386871107-kube-upgrade-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.247 PENDING_UPDATE amphora d5e402fe-2c4b-49af-a700-532cb408cee5 k8s-clusterapi-cluster-default-ci-6386871107-latest-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.126 PENDING_DELETE amphora These resources are marked immutable and so cannot be failed over or deleted; $ openstack loadbalancer amphora failover a2323f14-3aaa-418a-8249-111aaa9c21fe Load Balancer 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe is immutable and cannot be updated. (HTTP 409) (Request-ID: req-6e66c4e8-c3d3-4549-a03c-367017c8c8b3) $ openstack loadbalancer failover 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe Invalid state PENDING_UPDATE of loadbalancer resource 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe (HTTP 409) (Request-ID: req-84b44212-e7a8-4101-a16f-18c774c0577e) The backing Nova instances for these Amphora do seem to exist and be in good working order. Is there any API way to purge these out of Octavia's service state, or would (very careful) DB hackery be required here? Many thanks, Paul Browne ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 6

GPU PCI passthrough woes.
by Andy Speagle 09 Apr '25

09 Apr '25

Hey Folks, I could use a little assistance getting GPU passthrough working. I had this working already for one flavor of nvidia gpu... and I've added some hosts with a much newer gpu... I've updated the pci_alias and pci_passthrough variables and those seem to be getting set properly in nova.conf passthrough_whitelist = [{"vendor_id":"10de", "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": "10de"} I believe I have all of the iommu stuff configured and have the pci- stub module entries... dmesg output shows that the GPUs are being claimed by the stub module. $ openstack flavor show t1.small_gpu_l40s +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 0 | | id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | | name | t1.small_gpu_l40s | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='gpu-l40s:1' | | ram | 2048 | | rxtx_factor | 1.0 | | swap | 0 | | vcpus | 1 | +----------------------------+--------------------------------------+ Yet... I can't seem to get an instance to run using that new flavor... keeps complaining about there not being enough hosts available. Fault: code: 500 created: 2024-10-30T02:04:33Z message: "No valid host was found. There are not enough hosts available." details: | Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Any clues on how/where to dig into this more to see what might be missing? Thanks. -- Andy Speagle

3 2

[all] OpenInfra Summit Europe Scheduled for October 2025
by Jeremy Stanley 08 Apr '25

08 Apr '25

In case anyone hasn't seen the news already, OpenInfra Summit Europe has been officially scheduled for October 17-19, 2025, at École Polytechnique near Paris, France: https://openinfra.dev/blog/openinfra-summit-2025 The OpenInfra Foundation is currently recruiting interested individuals to join a Programming Committee kickoff call on Thursday (2025-03-27). This call will be used to plan the conference tracks for the OpenInfra Summit ahead of the CFP, and will greatly benefit from OpenStack's involvement as always. If you would like to participate in the Programming Committee or assist in any other way, please email Helena Spease <helena(a)openinfra.dev>. I am also happy to try and point people in the right direction if you have any questions and prefer to reach out to me. On behalf of the foundation, we’re excited to bring the OpenInfra community together in France, and sincerely hope you'll join us! -- Jeremy Stanley

1 1

[manila]NFS-Ganesha exports
by Mikhail Okhrimenko 06 Apr '25

06 Apr '25

Hello community, Trying to setup the Manila version of Caracal with NFS-Ganesha backend. Ganesha version 4.0 is deployed using ceph orch in a ceph cluster version 17. Backend configuration in manila.conf [ganesha_nfs] driver_handles_share_servers = False share_backend_name = ganesha_nfs Share_driver = manila.share.drivers.cephfs.driver.CephFSDriver cephfs_protocol_helper_type = NFS cephfs_conf_path = /etc/ceph/ceph.conf cephfs_auth_id = manila cephfs_cluster_name = ceph cephfs_filesystem_name = {{ cephfs_filesystem_name }} cephfs_nfs_cluster_id = {{ cephfs_nfs_cluster_id }} I see the subvolume being created in ceph ceph fs subvolume ls <volume_name> but no export ceph nfs export ls <cluster_id> what am I doing wrong? what am I missing? in the manila-share logs I see a request to create a subvolume, but I don't see a request to create an export, for some reason manila doesn't even try to create an export

3 6

bug report
by Xavier Benjamin 05 Apr '25

05 Apr '25

whenever i try to connect to trove pts servers it would usually say fails to authenticate or something. Can you fix that please?

2 2

[octavia][nova][neutron] failed to failover loadbalancer
by Winicius Allan 02 Apr '25

02 Apr '25

Hello stackers! release: zed deploy-tool: kolla-ansible After an outage, all the load balancers from my cluster came to provisioning status ERROR because the o-hm0 interface was unavailable on controller nodes. I've created them again and tried a failover on load balancers. The octavia-worker logs show that failover was completed successfully, but the provisioning status remains with ERROR. Looking into nova-compute logs, I see that an exception was raised os_vif.exception.NetworkInterfaceNotFound: Network interface qvob8ac0f5f-46 not found The instance id for that interface matches the new amphora id. I'll attach the nova-compute logs here[1]. In neutron-server logs, there is no ERROR or WARNING that I can found, the only suspicious entry is Port b8ac0f5f-4613-4b95-9690-7cf59f739fd0 cannot update to ACTIVE because it is not bound. _port_provisioned /var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/plugin.py:360 Can anyone give me a light? [1] https://pastebin.com/hV3xgu0e

2 6

[nova] nova-manage fails when upgrading to Antelope
by Roby G 01 Apr '25

01 Apr '25

Hello everyone, I´m trying to upgrade to Antelope and have a problem when executing the db migrations. `nova-manage db sync` fails when upgrading from Zed to Antelope (same for Yoga --> Antelope) - Nova: 27.4.0 - Database: PostgreSQL 16 - Operating System: Ubuntu 22.04 - OpenStack Zed --> Antelope Full Log. /var/log/nova-manage.log: ``` 2025-03-04 11:42:14.963 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Context impl PostgresqlImpl. 2025-03-04 11:42:14.964 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Will assume transactional DDL. 2025-03-04 11:42:14.967 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Running upgrade ccb0fa1a2252 -> 960aac0e09ea, de-duplicate_indexes_in_instances__console_auth_tokens 2025-03-04 11:42:14.970 2519974 WARNING oslo_db.sqlalchemy.exc_filters [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] DBAPIError exception wrapped.: psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid constraint consoles_instance_uuid_fkey on table consoles depends on index uuid constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid constraint migrations_instance_uuid_fkey on table migrations depends on index uuid constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters self.dialect.do_execute( 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint consoles_instance_uuid_fkey on table consoles depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint migrations_instance_uuid_fkey on table migrations depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters ``` Any suggestions? Thanks!

3 3

[Octavia]
by Vincent Godin 01 Apr '25

01 Apr '25

Hello Octavia Team, We have an Openstack infrastructure with 3 Nova AZ and 3 Cinder AZ. Furthermore, Octavia will use cinder for Amphora disk. Is there a way to create an Amphora so that the Compute AZ match the Cinder AZ ? As the storage is AZ specific, it's not possible to launch an Amphora on a Compute AZ with a storage on an alternate AZ. As a client, there is no need to fix where the Amphora will be launched, we just want to be sure that if the Amphora is planned to be launched on a specific AZ, its storage will be selected on the corresponding AZ too ! Vincent

3 2

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R+0)
by Goutham Pacha Ravi 31 Mar '25

31 Mar '25

Hello Stackers, The release team will be tagging the GA release for all cycle-with-rc projects for 2025.1 ("Epoxy") on 2025-04-02 [1][2]. Remarkably, this will be the 31st release of OpenStack and one of two that will mark 15 years of the community! There will be an OpenInfra Live event on 2025-04-03, where a few project maintainers will reflect on how the release cycle went, what it will bring to users, and project priorities for the 2025.2 ("Flamingo") release cycle [3]. Alongside project teams, the OpenStack Technical Committee has been preparing for the upcoming virtual Project Teams Gathering. We're still gathering topics, so please add yours if you're interested in seeding a relevant discussion. You can find all the discussion etherpads linked on the PTG Website [4]. In the past week, we completed the community testing goal to upgrade CI integration testing ("migrate-to-noble") [5]. Kudos to our goal champion, Ghanshyam Mann (gmann), and to everyone who helped with the effort! === Weekly Meeting === The last weekly meeting of the OpenStack TC was held over IRC in OFTC's #openstack-tc channel on 2025-03-25 [6]. We added a discussion topic on operator engagement to the os-operators' PTG slot. We caught up on CI issues that needed attention from several quarters, including a kernel regression that prevented release-critical patches from merging in Neutron and Kolla and warranted a workaround. A breaking change in Setuptools 78 impacted many projects on Zuul; however, OpenStack was minimally affected before the issue was resolved in Setuptools. A broken CentOS 9 mirror briefly caused problems due to an inconsistent state from an upstream source. It was brought to the TC's attention that the Skyline project commits pre-built static files (e.g., JavaScript bundles generated using Yarn) directly into its repository. This manual intervention creates a potential security risk similar to the XZ backdoor attack since there is no verifiable way to check whether these builds contain malicious code. The TC brainstormed that it would be better to offload the build process to Zuul, preventing manual intervention, and to ensure the yarn.lock file is always included in releases for better traceability. Operators and project contributors must be aware of this issue and the fact that builds are currently not reproducible, complicating packaging. There was a suggestion that OpenStack governance should enforce standardized project security practices. Since we haven't identified concrete steps to resolve the current issue, we'll revisit this topic in a future discussion. The next meeting of the OpenStack TC is today, 2025-04-01, at 1700 UTC. Please note the change in timing—it is an hour earlier than our past meetings. This meeting will be held simultaneously on Video (Zoom) and IRC (OFTC's #openstack-tc channel). Please find the conference details and agenda on the meeting's wiki page [7]. I hope you'll be able to join us there. === Governance Proposals === ==== Merged ==== - Mark CI migration to Ubuntu Noble goal complete | https://review.opendev.org/c/openstack/governance/+/945641 ==== Open for Review ==== - [resolution] Extend scope of VMT to cover all project teams | https://review.opendev.org/c/openstack/governance/+/944817 === Upcoming Events === - 2025-04-01 - 2025-04-04: KubeCon+CloudNativeCon EU: https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/ - 2025-04-02: OpenStack 2025.1 ("Epoxy") Release - 2025-04-03: OpenInfra LIVE - Epoxy release [3] - 2025-04-08: OpenInfra Board Meeting: https://board.openinfra.org/ - 2025-04-07 - 2025-04-12: OpenInfra PTG Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.1 Release tags: https://review.opendev.org/c/openstack/releases/+/945949 [3] Introducing OpenStack 2025.1 Epoxy on OpenInfra Live: https://openinfra.org/live/ [4] OpenInfra Flamingo vPTG: https://ptg.opendev.org/ptg.html [5] "migrate to noble" community goal: https://governance.openstack.org/tc/goals/completed/2025.1/migrate-ci-jobs-… [6] TC Meeting IRC Log 2025-03-25: https://meetings.opendev.org/meetings/tc/2025/tc.2025-03-25-18.00.log.html [7] TC Meeting Agenda 2025-04-01: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0