March 2025 - openstack-discuss - lists.openstack.org

[manila]NFS-Ganesha exports
by Mikhail Okhrimenko 05 Apr '25

05 Apr '25

Hello community, Trying to setup the Manila version of Caracal with NFS-Ganesha backend. Ganesha version 4.0 is deployed using ceph orch in a ceph cluster version 17. Backend configuration in manila.conf [ganesha_nfs] driver_handles_share_servers = False share_backend_name = ganesha_nfs Share_driver = manila.share.drivers.cephfs.driver.CephFSDriver cephfs_protocol_helper_type = NFS cephfs_conf_path = /etc/ceph/ceph.conf cephfs_auth_id = manila cephfs_cluster_name = ceph cephfs_filesystem_name = {{ cephfs_filesystem_name }} cephfs_nfs_cluster_id = {{ cephfs_nfs_cluster_id }} I see the subvolume being created in ceph ceph fs subvolume ls <volume_name> but no export ceph nfs export ls <cluster_id> what am I doing wrong? what am I missing? in the manila-share logs I see a request to create a subvolume, but I don't see a request to create an export, for some reason manila doesn't even try to create an export

3 3

[all][dev][ops][tc] Bridging the gap between community and contributing orgs
by Jeremy Stanley 02 Apr '25

02 Apr '25

The tl;dr on this is that I'm opening a broad community discussion on how we can, collectively through improved communication, better bring together our established contributors with casual and prospective contributors who struggle to find successful patterns of contribution, so that we might all support each other in order to benefit OpenStack as a whole. Read on for details... OpenStack's TC has requested that foundation staff, when in conversation with representatives of member organizations, encourage participation in the project and collect feedback on any related challenges those organizations encounter in their attempts to do so. In order to better understand the feedback and brainstorm achievable recommendations for the broader community[*], Community Managers on the foundation staff started to work with a small focus group of established contributors with a solid understanding of what successful patterns of contribution look like. It also came to light that established contributors experience many of the same sorts of challenges, even more so if they don't have the luxury of focusing full-time upstream, and so success often comes down to knowing how to effectively navigate those challenges. The community's goal with this exercise is to improve everyone's experience, and your help is needed to do that! The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. Ideas so far to address these gaps include: * Review bandwidth - Ensure contributors and companies understand the importance and benefits of code review - Incentivize and credit meaningful "+1" reviews more - Understand that the volume of changes and limited reviewer bandwidth often means proposals aren't reviewed quickly (or at all), and their proponents may need some additional tenacity and engagement with the team to get their work noticed - Utilize "review dashboards" as a way to find stale changes which have fallen through the cracks * Review strategy and etiquette - Be clear about the meaning of votes on changes, especially negative ones, and set explicit timelines for things like "procedural -2" or WIP blocks when their approval needs to be delayed - Focus on comments and requests that affect whether or not the change can be merged after the fixes are submitted, rather than requesting trivial adjustments to an unsuitable change which has more fundamental problems - Reciprocate with reviews of changes from new reviewers you see leaving insubstantial reviews, demonstrating to them what deeper and more meaningful reviewing looks like * Mentoring newcomers - Established contributors with sufficient bandwidth can help mentoring newcomers, potential core reviewers and new leaders - Mentoring can happen as part of internship programs as well as by helping newcomers determined to become active contributors; both ways should be embraced and utilized What challenges are you facing? And, how would you improve them? Two weeks from now, we'll have a forum session at the OpenInfra Summit Asia in Suwon[**] where our community can refine these possible approaches and integrate others, as well as bring attention to them and promote their application. I've also proposed a similar forum session for OpenInfra Days North America in Indianapolis a month after that, where we can hopefully continue this conversation. The goal for this multi-stage effort is to improve the contribution experience for established participants, casual contributors and newcomers alike. While it's easy to jump to blaming in these situations, it will be far more productive if we can focus on the challenges and experiences that we would each like to remove or improve for ourselves and others in our community. As we find ways together to improve our efficiency, we can reduce the load on all contributors and lower the barriers for people to join and participate. [*] https://etherpad.opendev.org/p/r.2205024e55689bccb82c20c960853cb5 [**] https://2024.openinfraasia.org/a/schedule#view=calendar&title=Bridging%20th… -- Jeremy Stanley

2 7

[octavia][nova][neutron] failed to failover loadbalancer
by Winicius Allan 02 Apr '25

02 Apr '25

Hello stackers! release: zed deploy-tool: kolla-ansible After an outage, all the load balancers from my cluster came to provisioning status ERROR because the o-hm0 interface was unavailable on controller nodes. I've created them again and tried a failover on load balancers. The octavia-worker logs show that failover was completed successfully, but the provisioning status remains with ERROR. Looking into nova-compute logs, I see that an exception was raised os_vif.exception.NetworkInterfaceNotFound: Network interface qvob8ac0f5f-46 not found The instance id for that interface matches the new amphora id. I'll attach the nova-compute logs here[1]. In neutron-server logs, there is no ERROR or WARNING that I can found, the only suspicious entry is Port b8ac0f5f-4613-4b95-9690-7cf59f739fd0 cannot update to ACTIVE because it is not bound. _port_provisioned /var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/plugin.py:360 Can anyone give me a light? [1] https://pastebin.com/hV3xgu0e

2 6

[nova] nova-manage fails when upgrading to Antelope
by Roby G 01 Apr '25

01 Apr '25

Hello everyone, I´m trying to upgrade to Antelope and have a problem when executing the db migrations. `nova-manage db sync` fails when upgrading from Zed to Antelope (same for Yoga --> Antelope) - Nova: 27.4.0 - Database: PostgreSQL 16 - Operating System: Ubuntu 22.04 - OpenStack Zed --> Antelope Full Log. /var/log/nova-manage.log: ``` 2025-03-04 11:42:14.963 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Context impl PostgresqlImpl. 2025-03-04 11:42:14.964 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Will assume transactional DDL. 2025-03-04 11:42:14.967 2519974 INFO alembic.runtime.migration [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] Running upgrade ccb0fa1a2252 -> 960aac0e09ea, de-duplicate_indexes_in_instances__console_auth_tokens 2025-03-04 11:42:14.970 2519974 WARNING oslo_db.sqlalchemy.exc_filters [None req-0e1f6874-3bac-49db-b525-94aebf9e0b51 - - - - - -] DBAPIError exception wrapped.: psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid constraint consoles_instance_uuid_fkey on table consoles depends on index uuid constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid constraint migrations_instance_uuid_fkey on table migrations depends on index uuid constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters self.dialect.do_execute( 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters psycopg2.errors.DependentObjectsStillExist: cannot drop index uuid because other objects depend on it 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters DETAIL: constraint block_device_mapping_instance_uuid_fkey on table block_device_mapping depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint consoles_instance_uuid_fkey on table consoles depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint fixed_ips_instance_uuid_fkey on table fixed_ips depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_actions_instance_uuid_fkey on table instance_actions depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_extra_instance_uuid_fkey on table instance_extra depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_faults_instance_uuid_fkey on table instance_faults depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_info_caches_instance_uuid_fkey on table instance_info_caches depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_metadata_instance_uuid_fkey on table instance_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint instance_system_metadata_instance_uuid_fkey on table instance_system_metadata depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint migrations_instance_uuid_fkey on table migrations depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint security_group_instance_association_instance_uuid_fkey on table security_group_instance_association depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters constraint virtual_interfaces_instance_uuid_fkey on table virtual_interfaces depends on index uuid 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters HINT: Use DROP ... CASCADE to drop the dependent objects too. 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters 2025-03-04 11:42:14.970 2519974 ERROR oslo_db.sqlalchemy.exc_filters ``` Any suggestions? Thanks!

3 3

[Octavia]
by Vincent Godin 01 Apr '25

01 Apr '25

Hello Octavia Team, We have an Openstack infrastructure with 3 Nova AZ and 3 Cinder AZ. Furthermore, Octavia will use cinder for Amphora disk. Is there a way to create an Amphora so that the Compute AZ match the Cinder AZ ? As the storage is AZ specific, it's not possible to launch an Amphora on a Compute AZ with a storage on an alternate AZ. As a client, there is no need to fix where the Amphora will be launched, we just want to be sure that if the Amphora is planned to be launched on a specific AZ, its storage will be selected on the corresponding AZ too ! Vincent

3 2

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R+0)
by Goutham Pacha Ravi 01 Apr '25

01 Apr '25

Hello Stackers, The release team will be tagging the GA release for all cycle-with-rc projects for 2025.1 ("Epoxy") on 2025-04-02 [1][2]. Remarkably, this will be the 31st release of OpenStack and one of two that will mark 15 years of the community! There will be an OpenInfra Live event on 2025-04-03, where a few project maintainers will reflect on how the release cycle went, what it will bring to users, and project priorities for the 2025.2 ("Flamingo") release cycle [3]. Alongside project teams, the OpenStack Technical Committee has been preparing for the upcoming virtual Project Teams Gathering. We're still gathering topics, so please add yours if you're interested in seeding a relevant discussion. You can find all the discussion etherpads linked on the PTG Website [4]. In the past week, we completed the community testing goal to upgrade CI integration testing ("migrate-to-noble") [5]. Kudos to our goal champion, Ghanshyam Mann (gmann), and to everyone who helped with the effort! === Weekly Meeting === The last weekly meeting of the OpenStack TC was held over IRC in OFTC's #openstack-tc channel on 2025-03-25 [6]. We added a discussion topic on operator engagement to the os-operators' PTG slot. We caught up on CI issues that needed attention from several quarters, including a kernel regression that prevented release-critical patches from merging in Neutron and Kolla and warranted a workaround. A breaking change in Setuptools 78 impacted many projects on Zuul; however, OpenStack was minimally affected before the issue was resolved in Setuptools. A broken CentOS 9 mirror briefly caused problems due to an inconsistent state from an upstream source. It was brought to the TC's attention that the Skyline project commits pre-built static files (e.g., JavaScript bundles generated using Yarn) directly into its repository. This manual intervention creates a potential security risk similar to the XZ backdoor attack since there is no verifiable way to check whether these builds contain malicious code. The TC brainstormed that it would be better to offload the build process to Zuul, preventing manual intervention, and to ensure the yarn.lock file is always included in releases for better traceability. Operators and project contributors must be aware of this issue and the fact that builds are currently not reproducible, complicating packaging. There was a suggestion that OpenStack governance should enforce standardized project security practices. Since we haven't identified concrete steps to resolve the current issue, we'll revisit this topic in a future discussion. The next meeting of the OpenStack TC is today, 2025-04-01, at 1700 UTC. Please note the change in timing—it is an hour earlier than our past meetings. This meeting will be held simultaneously on Video (Zoom) and IRC (OFTC's #openstack-tc channel). Please find the conference details and agenda on the meeting's wiki page [7]. I hope you'll be able to join us there. === Governance Proposals === ==== Merged ==== - Mark CI migration to Ubuntu Noble goal complete | https://review.opendev.org/c/openstack/governance/+/945641 ==== Open for Review ==== - [resolution] Extend scope of VMT to cover all project teams | https://review.opendev.org/c/openstack/governance/+/944817 === Upcoming Events === - 2025-04-01 - 2025-04-04: KubeCon+CloudNativeCon EU: https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/ - 2025-04-02: OpenStack 2025.1 ("Epoxy") Release - 2025-04-03: OpenInfra LIVE - Epoxy release [3] - 2025-04-08: OpenInfra Board Meeting: https://board.openinfra.org/ - 2025-04-07 - 2025-04-12: OpenInfra PTG Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.1 Release tags: https://review.opendev.org/c/openstack/releases/+/945949 [3] Introducing OpenStack 2025.1 Epoxy on OpenInfra Live: https://openinfra.org/live/ [4] OpenInfra Flamingo vPTG: https://ptg.opendev.org/ptg.html [5] "migrate to noble" community goal: https://governance.openstack.org/tc/goals/completed/2025.1/migrate-ci-jobs-… [6] TC Meeting IRC Log 2025-03-25: https://meetings.opendev.org/meetings/tc/2025/tc.2025-03-25-18.00.log.html [7] TC Meeting Agenda 2025-04-01: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[scientific-sig] Wiki update
by Jimmy McArthur 01 Apr '25

01 Apr '25

Hi Scientific Folks! I was trying to direct someone to your meetings, but noticed the wiki page is quite out of date: https://wiki.openstack.org/wiki/Scientific_SIG I know you all are meeting at the PTG, is this something that could be added to the agenda? I’ve got a few orgs that I’d like to send your way. Cheers, Jimmy

1 0

[openstack-helm] Merge openstack-helm and openstack-helm-infra git repos
by Vladimir Kozhukalov 01 Apr '25

01 Apr '25

Hi dear Openstack-Helmers, I would like to suggest merging the two project git repos: https://opendev.org/openstack/openstack-helm-infra.git https://opendev.org/openstack/openstack-helm.git At some point it seemed reasonable to have separate git repos for openstack and infrastructure charts. But now it looks like we are experiencing more cons than pros. When working on some features we are forced to prepare sets of interdependent patches. Some code can be deduplicated. Also when we publish charts we append two commit hashes as package metadata. Also we can reduce the number of test jobs if we have all chats in one repo. Other people also proposed this earlier. So if there are no objections let's make it done. I understand this is going to affect many of our users. Please express your opinions so we don't break any downstream. If we all agree about this, then I would say we can do this in the mid of March/2025 before 2025.2 PTG. The procedure is going to be smth. like 1. Freeze the infra repo and prepare retirement PSs for it 2. Merge openstack-helm-infra repo into openstack-helm ``` git clone https://opendev.org/openstack/openstack-helm.git cd openstack-helm git remote add infra https://opendev.org/openstack/openstack-helm-infra.git git fetch infra git checkout -b infra infra/master git checkout master git merge --allow-unrelated-histories infra # resolve not so many conflicts git review ``` 3. Accept merger PS 4. Update documentation 5. Accept infra retirement PSs (openstack-helm-infra and project-config) -- Best regards, Kozhukalov Vladimir

4 10

[neutron] Bug deputy report March 24 - March 30
by Oleg Bondarev 31 Mar '25

31 Mar '25

Hello Neutron Team, Bug report for the week of March 24 is below: *High:* - https://bugs.launchpad.net/neutron/+bug/2104019 - random failures in L3 HA fullstack failover tests - Confirmed - assigned to Miguel Lavalle - https://bugs.launchpad.net/neutron/+bug/2104134 - ovs/linuxbridge jobs running on ubuntu noble broken with latest kernel 6.8.0-56.58 - Confirmed - unassigned (looks like yatin is tracking it) - https://bugs.launchpad.net/neutron/+bug/2105383 - [OVN] The filter to add the MTU to a LRP changes the options dictionary population - In progress - patch from Rodolfo Alonso: https://review.opendev.org/c/openstack/neutron/+/945905 - https://bugs.launchpad.net/neutron/+bug/2104160 - [DB] Stop expand migrations with field drop exceptions - Confirmed - unassigned ---------- *Low:* - https://bugs.launchpad.net/neutron/+bug/2104927 - [OVN] HA_Chassis_Group creation returns an exception in the logs Edit - In progress - patch from Rodolfo Alonso: https://review.opendev.org/c/openstack/neutron/+/945837 - https://bugs.launchpad.net/neutron/+bug/2104979 - Possible deadlock on closing ip_monitor thread - Triaged - unassigned ---------- *Undecided:* - https://bugs.launchpad.net/neutron/+bug/2104037 - Unable to open stream to ssl::6641 to retrieve schema: Exec format error - Incomplete Thanks, Oleg

1 0

PCI passthrough - flavor not found
by Francesco Di Nucci 31 Mar '25

31 Mar '25

Hi all, I am trying to setup PCI passthrough with Nova, following guides such as this <https://superuser.openinfra.org/articles/a-comprehensive-guide-to-configuri…> and this <https://medium.com/@thomasal14/gpu-passthrough-in-openstack-da2a98a16f7b>: * installed a GPU on a compute node, configured kernel etc and now it is using the vfio-pci driver * set [pci]/device_spec = { "vendor_id": "10de", "product_id": "1ff2" } on compute node (formerly [pci]/passthrough_whitelist) and restarted openstack-nova-* * On the controller node set [pci]/alias: { "vendor_id":"10de", "product_id":"1ff2", "device_type":"type-PCI", "name":"nvidia-t400" }, [filter_scheduler]/enabled_filters = PciPassthroughFilter, [filter_scheduler]/available_filters = nova.scheduler.filters.all_filters and restarted openstack-nova-* * Created a flavor with openstack flavor create --vcpus 4 --ram 8192 --disk 40 --property "pci_passthrough:alias"="nvidia-t400:1" gpu_flavor, it is shown with openstack flavor list * Created an image with --property img_hide_hypervisor_id=true * Tried to create an instance with openstack server create --flavor gpu_flavor --image Almalinux_GPU --key-name "My Key" --network my_network test-gpu (also with --availability-zone nova:my-gpu-node.example.com) * VM creation fails with this errors on the controller node /nova-api.log/ HTTP exception thrown: Flavor gpu_flavor could not be found. /nova-scheduler.log/ Filter PciPassthroughFilter returned 0 hosts Filtering removed all hosts for the request with instance ID '131d5c1c-1927-42c0-bb48-618c05d31c2a'. Filter results: ['PciPassthroughFilter: (start: 19, end: 0)'] /nova-conductor.log/ Failed to schedule instances: nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. If I understand correctly it's a cascading error generating from the flavor not found, even though it exists... Anyone has encountered something like this/has suggestions? Am I missing some config? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email:francesco.dinucci@na.infn.it

6 10