Hello Zorillas and interested stackers,
Last week's PTG had plenty of topics and good takeaways.
In case you would like to watch any of the discussions, please take a look at the videos in the OpenStack Manila Youtube channel [0].
The PTG etherpad has all of the notes we took [9]. Here is a summary of the discussions grouped by each topic:
Retrospective
==========
Highlights
-------------
Mid cycle alongside feature proposal freeze provided a good opportunity for us to have collaborative review sessions and move faster on reviews.
Two bugsquashes had a good impact on the bug backlog and the bug trend was more positive on this cycle, despite the numbers growing due to low-hanging-fruits we started reporting.
Internships with City University of Seattle, Valencia College and North Dakota State University - they are definitely helping with progress on manila-ui and OpenAPI. We will continue the effort.
We would like to speed up reviews and improve our metrics [1] on how long changes are open before being merged. Review dashboards can help and we can work with our reviewers to have a more disciplined approach on reviews.
Broken third party CI systems currently mean that we have little testing. We need to rely on the authors' or their peers to test and ensure that a feature is working. We will look into documenting CI setup procedures and gather thoughts from maintainers.
New API Features should be tested as early as possible to ensure it won't break any workflows. Our contributor documentation will be updated with extra guidelines.
AIs:
(carloss) Encourage Bug Czar candidates and bring this up more often in the manila weekly meetings
(carloss) Encourage spec authors to schedule a meeting to discuss the spec to speed up the review process.
(carloss) include iCal with event announcements (bugsquash / mid cycle)
(gouthamr) Creating a review dashboard
(carloss) Record "expert seminars" on FAQs: it would be great to have some videos documenting how-tos in OpenStack and help people to unblock themselves when they are hitting common openstack-developer issues: https://etherpad.opendev.org/p/manila-howcasts
(carloss) communicate a deadline for the manila CLI -> OSC documentation changes. The work with our interns should go until FPF. It needs to be done before the client release, when we are planning to drop the manilaclient support. ashrodri offered help to get it completed after we come to the FPF deadline.
(carloss) We should update these docs and mention that first party driver implementations should be done for features and be more strict about the testing requirements.
All things CephFS [2]
================
Deprecation of standalone NFS-Ganesha
-------------------------------------------------------
We added a warning in Dalmatian, deferred plans to deprecate based on community feedback. Our plan is to remove it in the 2026.1 release. There is a suggested update procedure, please reach out in case there are questions.
AI: (carloss) send a reminder email in this cycle to incentivize people to move to clustered NFS
Supporting NFSv3 for Windows workloads
--------------------------------------------------------
manila-tempest-plugin now supports multiple NFS protocol versions in one of the scenario tests. As soon as we get the build, we will update the CephFS NFS job to run tests for NFSv3 as well.
Testing and stabilization
--------------------------------
Bumped Ceph version in the CI jobs to Reef in Antelope, Bobcat, Caracal, Dalmatian. We are starting to test with Ceph Squid; we intend to test with Squid on "master" and "stable/2025.1" (epoxy) branches.
A couple of Ceph and NFS-Ganesha issues are impacting us at the moment [4] [5] [6] and we managed to find the workaround for some.
We needed to stop testing with ingress daemon at the moment and we will get back to testing as soon as the fix is out.
Manage unmanage of shares and snapshots
-----------------------------------------------------------
Feature is merged and working and we are going to backfill tempest test patches
AI: (carloss) will propose a new job variant to allow testing this feature.
Plans for 2025.2 Flamingo
-----------------------------------
Investigate support for SMB/CIFS
Ceph-NFS QoS: we will follow the implementation of this feature in NFS Ganesha and start discussing and drafting the Manila implementation when the code is merged in Ganesha upstream.
Out of place restores and backup enhancements [7]
========================================
CERN is pursuing a backup backend with their C-Back tool. Currently Manila backups can be restored back to the same share; there are some problems with such approach when the source share backend is down and to prevent browse by restore behavior.
Zachary Goggins (za) proposed a specification, and plans to work on it during the Flamingo Cycle. The share backups feature also needs some enhancements like a get progress and get restore progress actions. Zach plans to make it part of the implementation.
We agreed that a backup resource should have a new "state" attribute, instead of only relying on the status in order to have well defined backup states.
AI: (za) update the out of place restore spec.
Tech debt
=======
Container driver failures
--------------------------------
The container driver tempest tests are perma-failing right now. We seem to have a problem with RBAC and pre-provisioned tempest credentials.
AIs:
(carloss) Report a tempest bug to track the issues;
(gouthamr) will propose a change to switch back to using dynamic credentials in our testing.
DockerHub rate limits
-----------------------------
We are only building an image in manila-image-elements. It's more pulls
than pushes. Pushes happen very rarely. The kolla team has moved away
from DockerHub as well.
Zach offered help in case we need another approach for registry. CERN has its own tool.
AI: we will look into moving to
quay.io
"manila" CLI removal
----------------------------
We added the deprecation warning 6 releases ago and we should proceed with the removal. We will need an additional push to update all of our documentation examples and move to keystoneauth.
We need more functional test coverage and we should have a hackathon just as we did some years ago.
AI: carloss will schedule a hackathon for enabling more tests and send the removal email to openstack-discuss. We are targeting the removal to 2025.2 Flamingo.
CI and testing
------------------
ZFSOnLinux job left on jammy: We created a bug for it and we can use it for tracking.
IPv6 testing: The BGP software we were using (quagga) is now deprecated and everything was migrated to FRR. We will need help to fix it as unfortunately, things didn't have a 1x1 translation between the libraries. If someone has experience on this, it would be nice to collaborate to get this fixed.
API
----
We are going to stop testing the v1 API and stop deploying it on DevStack test jobs. We'll update the install guide as well that we've stopped supporting it. It was deprecated in 2015 ("Liberty" release). That's a good code cleanup opportunity.
V2 is an extension of v1 with microversions.
If we stop supporting it, who is affected? Mostly people that have automations using it.
What's the impact on manila-tempest-plugin? We have v1 and v2 tests. We have a lot of coverage for v2. If you don't have the v1 API in the cloud, the tests refuse to run. We will need to fix it.
AIs:
Work on the removal patches during the 2025.2 Flamingo release;
(carloss) will send an announcement email to the ML, including operators tag.
Manila UI
-------------
We have been making progress in the Manila UI feature gap. Currently working on manage/unmanage share servers, manage share with dhss=true, filtering user messages on date, updating quotas table.
Share limits view broke some time ago, code lives in horizon.
We hit some issues using horizon's tox "runserver" environment, apparently more people ran into the same issue. We will talk to other impacted parties and check how to overcome this issue.
AI: (carloss) will reach out to the horizon team and ask how we can re-introduce Manila limits to the overview tab.
Enable share encryption at-rest (back-end) with secret refs stored on Barbican/Castellan. [8]
=====================================================================
We merged a specification some time ago with an implementation architecture. That spec contemplated both Share encryption and Share server encryption.
NetApp is now planning to work only on share server encryption. Encryption can be disabled per share, but shares exported via a share server cannot have a separate encryption key on ONTAP.
We reached an agreement that when a new share creation is triggered, if there isn't a share server matching the provided key, a new share server will need to be spawned. We also agreed that we should allow using names for the secret reference for better user experience.
2025.2 Flamingo is the target release.
AIs: (kpdev/Sai) The spec will be updated and only the DHSS=True scenario will be documented; The manila team will review the spec as soon as it is proposed
Replication Improvements
====================
Back when we implemented replication, we didn't account for specific configurations that the storage backends can have, for example whether the backend could support zero RPO technologies or not.
Zero RPO is is an important feature that allows data to be written simultaneously between the share and its replicas.
We agreed that the way we should send the information to the backend is through a backend specific share type extra spec. Administrators will be able to define it in the share type and the backend will pick it up.
Operator concerns / questions
=======================
Where to put parameters that change behaviour only of one protocol (NFS in this case)? We agreed that we should have write once type of metadata and not allow the metadata to be updated afterwards. A configuration option can be introduced for this where the operator can determine what metadata will not be updated.
AI: carthaca will propose a lite-spec for this
Lustre FS Support for HPC Use Cases in OpenStack
Is there any possibility for OpenStack to officially integrate or support parallel file systems like Lustre, either through Manila or other components? We've heard in the past as a request from the scientific-sig group. Building a driver should be straightforward and it does not necessarily need to be in-tree, and it would be easier to maintain. This is a very good use case. This discussion will continue with the scientific-sig group.
Replica / Snapshot Retention / Expiration Policy
While replicas in Manila are designed to be continuously in sync with the active share, certain use cases — such as disaster recovery (DR) replicas or manually created replicas that are no longer needed — could benefit from lifecycle management.
Replicas are continuously synced with the source share, so if they're "unused", they're there for some reason is the assumption. We had a spec a while ago about automating snapshots (creation and deletion) on schedule.
It would be preferable that an external automation tool is used to achieve such behavior. Maybe openstack/mistral can be a good approach (Support for manila snapshots already exists on Mistral)
Affinity/Anti-affinity spec updates
=========================
This feature allows users to create share groups with affinity policies, which determine the affinity relationship between shares within the group. There was an open question about strategies of locking. We came to an agreement that we can use tooz, database or oslo.
AI: (chuanm) will update the spec.
Force deleting subnets
=================
This is a feature that follows the ability to add multiple subnets to a share server. We should also be able to remove them. This spec is under review.
We agreed that we should also implement the "check" mechanism before deleting the subnet.
AIs: (sylvanld) will update the spec
Eventlet removal
=============
Need to remove wsgi uses, use oslo service's new Threading based backend instead for the ProcessLauncher, periodic tasks. Neutron is doing some work around periodic tasks and we can benefit from their learning.
AI: Work on this in Flamingo, aiming for completion in 2026.1 cycle.
Manila/Nova Cross-project session: VirtioFS
=================================
VirtioFS implementation is now complete and we are looking at the next steps. We currently don't have CI testing the feature and the Manila team is planning to work on it during the 2025.2 Flamingo release.
The nova team intends to drive the remaining SDK and OSC patches to completion during the 2025.2 Flamingo release.
We also discussed some possible enhancements: mem_fs support, online attach and detach and live migration. These will take some time and the Nova team will work on such features gradually.
AIs: (carloss) will share the test scenarios with the Nova team and ask for reviews and the Manila team will work on the implementation of the tests. (rribaud) will work on the remaining SDK patch and work on mem_fd support.
[1]
https://openstack.biterg.io/app/dashboards#/view/Gerrit-Backlog?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-3M,to:now))&_a=(description:'Gerrit%20Backlog%20panel%20by%20Bitergia.',filters:!(('$state':(store:appState),meta:(alias:'Changesets%20Only',disabled:!f,index:gerrit,key:type,negate:!f,params:(query:changeset),type:phrase),query:(match:(type:(query:changeset,type:phrase)))),('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:gerrit,key:author_bot,negate:!t,params:(query:!t),type:phrase),query:(match:(author_bot:(query:!t,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:gerrit,key:project,negate:!f,params:(query:manila),type:phrase),query:(match_phrase:(project:manila)))),fullScreenMode:!f,options:(darkTheme:!f,useMargins:!t),query:(language:lucene,query:(query_string:(analyze_wildcard:!t,default_field:'*',query:'*',time_zone:Europe%2FMadrid))),timeRestore:!f,title:'Gerrit%20Backlog',viewMode:view)
[2]
https://etherpad.opendev.org/p/flamingo-ptg-manila-cephfs[3]
https://bugs.launchpad.net/manila/+bug/2049538[4]
https://github.com/nfs-ganesha/nfs-ganesha/issues/1227[5]
https://tracker.ceph.com/issues/69214 [6]
https://tracker.ceph.com/issues/67323[7]
https://review.opendev.org/c/openstack/manila-specs/+/942694[8]
https://etherpad.opendev.org/p/share-encryption-with-barbican-secret-ref
Thank you everyone that participated on the PTG!
Best regards,
carloss