Hello Zorillas, and other interested Stackers! Sorry this is getting to you later than usual :) We concluded a productive Project Team Gathering on 23rd Apr '21. I'd like to thank everyone that participated as well as the friendly folks at the OpenInfra Foundation who worked hard to organize this event. The following is my summary of the proceedings. I've linked associated etherpads and resources to dig in further. Please feel free to follow up here or on freenode's #openstack-manila if you have any questions. == Day 1 - Apr 19, 2021, Mon == === Manila Retrospective === Topic Etherpad: https://etherpad.opendev.org/p/manila-wallaby-retrospective - interns and their mentors did a great job through Wallaby! Many accomplishments, thanks to the sponsoring organizations for their investment - the two Bugsquash events at M1 and M3 were very effective. - we added new core members (python-manilaclient, manila-tempest-plugin) - sub-teams (manila-ui, manilaclient and manila-tempest-plugin) increased focus on the individual projects and sped up reviews - good cross project collaboration with OpenStack QA, Horizon, NFS-Ganesha and Ceph communities - we discussed strategies to avoid reviewer burnout and performed health checks on the team growth initiatives we took up in the Wallaby cycle Actions: - need to get proactive about reviews so we can avoid burn-out at the end of the cycle - continue bug squash days in Xena - look for interested contributors to join the core maintainer team - bring up reviews in weekly meetings and assign review owners earlier than M-2 - contributors need to be reminded to help with reviews == Day 3 - Apr 21, 2021, Wed == === manila-tempest-plugin plans and update === - lkuchlan and vhari highlighted difficulties around "capabilities" testing that was exposed due to feature changes in the CephFS driver in the Wallaby cycle - optional feature testing requires rework, the configuration options and test strategy have gotten confusing over time - discussion continued on dropping testing for older API microversions where some optional features (snapshots, snapshot cloning) weren't really optional. It was agreed that this was overkill. The problem is specifically in share type extra-specs. If these are setup with API version >= 2.24, we could circumvent this: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/785307. The only issue is that tests could theoretically be run against a cloud that doesn't support API version 2.24 (has to be Newton release or older). The team agreed that we could document this deficiency in the configuration file since Newton release has been EOL for a while. Actions: - review and merge the fix to share type extra-specs - rework existing "run_XYZZY_tests" and "capability_XYZZY_support" flags to "share-feature-enabled" section where these capabilities are always set when a feature is enabled - this is in-line with the way optional feature testing for other OpenStack services is done === Service Recovery === Topic Etherpad: https://etherpad.opendev.org/p/xena-ptg-manila-share-recovery-polling - gouthamr discussed the use cases and architectures needing to run manila-share in active-active HA - service recovery after failure needs to be coordinated to prevent metadata corruption - clustering services will provide coordination for cleanup - vendor driver authors will need to audit critical sections and replace oslo.concurrency locks with distributed locks. Actions: - there will be a specification submitted for review === Polling operations in manila-share === Topic Etherpad: https://etherpad.opendev.org/p/xena-ptg-manila-share-recovery-polling - haixin and gouthamr summarized the periodic tasks conducted by the share manager service - to update share service capabilities and real time capacity information, - to monitor the health of share replicas, - to "watch" the progress of long running clone, snapshot and migration operations. - these periodic tasks will be inefficiently duplicated if running across multiple nodes associated with the same backend storage system/driver - many of these periodic tasks are meant to be read-only, however, the health-check operations modify database state, currently these assume a single writer, and unless staggered, updates may be duplicated as well - proposed mitigation was to allow clustered hosts to ignore polling operations if one is in progress elsewhere, and for the service instance executing the tasks to distribute the workload to other hosts via the message queue - another issue with periodic tasks was the lack of ability to stop an unnecessary task from running - we could take advantage of oslo's threadgroup that exposes a start/stop API Actions: - replace loopingcall periodic tasks with threadgroups to selectively run periodic tasks - propose a specification to coordinate and distribute polling workloads === Interop Working Group === - arkady_kanevsky presented the mission of the InteropWG, and laid out plans for the upcoming Interop guideline - manila tests were added to the guideline as part of the 2020.11 guideline and have been available for the community to provide results - there was an optional capability (shrinking shares) that was flagged, and the team discussed if it was to remain flagged, or be moved to an advisory state - including reduction of the cutoff score - the team wants to stabilize the tests exposed through the next guideline, and work on a way to remove the necessity of an admin user to bootstrap the manila's tempest tests where unnecessary Actions: - improve manila tempest plugin to not request admin credentials for the tests exposed, and work with a configured "default" share type === Share server migration improvements === - share server migration was added as an experimental feature in the Victoria release - carloss presented use cases for nondisruptive share server migration - nondisruptive migration would mean that network changes are unnecessary, so the share manager needn't seek out any new network ports or relinquish existing ones - the share server migration is experimental, however, we'd still make nondisruptive migration possible only from a new API microversion - manual triggering of the second phase is still necessary to allow control of the cutover/switchover. All resources will continue to be tagged busy until the migration has been completed Actions: - no specification is necessary because the changes to the API and core manila are expected to be minimal === OpenStackSDK and Manila === - ashrod98, NicoleBU and markypharaoh, seniors at Boston University worked through the past cycle to expose manila resources to the OpenStackSDK - an outreachy internship proposal has been submitted to continue their work to expose more API functionality via the SDK - we'd prioritize common elements - quotas, limits and availability zones so we can parallely work on the OpenStackClient and Horizon implementation for these - need review attention on the patches Actions: - some domain reviewers for manila patches to be added to the OpenStackSDK core team to help with reviews == Day 4 - Apr 22, 2021, Thu == === Support soft delete/recycle bin functionality === - haixin told us that inspur cloud has a "soft-delete" functionality that they would like to contribute to upstream manila - the functionality allows users to recover deleted shares within a configurable time zone via Manila API - the proposal is to only allow soft-deletion and recovery of shares, but in the future this can be extended to other resources such as snapshots, replicas, share groups - soft deletions have the same validations as deletions currently do - quota implications and alternatives (unmanage, an system admin API) were discussed Actions: - a specification will be proposed for this work === New Driver Requirements and CI knowledge sharing === - there was a informative presentation from the Cinder PTG [3] about using Software Factory to deploy a Third Party CI system with Gerrit, Nodepool, Zuulv3 and other components - some helpful but unmaintained test hook code exists in the repos to aid with CI systems using devstack-gate - since devstack-gate has been deprecated for a long time, third party CIs are strongly recommended to move away from it Actions: - complete removal of devstack-gate related manila-tempest-plugin installation from manila's devstack plugin - create a Third Party CI and help wiki/doc that vendor driver maintainers can curate and contribute to === New Driver for Dell EMC PowerStore === - vb123 discussed Dell EMC powerstore storage, its capabilities and their work in cinder, and made a proposal for a driver in manila - this driver's being targeted for the Xena release: https://blueprints.launchpad.net/manila/+spec/powerstore-manila-driver - community shared updates and documentation from the Victoria and Wallaby cycles that may be helpful to the driver authors - driver submission deadlines have been published: https://releases.openstack.org/xena/schedule.html === Enabling access allow/deny for container driver with LDAP === - esantos and carloss enabled the container driver in the Wallaby release to support addition of and updates to LDAP security services - however, this work was just intended to expose a reference implementation to security services - the container driver does not validate access rules applied with the LDAP server today - the proposal is to enable validation in the driver when a security service is configured Actions: - publish container driver helper image to the manila-image-elements repository and explore container registries that community can use for the long term for CI and dev/test - file a bug and fix the missing LDAP validation in the container share driver === Keystone based user/cephx authentication === - "cephx" (CEPHFS) access types are not validated beyond their structure and syntax - with cephx access, an access key for the ceph client user can be retrieved via the access-list API call - access keys are privileged information, while we've always had a stance that Native CephFS is only suitable in trusted tenant environments, there's a desire to hide access keys from unrelated users in the project - gouthamr proposed that we allow deployers to choose if Keystone user identity validation must be performed - when validation is enabled, manila ensures that users may only allow share access to other users in their project - users may only retrieve access keys belonging to themselves, and no other project users - an alternative would be to keystone validation would be to allow controlling the access_key via a separate RBAC policy - with the alternative, we wouldn't be able to validate if the access to user is in the same project as the user requesting the access Actions: - a specification will be proposed, however, the work's not expected to be prioritized for Xena === Addressing Technical Debt === - we discussed tech debt that was piling up from the last couple of cycles and assigned owners to drive them to completion in the Xena cycle Actions: - python-manilaclient still relies on keystoneclient to perform auth, there's ongoing work from vkmc+gouthamr to replace this with keystoneauth [4] - code in manila relies on the retrying python library that is no longer maintained; kiwi36 will propose replacing the usage with tenacity with tbarron's help [5] - tbarron will be resurrecting a patch that removes old deprecated configuration options [6] - carloss will be picking up rootwrap-to-privsep migration === Unified Limits === Topic Etherpad: https://etherpad.opendev.org/p/unified-limits-ptg - kiwi36, our Outreachy intern, proposed the design for using keystone's unified limits in manila with the help of the oslo.limit library - he walked through the prototype using resource quotas and highlighted the differences with the current quota system - we discussed why nested quotas were preferable to user quotas Actions: - a release independent specification will be proposed for the community to review and continue this work - explore how share type quotas will be handled === Secure RBAC Follow up === - vhari presented the RBAC changes that were made in the Wallaby release, including support for the system scope and reader role - a tempest test strategy was discussed, along with improvements made to tempest itself to make test credential setup with the new default roles - there's no plan to enforce the new defaults in Xena, we'll take the release to stabilize the feature and turn on deprecation warnings indicating intent to switch to the new defaults in the Y release Actions: - vhari and lkuchlan will start working on the tempest tests - wrap up known issues with the new defaults and backport fixes to the Wallaby release - manila's admin model and user personas will be documented == Day 5 - Apr 23, 2021, Fri == === VirtIOFS plans and update === Topic Etherpad: https://etherpad.opendev.org/p/nova-manila-virtio-fs-support-xptg-april-2021 - tbarron and lyarwood presented an update on the research that was done in the wallaby release wrt live attaching virtiofs volumes to running compute instances - qemu supports live attaches, the feature is coming to libvirt soon. live migration of instances with virtiofs volumes isn't supported yet - this is ongoing work in the qemu/kvm/libvirt communities - we discussed the connection initiation and information exchange between manila and nova, and what parts will be coordinated via the os-share library - there are as yet no anticipated changes to the manila API Actions: - continue working with the libvirt community to have the live-attach APIs added - a specification will be proposed to nova (and manila if changes are necessary to the manila API) === CephFS driver update === Topic Etherpad: https://etherpad.opendev.org/p/cephfs-driver-update-xena-ptg - vkmc presented the changes and new features in the cephfs driver in the wallaby cycle - the driver was overhauled to interact with ceph via the ceph-mgr daemon instead of the deprecated ceph_volume_client library - there's an upgrade impact going from victoria to wallaby. Deployers must consult the release notes [7] and the ceph driver documentation [8] prior to upgrading. Data path operations are unaffected during an upgrade: - ceph clusters must be running the latest version of the release they're on to allow the wallaby manila driver to communicate with the ceph cluster - the ceph user configured with the driver needs mgr "caps", and mds/osd privileges can be dropped/reduced [8] - we discussed dropping the use of dbus in the nfs-ganesha module, and using the "watch-url" approach that ganesha's ceph-fsal module provides - we also discussed ceph-adm and ceph-orch based deployment which replaces ceph-ansible and the impact that would have on ganesha configuration - ceph quincy will support active/active nfs-ganesha clusters (even with ceph-adm based deployment), we discussed manila side changes to take advantage of this feature === manilaclient and OpenStackClient plans and update === Topic Etherpad: https://etherpad.opendev.org/p/manila-osc-xena-ptg - maaritamm provided an update on our multi-cycle effort to gain parity between manila's shell client and the osc plugin - we made great progress in the wallaby cycle and covered all "user" related functionality that was requested - we discussed what's left, and sought community priority of the missing commands - we discussed deprecation of the manila shell client and only supporting the osc plugin - albeit addressing the "standalone" use case where "manila" can be used in place of "openstack share" if users desire - we could use help to add new commands, and to review code in the osc plugin Actions: - achieve complete feature parity between the two shell client implementations, deprecate the manila shell client === manila-ui plans and update === Topic Etherpad: https://etherpad.opendev.org/p/manila-ui-update-xena-ptg - disap, our Outreachy intern and vkmc, highlighted the changes made in the wallaby cycle - we lauded the cross-project collaboration that was renewed in this cycle to triage issues, and proactively work on incoming changes in horizon and elsewhere - manila-ui's microversion catchup is progressing slowly, but surely - we have another outreachy project proposal that overlaps with the xena cycle Actions: - continue to catch up to API feature functionality in the UI Thanks for staying with me so far! :) There were a number of items that we couldn't cover, that you can see on the PTG Planning etherpad [1], if you own those topics, please add them to the weekly IRC meeting agenda [2] and we can go over them. The meeting minutes etherpad continues to be available [9]. As usual, the whole PTG was recorded and posted on the OpenStack Manila Youtube channel [10] We had a great turn out and we heard from a diverse set of contributors, operators, interns and users from several affiliations and time zones. On behalf of the OpenStack Manila team, I deeply appreciate your time, and help in keeping the momentum on the project! Best, gouthamr [1] https://etherpad.opendev.org/p/xena-ptg-manila-planning [2] https://wiki.openstack.org/wiki/Manila/Meetings [3] https://www.youtube.com/watch?v=hVLpPBldn7g [4] https://review.opendev.org/647538 [5] https://review.opendev.org/380552 [6] https://review.opendev.org/745206/ [7] https://docs.openstack.org/releasenotes/manila/wallaby.html <https://docs.openstack.org/releasenotes/manila/wallaby.html> [8] https://docs.openstack.org/manila/wallaby/admin/cephfs_driver.html [9] https://etherpad.opendev.org/p/xena-ptg-manila [10] https://www.youtube.com/playlist?list=PLnpzT0InFrqDmsKKsF0MQKtv9fP4Oik17