October 2021 - openstack-discuss - lists.openstack.org

[manila][ptg] Yoga cycle PTG summary
by Goutham Pacha Ravi 28 Oct '21

28 Oct '21

Hello Zorillas and other awesome Stackers, Thank you for participating in the Yoga cycle Project Teams Gathering. A special thanks to the folks at the OpenInfra Foundation who provided the platform for this event! We brainstormed on numerous topics. You'll find the unabridged notes and links in the main etherpad [1]. Video recordings from the event are posted on our Youtube channel [2]. The following is my summary of the proceedings. Please feel free to follow up here, or on OFTC's #openstack-manila if you have any questions. == Xena cycle retrospective == Etherpad: https://etherpad.opendev.org/p/manila-xena-retrospective * We celebrated successful Outreachy internships (Kafilat Adeleke and Archana Kumari), completion of the senior design project of students from Boston University (Ashley Rodriguez, Nicole Chen and Mark Tony) and two Open Source Day mentoring events at the Grace Hopper Celebration. * We have long term contributors in kafilat and archanaserver, along the lines of ashrod98 (who is now a Red Hatter!), disap (at Microsoft!), kiwi_36 (continuing his unified limits work), maaritamm (at city network, is manila core!) and many others. Very proud of the effort the team puts in to attract and retain such talent in the community. * We also discussed the mentoring efforts coinciding with the Yoga release (Northeastern University Cloud Computing course project, Outreachy internships, others). * We anticipate more changes to the maintainers team in the Yoga cycle in our continued effort to build domain expertise around the team, and avoid reviewer burn out. * Thanks to the enthusiastic participation, and to a well groomed bug backlog (kudos, vhari) we had two successful bugsquash weeks and a hack-a-thon during the Xena cycle. We're pruning down the list of stale issues collaboratively. The team liked the frequency of these events. * We discussed the responses for the 2021 User Survey and key takeaways (need for active/active HA, unified limits, unified cli, virtiofs) * Action Items: ** Plan a hack-a-thon in Yoga (topics welcome) ** Update questions for the 2022 user survey ** Plan periodic collaborative code review events alongside the bug squash events to prevent languishing patches. == Share transfers == * haixin highlighted the need for APIs to safely transfer shares across project namespaces and discussed some caveats, and a rough workflow for the feature * To ensure secure multi-tenancy, many back end share drivers use the project_id metadata in their provisioning paths - so transferring resources must send appropriate notifications * For DHSS=True, transfering the share alone makes little sense without first transfering the share network onto which the share is exported - however, multiple shares can be associated with one share network, so this could need a multi-phase transfer needing an ability to rollback. * Action Items: * Propose a spec and discuss the multi-stage implementation of this feature == Project testing and Interoperability improvements == * vhari shared plans for the improvement of the interop guidelines associated with manila * We touched upon the work lkuchlan and vhari are doing to align optional feature testing with tempest's best practices * Third party CI maintainers were reminded that they must be running scenario tests * Action Items: * Team looks at the possible tests to add as advisory to the 2021.11 guideline: https://etherpad.opendev.org/p/refstack-test-analysis == CephFS driver and integration changes == Etherpad: https://etherpad.opendev.org/p/yoga-ptg-manila-cephfsnfs-cephadm * vkmc highlighted the changes intended in the Manila CephFS driver to adopt the ceph mgr nfs APIs instead of the DBUS interactions that the Ganesha interace is coded to perform. The ceph mgr NFS APIs would only work on cephadm-deployed ceph nfs (nfs-ganesha) clusters. * Ceph nfs can be deployed like any other ceph daemon in an active/active HA cluster. One concern was the backwards incompatible changes to default nfs-ganesha configuration - this is being worked out. * TripleO developers joined us to discuss deployment changes, and the upgrade concerns including the eventual change in the NFS VIP. * Action Items: * Share notes about the deployment and upgrade changes necessary to get manila's CephFS driver to work with an active/active NFS HA cluster for the benefit of operators as well as deployment tools like tripleo, kolla-ansible, juju charms, etc. == Manila deployment at the Edge == Etherpad: https://etherpad.opendev.org/p/yoga-ptg-manila-at-the-edge * I quizzed fultonj about the use cases and architecture of the Distributed Compute Node edge deployments that tripleo is capable of orchestrating. * DCN now supports persistent block storage, and enables setting up the service in a multi-backend configuration with local-to-compute backends associated with availability zones that represent the edge sites. * We captured work items to similarly support manila-share at edge sites in this architecture to enable RWX storage at edge sites, and brainstormed future possibilities around the feature (cross site share replication) * Action Items: - A spec to support manila-share in an active/active HA configuration - Testing template for A/A with vendor share drivers == Metadata APIs for all user facing resources == Etherpad: https://etherpad.opendev.org/p/yoga-manila-metadata * ashrod98 discussed the ongoing work to generalize user defined metadata to all end-user facing resources. Currently users can attach metadata to shares and access rules, and the service provides metadata to export locations. * There are changes to the initial design that were brainstormed wrt RBAC and service/operator metadata. * The new APIs are going to be supported in the openstackclient shell implementation * Action Items: * Reviewer attention needed for the changes to the metadata spec, and for new RBAC guarding operator metadata. == Manila UI updates == * vkmc, carloss and haixin presented the plan to update the UI's feature parity * We have two outreachy internship proposals around implementing new UX around share networks and for enhancing integration testing for manila-ui * Action items: * Outreachy contributor period is open, lookout for applicants seeking reviews * Identify trivial API features that can be supported with little effort for the Yoga cycle hack-a-thon == VirtIOFS == Etherpad: https://etherpad.opendev.org/p/nova-manila-virtio-fs-support-yptg-october-2… * tbarron presented updates regarding the ongoing prototyping for nova's support of virtiofs. * There's a demo of how the feature's expected to work: https://asciinema.org/a/IZ7UrhwspxBN63XsOl9JrTcUX?speed=1.5 * The nova discussion (https://etherpad.opendev.org/p/nova-yoga-ptg) included mechanics of supporting read-only attachments, os-share library, mount tags and use cases for baremetal nodes. The specification will be refined to answer some of these questions. * This continues to be a multi-release effort as changes in libvirt, qemu, kvm were needed for live virtiofs attachments. * Action Items: * Review/Refine nova spec: https://review.opendev.org/c/openstack/nova-specs/+/813180 == Multiple share network subnets in an AZ == Etherpad: https://etherpad.opendev.org/p/ptg-multiple-subnet * nahimsouza and felipe_rodrigues presented the use case for share networks to allow multiple subnets per availability zone. * This proposal would make dual stack ipv4/ipv6 networking possible for DHSS=True * Users would also have the ability to modify share networks that were in use - new network allocations will be possible for existing share servers, however, network allocations cannot be deleted. * Concerns were raised regarding the user experience and error signalling when port allocations cannot be made from all relevant subnets. * Action items: * Determine what needs to happen when a subnet has run out of allocations * Propose a specification discussing the API and driver interface impact to accommodate this change == FIPS compatibility and compliance == Etherpad: https://etherpad.opendev.org/p/yoga-manila-FIPS * carloss, ashrod98 and ade_lee spoke of the cross project effort to get FIPS compatibility and compliance testing * We discussed manila's direct/indirect use of cryptographic libraries and FIPS compliant alternatives (md5 in code, and paramiko's use of non-fips compliant digests); manila-tempest-plugin needs fixes along the same lines since it relies on paramiko. * Currently, the team's writing new CI jobs to test compatibility - these should merge by the end of the Yoga release cycle; the goal for compliance would be the Zorilla release (yes, i said it). * Action items: * Work on FIPS compatibility jobs in the Yoga release cycle == Secure RBAC changes in Yoga == Etherpad: https://etherpad.opendev.org/p/yoga-ptg-manila-srbac * vhari and I rounded up the work done so far to update the default RBAC policies across manila APIs and highlighted known issues, and pending work items * Cross service arbitrations on behalf of a user are ongoing discussions elsewhere - examples of these include manila's generic driver where the user's namespace and quota are consumed to create the nas server, backing volume and networking * We are continuing to implement tempest tests and anticipate a large portion to be completed during the Yoga cycle. * We don't plan to support cloud profiles with the manilaclient shell. Users are encouraged to switch to the openstackclient shell to use cloud profiles. * Action items: * Close on known issues in manila code * Complete protection and tempest test coverage == Paying down some technical debt in Yoga == * python-manilaclient still uses keystoneclient instead of keystoneauth1 - vkmc and gouthamr will work on this early in the Yoga release cycle * sqlalchemy2.0 changes - ack'ing SADeprecationWarnings and switching to using the new db engine facade exposed by oslo.db - felipe_rodrigues and gouthamr will work on this during the Yoga cycle * these are wishlist items that need volunteers to drive them to completion: * the generic driver doesn't yet support online extensions - * the generic driver can support more than 26 volumes per share server - this is a wishlist item that needs a volunteer to drive it * we need to publish the container driver image to a container registry from its development within the manila-image-elements project == Backing up shares == Etherpad: https://etherpad.opendev.org/p/ptg-backup-manila * We were joined by operators and storage experts from SAP, CSI Piemonte, NetApp, Red Hat and CERN to discuss current use cases and methods to backing up manila shares * Existing data protection and disaster recovery tools within manila (creating and mounting snapshots, cloning snapshots to new shares, reverting snapshots in-place, replicating shares and snapshots) were compared against the need for scheduling, efficient, incremental, durable/ex-situ/non-homogenous backup destinations, whole container vs selective file backup and recovery. * Available DIY external tooling (restic, borg, urbackup, other) and wrappers were discussed * Creating a backup solution based off-of the manila-data service was considered in the past, the spec was abandoned. It could be revived; however, we could benefit from presenting the need, a state of affairs as they are before exploring solutions. * Action Items: * We'll write up a doc on data protection for manila shares and invite operators to review them and propose future plans; anyone interested in collaborating in this space can get in touch with felipe_rodrigues == OpenStackCLI updates == Etherpad: https://etherpad.opendev.org/p/manila-osc-yoga * Thanks to maaritamm's awesome work, and the enthusiastic hack-a-thon participation from the community, we're very close to parity between OSC and the manilaclient shell. So, the team decided to stop adding new features in the manilaclient shell as a way to gradually wean users off of it. * When using the manilaclient shell from the Yoga release, users will be shown a deprecation warning suggesting an eventual removal (looking at the "A" release for this one, thoughts are welcome). * maaritamm proposed a hack-a-thon for completing the functional testing around manilaclient's osc plugin. * The osc plugin also needs the api microversion negotiation bits that the manilaclient shell currently has. * Action items: * We assigned owners to pending reviews, and missing osc commands * Deprecate the native manilaclient shell. == Manila CSI updates == Etherpad: https://etherpad.opendev.org/p/yoga-ptg-manila-csi * gman0 and tbarron presented the state of the kitchen for manila-csi since the Xena PTG and shared future plans * share backup is now a focus area for pvs serviced by manila-csi - the design involves being able to mount manila snapshots as read only PVs on backup pods and extracting relevant files. * we discussed e2e testing that's being added to the cloud-provider-openstack repository and the results of the perf/scale testing with manila-csi on openshift == OpenStackSDK updates == Etherpad: https://etherpad.opendev.org/p/yoga-manila-openstacksdk-update * we've had several interns working on exposing manila APIs within the openstacksdk since the wallaby release * kafilat has numerous open changes, wrapped up a successful outreachy internship recently * megharth, Jiabo, tutkuna and rishabhdutta from Northeastern University picked up the implementation of the remaining API resources for the Yoga cycle * this continues to be a multi-cycle effort and the plan is to eventually use manila APIs via the openstacksdk within manila-ui, ansible-collections-openstack and manila's OSC plugin. * Action Items: * team will review open changes against the openstacksdk repository pertaining to manila Thanks for reading thus far. It's possible a lot can get lost in translation :) As a reminder, do check out the complete recordings on our Youtube channel! [2] -- Goutham Pacha Ravi PTL, OpenStack Manila [1] https://etherpad.opendev.org/p/yoga-ptg-manila [2] https://www.youtube.com/watch?v=wVAClI82Ths&list=PLnpzT0InFrqDhp1A3lBi_guLA…

1 0

Router interfaces are down
by Jibsan Joel Rosa Toirac 28 Oct '21

28 Oct '21

Hello, I'm trying to route all the requests from a private vlan to internet. I have a private network and all the Virtual Machines inside the subnet config can do everything between they, but if I ping to Internet, it doesn't work. When I see the router_external it says all the interfaces are DOWN. I have search in everywhere but I can't find a solution for this. Thank you for your time

2 1

[all][tc] Technical Committee next weekly meeting on Oct 28th at 1500 UTC
by Ghanshyam Mann 28 Oct '21

28 Oct '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Oct 28th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Oct 27th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

SSSE3 CPU flag not available
by tarun singhal 28 Oct '21

28 Oct '21

Hello, I have an Openstack cluster which was created using devstack. And recently I was involved in project in which there is a dependency on SSSE3 cpu flag in an Openstack VM instance, to accomplish certain tasks. I searched many posts but couldn't find a definitive solution. The suggestion mentioned didn't worked for me either. I have added below in /etc/nova/nova.conf and restarted devstack(a)n-cpu.service cpu_mode = host-model cpu_model_extra_flags = ssse3 Please let me know a possible way to enable SSSE3 cpu flag in a vm instance. Thanks, Tarun

1 0

[tripleo] x509 certificate Issue in Overcloud Deployment
by Anirudh Gupta 27 Oct '21

27 Oct '21

Hi Team, I am trying to install the Tripleo setup on the Openstack Train. While deploying the overcloud, I am getting an x509 certificate error in downloading the images from quay.io This is because of a proxy certificate in the LAB. Before installing Undercloud, I manually placed the certificate in order to avoid this issue. How can I take care of placing such certificates in case of an overcloud. Regards Anirudh Gupta

2 1

Xena RDO Release Announcement
by Amy Marrich 27 Oct '21

27 Oct '21

If you're having trouble with the formatting, this release announcement is available online at https://blogs.rdoproject.org/2021/10/rdo-xena-released/ ---- *RDO Xena Released* The RDO community is pleased to announce the general availability of the RDO build for OpenStack Xena for RPM-based distributions, CentOS Stream and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds. Xena is the 24th release from the OpenStack project, which is the work of more than 1,000 contributors from around the world. The release is already available on the CentOS mirror network at http://mirror.centos.org/centos/8-stream/cloud/x86_64/openstack-xena/. The RDO community project curates, packages, builds, tests and maintains a complete OpenStack component set for RHEL and CentOS Stream and is a member of the CentOS Cloud Infrastructure SIG. The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS users looking to build and maintain their own on-premise, public or hybrid clouds. All work on RDO and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first. PLEASE NOTE: RDO Xena provides packages for CentOS Stream 8 only. Please use the Victoria release for CentOS Linux 8 which will reach End Of Life (EOL) on December 31st, 2021 (https://www.centos.org/centos-linux-eol/) *Interesting things in the Xena release include:* - The python-oslo-limit package has been added to RDO. This is the limit enforcement library which assists with quota calculation. Its aim is to provide support for quota enforcement across all OpenStack services. - The glance-tempest-plugin package has been added to RDO. This package provides a set of functional tests to validate Glance using the Tempest framework. - TripleO has been moved to an independend release model (see section TripleO in the RDO Xena release) The highlights of the broader upstream OpenStack project may be read via https://releases.openstack.org/xena/highlights.html *TripleO in the RDO Xena release:* - In the Xena development cycle, TripleO has moved to an Independent release model( https://specs.openstack.org/openstack/tripleo-specs/specs/xena/tripleo-inde… ) and will only maintain branches for selected OpenStack releases. In the case of Xena, TripleO will not support the Xena release. For TripleO users in RDO, this means that: - RDO Xena will include packages for TripleO tested at OpenStack Xena GA time. - Those packages will not be updated during the entire Xena maintenance cycle. - RDO will not be able to included patches required to fix bugs in TripleO on RDO Xena. - The lifecycle for the non-TripleO packages will follow the code merged and tested in upstream stable/xena branches. - There will not be any tripleoxena container images built/pushed, so interested users will have to do their own container builds when deploying xena. You can find details about this [in RDO webpage](https://www.rdoproject.org/documentation/tripleo-in-xena/) *Contributors* During the Xena cycle, we saw the following new RDO contributors: - Chris Sibbitt - Gregory Thiemonge - Julia Kreger - Leif Madsen Welcome to all of you and Thank You So Much for participating! But we wouldn’t want to overlook anyone. A super massive Thank You to all 41 contributors who participated in producing this release. This list includes commits to rdo-packages, rdo-infra, and redhat-website repositories: - Alan Bishop - Alan Pevec - Alex Schultz - Alfredo Moralejo - Amy Marrich (spotz) - Bogdan Dobrelya - Chandan Kumar - Chris Sibbitt - Damien Ciabrini - Dmitry Tantsur - Eric Harney - Gaël Chamoulaud - Giulio Fidente - Goutham Pacha Ravi - Gregory Thiemonge - Grzegorz Grasza - Harald Jensas - James Slagle - Javier Peña - Jiri Podivin - Joel Capitao - Jon Schlueter - Julia Kreger - Lee Yarwood - Leif Madsen - Luigi Toscano - Marios Andreou - Mark McClain - Martin Kopec - Mathieu Bultel - Matthias Runge - Michele Baldessari - Pranali Deore - Rabi Mishra - Riccardo Pittau - Sagi Shnaidman - Sławek Kapłoński - Steve Baker - Takashi Kajinami - Wes Hayutin - Yatin Karel *The Next Release Cycle* At the end of one release, focus shifts immediately to the next release i.e Yoga. *Get Started* To spin up a proof of concept cloud, quickly, and on limited hardware, try an All-In-One Packstack installation. You can run RDO on a single node to get a feel for how it works. Finally, for those that don’t have any hardware or physical resources, there’s the OpenStack Global Passport Program. This is a collaborative effort between OpenStack public cloud providers to let you experience the freedom, performance and interoperability of open source infrastructure. You can quickly and easily gain access to OpenStack infrastructure via trial programs from participating OpenStack public cloud providers around the world. *Get Help* The RDO Project has our users(a)lists.rdoproject.org for RDO-specific users and operators. For more developer-oriented content we recommend joining the dev(a)lists.rdoproject.org mailing list. Remember to post a brief introduction about yourself and your RDO story. The mailing lists archives are all available at https://mail.rdoproject.org. You can also find extensive documentation on RDOproject.org. The #rdo channel on OFTC IRC is also an excellent place to find and give help. We also welcome comments and requests on the CentOS devel mailing list and the CentOS and TripleO IRC channels (#centos, #centos-devel in Libera.Chat network, and #tripleo on OFTC), however we have a more focused audience within the RDO venues. *Get Involved* To get involved in the OpenStack RPM packaging effort, check out the RDO contribute pages, peruse the CentOS Cloud SIG page, and inhale the RDO packaging documentation. Join us in #rdo and #tripleo on the OFTC IRC network and follow us on Twitter @RDOCommunity. You can also find us on Facebook and YouTube.

1 0

[largescale-sig] Next meeting: Oct 27th, 15utc
by Thierry Carrez 27 Oct '21

27 Oct '21

Hi everyone, The Large Scale SIG meeting is back this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20211027T15 A number of topics have already been added to the agenda, including discussing the topic of our next OpenInfra.Live show! Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

How to control data write into a multiattach volume
by open infra 27 Oct '21

27 Oct '21

Hi, I have created a multiattach volume and attached it to several Windows VMs/Instances. Is there away in OpenStack to drop all data written by each VM/instances? I am supposed to periodically detach and attach the multiattach volume. Regards, Danishka

2 1

[cinder] Bug deputy report for week of 10-27-2021
by Sofia Enriquez 27 Oct '21

27 Oct '21

This is a bug report from 10-13-2021-15-09 to 10-27-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1948916 'Inconsistency for resource column between quota_usages and reservations table'. Unassigned. - https://bugs.launchpad.net/cinder/+bug/1947518 '[RBD] Cinder started requiring write access to glance images of RBD pool'. Unassigned. - https://bugs.launchpad.net/cinder/+bug/1947123 '[IBM GPFS] NFS driver throws an error while volume creation in copy-on-write mode'. Assigned to Digvijay Ukirde. - https://bugs.launchpad.net/cinder/+bug/1947134 '[IBM GPFS] NFS driver does the same filesystem check locally and fails initializing the driver in copy-on-write mode'. Assigned to Digvijay Ukirde. - https://bugs.launchpad.net/cinder/+bug/1946262 'NetApp ONTAP is losing QoS with assisted storage migration'. Assigned to Felipe Rodrigues. - https://bugs.launchpad.net/cinder/+bug/1948934 '[Victoria] Backported test requires newer ddt than included in test-requirements.txt'. Unassigned. Incomplete - https://bugs.launchpad.net/cinder/+bug/1947066 'snapshot status hang in backing-up'. Unassigned. Low - https://bugs.launchpad.net/cinder/+bug/1948507 'TypeError when computing QOS feature name'. Unassigned. - https://bugs.launchpad.net/cinder/+bug/1947834 'wrong href returned in volume detail'. Unassigned. -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[qa][ptg] PTG Summary
by Martin Kopec 27 Oct '21

27 Oct '21

Hi everyone, thank you all who participated in the PTG discussions and shared their thoughts and opinions. It's very much appreciated! Here is the summary of the topics we have discussed [1]: * Migration of devstack and Tempest tests to new secure RBAC ** Tempest side appear to be ready, further work is required on Devstack's side. * Xena retrospective ** Among good things belong: *** a few long term unfinished effort got completed *** a participation in an open source contest, which also helped in the above point *** a great response time in devstack patches *** quite prompt fixing devstack gate issues - thank you all who are active in Devstack, great job! ** Bad things to mention: *** not enough active contributors in Tempest and especially in Patrole project *** long review time in Tempest and Patrole * Cleanup of duplicated scenario.manager ** we will start with removing duplicated scenario.manager (Tempest's stable interface) methods from plugins ** this follows the 'making tempest.scenario.manager interface stable' effort done ~2 cycles ago * Test inheritance conventions, are there any ** we discussed usage of DDT library - it has turned out it complicates things when the test names are generated dynamically (it's hard to track the tests then) and because in DDT case the tests are not associated with any UUID ** * FIPS support ** current issues with FIPS at [3] - raised by Ade Lee ** the plan here from high level point of view is we'll use new jobs (with FIPS enabled) in Tempest so that we can identify and fix the tests that don't comply with FIPS * Patrole stable release updates and discussion ** A big question has been raised, do we have someone who can drive the patrole's migration to new RBAC? That's important because as mentioned before, there aren't many contributors in Patrole. ** There was an idea raised to retire the project - mainly because it's hard to maintain it (not enough contributors) and because the gates were broken for several months during the Xena cycle and mostly none had noticed - which raises another question - is the project needed/used by anyone? The discussed topics are transformed into priority items [2] we will be focusing on this cycle. [1] https://etherpad.opendev.org/p/qa-yoga-ptg [2] https://etherpad.opendev.org/p/qa-yoga-priority [3] https://etherpad.opendev.org/p/state-of-fips-in-openstack-ci-yoga Regards, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA

1 0