- openstack-discuss - lists.openstack.org

[TripleO][Ceph] Zed PTG Summary
by Francesco Pantano 15 Apr '22

15 Apr '22

Hello everyone, Here are a few highlights on the TripleO Ceph integration status and the plans for the next cycle. *1. Deployed ceph as a separate step* TripleO now provides a different stage (with a new cli set of commands) to bootstrap the Ceph cluster before reaching the overcloud deployment phase. This is the new default approach since Wallaby, and the short term plan is to work on the upstream CI consolidation to make sure we run this stage on the existing TripleO standalone scenarios, extending the coverage to both phases (before the overcloud deployment, when the Ceph cluster is created, and during the overcloud deployment, when the cluster is finalized according to the enabled services). It's worth mentioning that great progress in this direction has been made, and the collaboration with the tripleo-ci is one of the key points here, as they're helping on the automation aspect to test upstream pending bits with daily jobs. The next step will be working together on the automation of the promotion mechanism, which is supposed to make this process less error-prone. *2. Decouple Ceph Upgrades* Nautilus to Pacific is still managed by ceph-ansible but the stage of upgrading the cluster has been moved before the overcloud upgrade, resulting in a different maintenance window. Once the cluster is moved to Pacific, cephadm is enabled, and from this moment onwards, the upgrade process, as well as minor updates, will be managed by cephadm and can be seen as a day2 operation. The operator can now perform these kinds of tasks without any interaction with TripleO, which is still used to pull the new containers (unless another registry reachable from the overcloud is used), but the scope has been limited. *3. Ganesha transitioning to Ceph orchestrator and Ingress migration* This has been the main topic for this first ptg session: the feature it's tracked by two already approved upstream specs and the goal is to support a Ganesha service managed by cephadm instead of a tripleo-managed one. The TripleO conversation impacted many areas: *a.* the networkv2 flow has been improved and it's now possible to reserve more than 1 VIP per network, but it applies only to the ceph services; *b.* a new TripleO resource, the CephIngress daemon, has been added, and it's a key component (provided by Ceph) that is supposed to provide HA for the ceph-nfs managed daemon *c.* The tripleo cli is extended and the ceph-nfs daemon can be deployed during the bootstrap of the ceph cluster *d.* This feature depends on the manila driver development [1], which represents an effort to implement a driver that can interact with the Ceph orch cli (and the layer it provides for nfs) instead of using dbus. Further information about this conversation can be found here [1]. Part of this conversation (and really good input here actually) was about the migration plan for already existing environments where operators would like to move from a TripleO managed Ganesha to a highly available ceph-nfs managed by cephadm. The outcome here is: *1.* It's possible to move to the cephadm managed ingress daemon during the upgrade under certain constraints, but we should provide tools to finalize the migration because there's an impact not only on the server-side (and the manila service itself) but also on the clients where the shares are mounted; *2.* We might want to have options to keep the PCS managed VIP for Ganesha and avoid forcing people to migrate, and this flow should be consistent at tripleo heat templates level; For those who are interested, here's the etherpad [2] and the recording of the session [3]. Thanks, Francesco [1] https://etherpad.opendev.org/p/zorilla-ptg-manila-planning [2] https://etherpad.opendev.org/p/tripleo-zed-ceph [3] https://slagle.fedorapeople.org/tripleo-zed-ptg/tripleo-zed-ptg-ceph.mp4 -- Francesco Pantano GPG KEY: F41BD75C

1 0

[tacker] Zed PTG summary
by Yasufumi Ogawa 15 Apr '22

15 Apr '22

Hi, Thank you for joining the PTG. Here is a summary of Zed PTG of Tacker. We discussed 17 proposals through three days and agreed to whole topics. The details of proposals are on the PTG's etherpad [1]. - heat-translator maintenance - Ayumu Ueha proposed himself as a core of heat-translator from Tacker team and approved by the team. - Enhance security of APIs from Tacker for validating the certificate of a destination server. - Enhancement of Tacker-horizon Dashboard for enabling some Day2 operations (might include a part of Day0/1 operations), such as manual heal, modify VNF, change the configuration of VNFM, etc, and monitoring through GUI This blueprint proposes to extend the tacker-horizon dashboard with some new interfaces and functionalities. - Support VNF using Hardware Acceleration such as GPU, FPGA, ASIC. - Remove restrictions of using Helm Chart. - v1 API Refactoring in order to maintain and extend Tacker continuously, it's important to improve the maintainability of the v1 API. - CLI support for paging query results to improve usability of the cli tool. - We still need to discuss how we provide the pagination of the results. - Support multi artifact of ansible driver. It's already to be under reviewing. https://review.opendev.org/c/openstack/tacker/+/836107 - Enhance Multi tenant policy in LCM for enhancing of the multi-tenant policy in the VNF lifecycle management. It enables a non-admin role user to instantiate VNF, and defines tenancy isolation between admin and the non-admin role users. - Enhancement of CNF operations for k8s usecases. - https://blueprints.launchpad.net/tacker/+spec/database synchronization - https://blueprints.launchpad.net/tacker/+spec/enhance-cnf-lcm - https://blueprints.launchpad.net/tacker/+spec/enhancement-container-update - https://blueprints.launchpad.net/tacker/+spec/suport-openid-k8s-vim - https://blueprints.launchpad.net/tacker/+spec/support-instantiationlevel-cnf - Support of automatic operations - https://blueprints.launchpad.net/tacker/+spec/support-auto-lcm - https://blueprints.launchpad.net/tacker/+spec/support-autoheal-queue - Improve V2 code's UT Coverage and refactoring Tacker from caishuwen. - Enhance VNF Update functionality. - https://blueprints.launchpad.net/tacker/+spec/enhance-change-package - https://blueprints.launchpad.net/tacker/+spec/individual-vnfc-management - Support HA-cluster for high availability is an important requirement to apply Tacker to commercial systems. HA cluster such as ACT-SBY or N-ACT will be supported. - Enhance vnflcm API (hirofumi-noguchi) - https://blueprints.launchpad.net/tacker/+spec/enhance-placement - https://blueprints.launchpad.net/tacker/+spec/enhance-utils-extmanagedvl - https://blueprints.launchpad.net/tacker/+spec/multi-version-vnfd - https://blueprints.launchpad.net/tacker/+spec/sol004-package-management - https://blueprints.launchpad.net/tacker/+spec/support-subscription-cli - https://blueprints.launchpad.net/tacker/+spec/vim-management-reorganization - https://blueprints.launchpad.net/tacker/+spec/abolish-default-vim - Tacker's new requirements for commercial systems (hirofumi-noguchi) - https://blueprints.launchpad.net/tacker/+spec/enhance-system-management - https://blueprints.launchpad.net/tacker/+spec/command-conflict-handling - https://blueprints.launchpad.net/tacker/+spec/system-performance-management [1] https://etherpad.opendev.org/p/tacker-zed-ptg Thanks, Yasufumi

1 0

[largescale-sig] Large Scale SIG Forum session in Berlin?
by Thierry Carrez 15 Apr '22

15 Apr '22

Hi everyone, The CFP for Forum sessions is open until Wednesday next week at: https://cfp.openstack.org/app/berlin-2022/18/presentations Should we file a session for the Large Scale SIG? Could be a generic session ("Meet the Large Scale SIG") or a more precise topic. -- Thierry Carrez (ttx)

1 0

client authentication - ? gui okey but cli fails
by lejeczek 15 Apr '22

15 Apr '22

Hi guys. An end user here thus please tailor your advice according - meaning that I have no admin access. With GUI in a web browser I do log in but with CLI I get "usual": The request you have made requires authentication. (HTTP 401) (Request-ID:.... (rc file downloaded off the GUI) What can be the issue? many thanks, L.

2 1

[all][tc] Dropping the lower constraints maintenance
by Ghanshyam Mann 15 Apr '22

15 Apr '22

Hello Everyone, In zed cycle PTG, TC discussed the lower constraints maintenance again and after considering all the existing and current issues (discussed any times in the ML thread[1]), we agreed to drop the lower-constraints.txt file, x env, and its testing on master as well as on stable branches. But for reference to anyone using them (Debian as we know), we will keep them as it is in the requirements.txt file with our best effort to keep them up to date. I have proposed the TC resolution for this, if you have any feedback please add it on Gerrit patch: - https://review.opendev.org/c/openstack/governance/+/838004 [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019659.… http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019390… http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019672.… -gmann

1 0

[cyborg][zun][masakari][freezer-api][heat][tacker][oslo] Pending removal of 'oslo_db.sqlalchemy.enginefacade.LegacyEngineFacade'
by Stephen Finucane 14 Apr '22

14 Apr '22

o/ This is a heads up to the maintainers of the aforementioned projects that the oslo team are planning to remove the 'oslo_db.sqlalchemy.enginefacade.LegacyEngineFacade' class and related 'get_legacy_facade' helpers this cycle. The legacy engine facade pattern has been deprecated since 2015 and we want to get rid of it to reduce load on the overburdened oslo maintainers. The prefered alternative is to rely on oslo.context RequestContext-based session management. The work required to migrate can be quite significant depending on how large your DB API is but it is pretty trivial. This issue has already been fixed in a couple of projects. These can be used as blueprints for fixing the remaining affected projects: * cinder (https://review.opendev.org/q/topic:remove-legacyfacade) * nova (https://review.opendev.org/q/topic:bp%252Fnew-oslodb-enginefacade) We expect this one to take affected projects a while to resolve, so we won't remove this module until near the end of the cycle. However, this thing really needs to be removed sooner rather than later. Please let us know if you have any concerns or if the above changes are not sufficient as a guide for how to address these issues. Cheers, Stephen

1 0

[neutron] Drivers meeting - Friday 15.4.2022 - cancelled
by Lajos Katona 14 Apr '22

14 Apr '22

Hi Neutron Drivers! We have Good Friday this week, so let's cancel the drivers meeting for this week. See you at the meeting next week. Lajos Katona (lajoskatona)

1 0

[all] Devstack jobs are failing due to a git security fix
by Michael Johnson 14 Apr '22

14 Apr '22

tldr: All devstack based jobs are going to fail with newer versions of git - don't bother rechecking git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack. We have opened a devstack bug to track this issue: https://bugs.launchpad.net/devstack/+bug/1968798 Please see the bug for the details and example error. Michael P.S. Thanks to clarkb, fungi, and ianw for helping track down the root cause! [1] https://github.com/git/git/commit/8959555cee7ec045958f9b6dd62e541affb7e7d9

7 11

[puppet] How to disable installation of mysql::server and only create databases?
by Christian Rohmann 14 Apr '22

14 Apr '22

Hey openstack-discuss, is there any intended way of using e.g cinder::db::mysql (which is using puppet-openstacklib) to create databases, but not also run / include mysql::server which apparently happens at https://github.com/openstack/puppet-openstacklib/blob/33fb90326fadd59759d4a… ? In short I already have a running database server and only want the databases to be created. Thanks, Christian

2 2

[manila] Zed PTG summary
by Carlos Silva 14 Apr '22

14 Apr '22

Hello, zorillas and interested stackers! Thank you for attending the PTG over the last week. We had a good audience and a lot of good discussions over the proposed topics. On the official etherpad [9] you will find the notes we took during the discussions, as well as links to additional etherpads or other references. The YouTube playlist [14] holds all the recordings we had over the week. Every video corresponds to one PTG day, and their description contains the exact time frame of the discussions. *== Xena cycle retrospective ==* * We agreed and were happy with the events we have been doing in the community (collaborative review sessions, bugsquashes and some other). They help us to understand the changes that are coming in every cycle, and also get to share knowledge with the contributors. For this cycle we will define different themes for our bugsquashes. Some dates are already defined. * We agreed on the importance of the reviews from different affiliations and we decided on a few practices to involve more community members in the review process. * We brought up the enhancements we need to do in the Manila UI, and mentioned some features that we are lacking for feature parity compared to the Manila core code. * We decided to host a hackathon in the beginning of the cycle, as we had a very positive output from the last one we had. This one can focus on the functional testing of the OSC for Manila, which we are lacking. * During the Z cycle we will host the first code walkthrough for Manila. (kudos for gouthamr for the idea). The intention of this event is to help the new contributors to better understand the code and allow them to ask questions, as well as explain the structure of our repositories and go over some key parts of Manila. * We discussed the difficulties we are facing with ethercalc and we got to a point where we saw there was no alternative way to replace it. More details about the retrospective available in [1]. *== Thin Provisioning and Oversubscription improvements ==* * haixin and gouthamr brought up this topic after a few discussions on how the oversubscription issue should be solved in Manila. * Storage pools that support thin provisioning are open to an "oversubscription". * We have a ratio for allowing oversubscriptions and it also influences the way the manila scheduler filters acts. We currently have issues with that calculation. * If the backends do not report their allocated_capacity_gb, it will be inaccurate. * We can have the share manager to have a more precise calculation. * Action Items: ** haixin will write a spec with more details on how this issue is going to be addressed. *== NetApp ONTAP - migration from ZAPI to REST API ==** nahimsouza and fabioaurelio talked about the NetApp ONTAP drivers that are currently using ZAPI for communication with NetApp hardware. * NetApp is now planning to deprecate ZAPI in favor of REST. This means that if they find issues within the ZAPI calls, NetApp won't fix them anymore. * The side effect of this is that NetApp wants to migrate their driver calls from ZAPI to rest, but it will impact pretty much all operations the driver performs. * These are going to be huge changes and they intend to backport it to the oldest maintained release. * They have lots of customers using the latest versions of ONTAP but those customers do not often upgrade their OpenStack deployments. * They are looking for approaches on how to backport these changes even further. *== Add support for automatic snapshot creation and removal ==** Kiran brought up the solution proposed with a spec [2], which intends to add an automatic snapshot creation policy and automatic snapshot deletion policy for a share, considering an interval defined by an administrator. * We've asked for possible alternatives such as automating the snapshots creation via manilaclient/REST API, and also raised a few questions that would help us to enforce the need of enhancing the API. * Action Items: ** Kiran will come back with updates on the discussion points. *== Manila UI updates ==** vkmc discussed the updates for Manila UI alongside the NDSU students. * They talked about the plans we have for the UI in this cycle, as well as bringing up the feature-parity issue we currently have. * We started by filing blueprints and identifying the scope of necessary changes to get feature parity. * A taiga board [3] was created to track the progress. * We also have a few blueprints [4] for that. *== Consistent and Secure RBAC changes ==** gouthamr, gmann and vhari brought up the plans for the Z cycle, which would be continuing the plans highlighted on [5] So for Z, the idea is to: * Harden Phase 1 by re-evaluating the use of "admin" at system scope, disallowing system scope users from creating/manipulating project resources * Continue working on tempest protection tests. Liron has a patch pending reviews that starts this work [6] * Test existing tempest API/scenario tests with the new RBAC defaults and adjust any credential use or API expectations by the end of the cycle * Start working on the phase 2 per the community goal in [5] Beyond Z: * A release - Phase 3: System Reader and System Member in the default RBAC * B release: Remove deprecated policies * == OpenStack Client updates for Manila ==** maaritamm presented the progress we had so far for the OSC integration. * We are quite close to feature parity. The idea is to reach feature parity in the Z cycle and add a deprecation warning to the native client shell in python-manilaclient. * We are lacking functional tests and we came up with a plan of organizing a hackathon to be hosted close to the z-3 milestone. * This hackathon will be focused on addressing the functional tests for the client and wrapping it up. * As enforced in the past cycle, maintainers implementing new functionalities to python-manilaclient must _only_ implement them for OSC. * New features in the python-manilaclient shell will actively be discouraged. * A more detailed version is available in [7]. *== Support for highly available NFS Ganesha in the CephFS driver ==** vkmc, fmount and gouthamr presented the the Ceph orchestrator tooling that now natively supports deploying nfs-ganesha gateway servers as "ceph-nfs" cluster daemons in active/active configurations. * The Ceph community has also added support for nfs APIs to create and manipulate exports on such ceph-nfs daemons. * We discussed the changes needed to support these in Manila and how users can migrate their workloads. To provide adequate control to deployers and end users, we agreed to introduce a new protocol helper layer that will live alongside the current DBUS API helper. * When the deployment supports new NFS clusters, deployers would be able to configure "alternate" server IPs in manila which can aid a slow transition for end users to migrate to newer "preferred" servers. * We're planning to automate this transition in Triple-O while also documenting this for developers of other deployment tools. *== Devstack-plugin-ceph changes ==* * vkmc and fmount presented the changes they are implementing in order to install/deploy the ceph cluster using cephadm. There is an open change for review [8]. We are also testing the Manila CephFS job in [9]. * Native CephFS scenario tests concerning ceph-fuse are failing sporadically with timeouts while mounting. * There are few issues with testing and there is more information about them in [9]. As next steps we have: * Testing with CentOS as the devstack host and Manila's client, so we have different approaches of testing and can ensure that ubuntu testing isn't compromising the VMs and causing the possible lack of resources we are experiencing. * Splitting CI changes into different test sets focusing on different services * Turning on cephadm based deployment in the multi-node job *== Topic #Bug updates ==** vhari has shown some bug analytics and the progress we have been making over the past cycles. * We have a bunch of bugs that are old and no one has been touching them in a while (at least 2 years). * Maintainers should check launchpad and evaluate those bugs we created and/or are assigned to us and see if they are still relevant. If they are, we should keep active on them, otherwise we can just let it fall into the auto-expiry policy. * Bugsquashes have been efficient for us, we should keep doing them. ** fabioaurelio suggested a bugsquash theme that would double up as a review jam. We'd be focusing on looking at the lingering changes that are closing bugs and getting closure for them. * When we don't have the time to do the bug triaging during the Manila meeting, we will do the triaging in #openstack-manila after. * We will be hosting our bugsquashes in Z-1 and the other one between Z-2 and 3. * Action items: ** carloss will update the change with manila specific milestones to reflect our scheduled events *== Metadata API status update ==** This is an effort that started some cycles ago. We intend to allow metadata on more user facing resources in Manila [10], and do that in a generic way. * Over the Yoga cycle that was accomplished for shares, and there is a change to also permit that for share snapshots. What's next: * Getting the share snapshots changes merged over the Z cycle (M-1) * Getting the new metadata mechanism also implemented for * Share Export Locations - Target: M3 (at least API + CLI) * Share Access Rules - Target: M3 (at least API + CLI) * Share Groups - Target: M3 (at least API + CLI) *== NetApp CI Zuul v3 Migration - Challenges and lessons learned ==** Building and maintaining a stable CI system has been a huge pain point to Manila's "third party" vendor storage developers for the past few releases. * NetApp contributors have talked about their Zuul v3 setup using Software Factory. They brought up some information on their Zuul v3 CI setup and talked about some difficulties they faced during the setup. * The recording is on Manila's YouTube channel [11]. There were questions asked and raised and the discussion documented in [9]. * Action items: * Through the release cycle, we hope to offer more venues for more knowledge sharing regarding third party CI maintenance. *== FIPS compliance ==** carloss ashrod98 and ade_lee talked the attendance through the challenges and testing for FIPS * We've been through the next steps and what has been done so far in Manila ** Few changes were proposed and merged ** Our CI job is working fine with CentOS 8 but we decided we should already have it on CentOS 9 * We needed to monkey patch paramiko for some drivers ** gouthamr is concerned that if this patching paramiko change is merged, this could break older branches in the future ** A discussion in openstack-discuss will be started to ensure if this is the best approach for this issue. * At the moment, we should stick with testing using CentOS but testing with Ubuntu is under way. * Compliance steps will start on Z release * ade_lee started an audit in OpenStack services to identify non FIPS compliant libraries * Goal should be completed on AA cycle * Action Items: ** Start a discussion on patching paramiko or switching to other lib ** Submit the jobs for python-manilaclient, manila-ui and manila-tempest-plugin *== Tech Debt ==*These are a few items we would like to get moving for Manila in the next cycles. They usually involve a community goal, a request for enhancement or recommendations from the community: * Migrating from privsep to rootwrap: In progress, few changes were merged over Y cycle and we intend to have even more progress over Z. * Dropping python-keystoneclient from python-manilaclient: this is also in progress and we got a few related changes merged. We will bring this up again in the upcoming manila meetings. * Cinder/Nova online extensions support with the generic driver: this hasn't started yet and we are looking for an owner. This is more of a request for enhancement that would benefit those using the generic driver. * 26 volumes limit in the generic driver: this is something that was called out in the generic driver documentation. We agreed to try an approach where we would update the manila image and check the output for it. * Publish the container driver image on tarballs.openstack.org (or quay.io): we had few changes to the container driver in the past cycles. One functionality was implemented on it (add/update security services in share networks that are used), and for that change to be tested, the container must have OpenLDAP installed. We have a way to publish new artifacts and we could benefit from reusing that. Action items: ** carloss will work on having the new image proposed ** Bring the keystoneclient change in the upcoming manila meetings. ** Update the manila image setting hw_scsi_model=virtio-scsi and investigate if the 26 volumes limit will be solved ** Work on finding people interested to contribute in the Cinder/Nova online extensions support with the generic driver *== Manila CSI updates ==** gman0, gouthamr and vkmc talked us through some of the updates in the Manila CSI since the last PTG [12]. * We don't currently test using RWO volumes with manila-csi in cloud-provider-openstack, so we could add an "external-attacher" sidecar to ensure RWO-ness * Doing backups with manila-csi and velero/restic * Action items: ** open an issue against the CPO repo and work on testing with RWO ** open a redmine tracker to call the limitation of giving huge names to snapshots in the docs *== Community Hour ==** gouthamr brought up some details about the release cadency adjustment [13] * Also, mentioned a few implications of this and how deprecations, testing and upgrades will work * Few details in the deprecation windows and how we should be working and targeting changes in the next few cycles. * A job was recently merged in Manila to test the upgrade from tick to tick releases * Action items: ** Update the tick release job to the Xena branch [1] https://etherpad.opendev.org/p/manila-yoga-retrospective [2] https://review.opendev.org/c/openstack/manila-specs/+/823165 [3] https://review.opendev.org/c/openstack/manila-specs/+/823165 [4] https://blueprints.launchpad.net/manila-ui/+spec/api-version-features [5] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… [6] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/805938 [7] https://etherpad.opendev.org/p/zorilla-ptg-manila-osc [8] https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 [9] https://etherpad.opendev.org/p/zorilla-ptg-manila [10] https://specs.openstack.org/openstack/manila-specs/specs/yoga/metadata-for-… [11] https://www.youtube.com/watch?v=Pn1ZEnlHE7A [12] https://etherpad.opendev.org/p/zorilla-ptg-manila#L413 [13] https://governance.openstack.org/tc/resolutions/20220210-release-cadence-ad… [14] https://www.youtube.com/watch?v=AIqrLdprkaE&list=PLnpzT0InFrqCjifjP1OzFgnfi…

1 0