- openstack-discuss - lists.openstack.org

[openstack-ansible][ptg] Epoxy Project Team Gathering summary for OpenStack-Ansible
by Dmitriy Rabotyagov 04 Nov '24

04 Nov '24

Hi folks, Despite low participation in the PTG this time, we still had quite productive discussions around a variety of topics. You can find some summary on our future plans below: * We agreed to proceed with upgrading RabbitMQ to version 4.0 for Dalmatian. This means that HA (replicated) Queues won't be available anymore so Caracal will be the only release having support for both HA Queues and Quorum Queues. Deployments are expected to accomplish their migration to Quorum Queues (or just Classic Queues v2) on Caracal before proceeding further. * Implementing unified limits At this point of time, support for unified limits across services is not clear enough for us as well as how from operator perspective a mix of unified limits and classic quota will look like, as these 2 options are very distinct from each other. While one use case for their implementation would be vGPUs, we need an interested party for implementing them at this point. * Reviving Watcher support We have retired the Watcher role in Stein. With the project itself getting more traction these days, we agreed to proceed with reviving the role code to re-add support for the Watcher. * uWSGI and eventlet replacement A big topic throughout this PTG was replacement of eventlet with more native async approaches. However, uWSGI has no support for any of them right now. Since uWSGI is also in a maintenance mode, there is no expectations for adding async support to the project either. With that we need to start looking at alternatives, like uvicorn, which is already used in Skyline role. With that we potentially want to re-purpose the uWSGI role to be more generic at serving WSGI, meaning we will look into implementing service selection (uwsgi/gunicorn/uvicorn/etc) and renaming the role to address the more generic nature of it. * PKI role improvements This was another big topic we have discussed this time. There are multiple issues we need to address with the role. It could be split into multiple sub-topics: - Improving CA private key storage method on the deploy host to be able to store them encrypted with ansible-vault/sops or some other lookup mechanism - We need to implement better way for creating and storing private keys for generated certificates as transferring them from the deploy host is not really a best practice. - Adding support for hashicorp vault to store it in an encrypted manner. With that we should also add support to have CA in hashi-vault directly [1] and issuing certs from there. - Improve certs rollout process to allow short-term certificates to be issued and rotated regularly. This includes dynamically reloading libvirt config to address VNC consoles and reducing runtime for certificate rotation. * Would be great to have finally another round on ProxySQL efforts * Document of making user_secrets not a plain-texted file. Also address upgrade script failure in case user_secrets contain ansible-vault/sops/etc rather than plain text. * Document 2 ways of doing "pretty endpoints" - domain based and subpath based. * We need to work closely on deprecation of gather_subset starting with ansible-core 2.18, as there is no straightforward replacement. * Some repositories are missing deb822_repository migration, ie: frrouting, zun, qdrouterd [1] https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engi…

1 0

[neutron][ovn-octavia-provider][octavia][ovn] Octavia OVN DBs sync cmd
by Rico Lin 04 Nov '24

04 Nov '24

Hi all I would like to raise attention to the patch set that introduces Octavia OVN DBs sync cmd. patches: https://review.opendev.org/q/topic:%22add-ovn-db-sync-utils%22 With command `octavia-ovn-db-sync-util {--debug}` It should allows us to do the OVN DB sync similar to what `neutron-ovn-db-sync-util` does. And thanks to Fernando Royo, the patches are small and earlier to review in focused functions. Would be great if we could have some more reviews for it. *Rico Lin*

2 1

[tacker][ptg] 2025.1 Epoxy vPTG summary
by Yasufumi Ogawa 01 Nov '24

01 Nov '24

Hi tacker team, Thank you for joining the vPTG in the last week. I'd like to share the summary of our two days discussion for the proposals on the etherpad[1]. ## [recap] Revise FTs(yasufum) We have revised functional test from Bobcat and it's a recap session for looking back the remained issues. All the issues are going to be fixed in Epoxy. * Revise directory structure * Names of functions, classes and files too long for maintenance to be done this release * Combine scenarios into one to reduce the number of tests if possible. In process * Reduce setting up infra for the tests to be done this release * Revising scenarios and reducing setups are going to be started in this release. ## Plan for Improvement FT in Epoxy release (yasuum) We discussed the items about strategies for improving functional tests. To complete the goal of this proposal, we will take several steps to focus on dominant ones. (1) Divide time consuming tests, (2) Move tests less frequently updated to non-voting (3) Drop tests almost not updated from zuul (4) Rename each test scinarios and revise directory structure. We agreed with this proposal, but which of tests will be divided or dropped was not decided in the discussion. So, continue the discussion in the next IRC meeting. ## Add option to enable/disable Cinder volume recreation in v1 Heal request (koba) In the current v1 Heal request, the related Cinder Volume for the specified VDU is recreated although it can be specified not to recreate it. To support it in v1, Clients specify a flag in the additional_params to indicate whether to recreate the volume or not when making a Heal request. * Tacker conductor checks this flag to decide whether to mark the storage resources when executing STACK_UPDATE. Heal process works by recreating the resources marked "mark-unhealthy" during STACK_UPDATE. *Consider ensuring version compatibility and consistency with v2 operations. * Default behavior will be the same as now (recreating the Cinde Volume). * The name of the flag parameter in the additonal_params can be configured by the admin in tacker.conf. ## Add default_secret_key Option to [vim_keys] for Multi-Master Tacker Deployment (koba) ## Enhancement of the Ansible Driver (sample mgmt driver) (koba) Tacker currently supports only a single version of Ansible. As a result, when tenants include Playbooks in the VNF-Package that only work with specific versions of Ansible, there may be compatibility problems with the Playbooks. Tacker administrators must modify the Ansible Driver’s source code directly if they want to enforce options related to ansible-playbook execution, such as specifying log storage locations or custom callback plugins. These issues lack flexibility for both tenants and administrators. We agreed the proposed approaches for supporting several versions of ansible, but still not sure it's appropriate to define all the supported versions in tacker.conf and select which version will be applied by specifying it in the definition file in VNF package. It going to be discussed the way of selecting version in the next meeting. ## Remove legacy engine facade (yasufum) It's one of the community goal, so we'll fix it in this release. ## Remove sphinxcontrib-*diag from your (docs) dependencies (yasufum) It's similar to the previous item, and fix it in this release. ## Ensuring Tacker Compatibility with TST and SOL Specifications (Renu Rani) Agreed to the approach proposed on the etherpad. * Addressing the Gap Between TST and SOL Specifications: Raise bugs on TST and request the TST team to update schemas and variables according to SOL specifications. * Support of Compliance Tests for Tacker V2 a. Resolving Issues with Interconnected Test Cases in Tacker: Update Tacker's code to ensure compatibility with the interconnected test cases in v4.5.1-dev. b. Supporting Tacker for TST Release-4: Implement new Tacker compliance test cases for TST Release-4. c. Validating Compliance Test Cases with Tacker V2: Seek the Tacker team's opinion on whether to use Release 3 or Release 4 for validation. ## Enhancing Tacker Testing Coverage in the OpenStack Helm Check Pipeline (Shivam Shukla) To further enhance Tacker testing within the OpenStack Helm Check pipeline by adding more testcases to ensure thorough verification of Tacker deployments. Although the test scenarios were proposed, but we were not sure all the scenario are really needed because some of them are existing scenarios. We agreed to accept the proposal itself, but still not all the scenario should be introduced. So, continue to discuss for that. 1. VIM management (create, update and delete VIMs) 2. VNFD management (create and delete VNF packages) 3. VNF Instance management (create, update and delete VNF instances) 4. VNF LCM operation (instantiate, terminate, heal and scale) 5. Subscription management (create and delete subscription for VNFs) [1] https://etherpad.opendev.org/p/oct2024-ptg-tacker

1 0

[ironic] Bare Metal SIG
by Julia Kreger 01 Nov '24

01 Nov '24

Greetings folks, During this past week's PTG, the Ironic community participants reached consensus to wrap up the Bare Metal SIG. There are many reasons, but the bulk of the reasoning revolves around spreading word and collaborating outward as opposed to focusing on our own vertical of interaction. Having discussions in different contexts helps bring strength to discussions and we encourage contributors to engage in adjacent community and group discussions where relevant then build connections between communities. Communities, when working together, are always stronger when we build these sorts of connections or bonds. We hope to continue having meetups with operators as we always have, even if it may not bear the Bare Metal SIG name. As always, operators of any bare metal provisioning software are welcome to join our discussions at any time in OFTC IRC #openstack-ironic or right here on the mailing list. Among the accomplishments of the Baremetal SIG, was a whitepaper[0] devoted to the Ironic use case. Special thanks goes to Chris Hoge, previously with the Open Infra Foundation who helped drive us to the finish line on that effort. Beyond the whitepaper, and likely even harder was to bring people together in person on multiple occasions! A special thank goes to Arne Wiebalck who championed the Bare Metal SIG and helped steward several meetups including the most recent meetup at CERN in late Spring of this year. The success of that event speaks volumes about our community and the power of collaboration! Many thanks also goes to everyone who helped with the videos we have posted to youtube[1]! Without everyone pitching in, we wouldn't have been able to achieve success! -Julia, on behalf of the Ironic team. [0]: https://www.openstack.org/use-cases/bare-metal/how-ironic-delivers-abstract… [1]: https://www.youtube.com/playlist?list=PLKqaoAnDyfgoBFAjUvZGjKXQjogWZBLL_

1 0

[devstack][nova][cinder][ironic][glance][swift][neutron][all] Deprecating/removing non-uWSGI deployment mechanisms?
by Stephen Finucane 01 Nov '24

01 Nov '24

o/ tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is. I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack: * Eventlet server * Apache + uWSGI * Apache + mod_wsgi In addition, there are a number of overlapping config options that could be set to modify which of these was in use: * ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc. Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi. My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following: * Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more. I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think: https://review.opendev.org/c/openstack/devstack/+/932193/1 Cheers, Stephen [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/

6 10

[openstack helm] [Cinder] [Ceph] keyring not getting decode when doing cinder-storage-init
by daniel890723＠gmail.com 31 Oct '24

31 Oct '24

Hello today i was testing to deploy openstack with helm in my kubernetes cluster and i encounter an error when trying to deploy cinder Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred: * job cinder-storage-init failed: BackoffLimitExceeded when i print the log of the cinder-storage-init pod it shows 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error setting modifier for [client.admin] type=key val=: Malformed input [buffer:3] 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 monclient: keyring not found [errno 5] RADOS I/O error (error connecting to the cluster) and the keyring secret is use looks like this: apiVersion: v1 data: key: AQDtmCFnh3STCBAAGKQIyIGuZjnmX3L4mPCQXw== kind: Secret metadata: name: pvc-ceph-client-key namespace: openstack after checking secret and the rook-ceph cluster works fine i change the template and makes the pod not executing the command so i can look inside the pod and it's keyrings the keyring looks like this: [client.admin] key = �!g�tȁ�f9�_r��_ so apparently the keyring secert or when it mount into the pod got some problem and change the key into the normal one fix the problem is there a reason why it cause this ? because the secret and ceph-cluster are all works fine

2 4

[manila] 2025.1 Epoxy PTG summary
by Carlos Silva 31 Oct '24

31 Oct '24

Hello everyone! Thank you for the great participation at the PTG last week. We've had great discussions and a good turnout. The recordings for the sessions are available on YouTube [0]. If you would like to check on the notes, please take a look at the PTG etherpad [1]. *2024.2 Dalmatian Retrospective* ========================== - New core reviewers in the manila group were impactful in reviews, we should continue actively working on maintaining/growing the core reviewer team. - We had the mid-cycle and managed to combine it with our well-known collaborative review sessions, around feature proposal freeze. This had a good impact on raising awareness on the changes being proposed, as well as prioritizing the reviews. - Great contributions ranging from new third party drivers to successful internships on manila-ui, bandit and the ongoing OpenAPI internships. *Action items:* - Carlos (carloss) will work with the manila team to help people gain context on the bug czar role and work with the team to rotate it. - Vida Haririan (vhari) will jot down the details of the Bug Czar role - Follow the discussions on teams joining the VMT and get Manila included too. - Spread the word on the removal of the manila client and switch to OpenStackClient *Share backup enhancements* ========================= - Out of place restore isn't supported currently. We have agreed that this is a good use case and that a design specification should be proposed to document this. - DataManager / BackupDriver - forcing the backup process to go through the DataManager service is supported through a config option, but Manila is currently not honoring it. We agreed that this is an issue in the code, and we will review the proposed change [2] to make the data manager honor this config. - DataManager to allow for a backup driver to provide reports on API call progress: Currently, the data manager fetches the progress of a backup using a generic get progress call, but it is failing with the generic backup driver. We suggested that this should be fixed in the base driver. - Context for Backup API calls: currently, only objects representing a Share and Backup are passed to the backup driver. The request context should also be forwarded in these calls. The backup driver interface can be changed for this, but we should be mindful of out of tree drivers that could break. *Action items:* - Zach Goggins (zachgoggins) will look into: - Proposing a spec for the share backup out of place restore. - Updating the backup driver interface and adding context to the methods that need it. - Updating the backup driver interface and adding the abstract methods/capabilities that will help with the `get_restore_progress` and `get_backup_progress` methods. - The manila team will provide feedback on [2] *All things CephFS* =============== *Updates from previous cycles* ------------------------------------------ *State of the Standalone NFS Ganesha protocol helper:* - We added a deprecation warning at the end of the previous SLURP release, and we are planning to complete the removal during the 2025.1/Epoxy release. There were no objections to this so far at the PTG. When this is removed, CephFS-via-NFS will only work with cephadm deployed ceph-nfs clusters. *Testing and stabilization:* - devstack-plugin-ceph has been refactored to deploy a standalone NFS-Ganesha service with a ceph orch deployed cluster. We also dropped support for package-based and container-based installation of ceph. cephadm is used to deploy/orchestrate ceph. - Bumped Ceph version to Reef in Antelope, Bobcat, Caracal, Dalmatian, as well as started testing with Squid. - There are some failures on stable branches jobs which are being triaged and fixed. *Manage/unmanage:* - Implementation completed in Dalmatian and the documentation has been updated. We are currently working to enable the tests on CI permanently, as well as doing some small refactors to the CI jobs. *Ensure shares:* - Merged in Dalmatian but testing is still challenging, as running the tests mean that the service would temporarily have a different status and shares within the backend would have their status changed, which is harmful for test concurrency. *Preferred export locations and export location metadata:* - The core feature merged, but we are still working to get the newly implemented tests passing and merged. *Plans for 2025.1/Epoxy* -------------------------------- - NFSv3 + testing: we are looking into enabling NFSv3 support as soon as the patch is merged in Ceph. We agreed that we should enable the tests within manila-tempest-plugin and make any necessary changes to the tests structure, so we can ensure that we are testing some scenarios with both NFSv3 and NFSv4. - We will start to investigate support for SMB/CIFS shares and look at the necessary changes for setting up devstack and testing. *Action items:* - Carlos (carloss) will write an email to the openstack-discuss mailing list announcing the removal of the deprecated ganesha helper - Carlos pursue the manage/unmanage testing patches to have tests enabled in the CephFS jobs during Epoxy. - Carlos will look into approaches to test ensure shares APIs. - Ashley (ashrod98) will continue working on the export location metadata tempest changes and drive them to completion. - The manila team will look into updating manila-tempest-plugin tests and enabling NFSv3 tests in the Ceph NFS jobs - Goutham (gouthamr) will be submitting a prototype of the SMB/CIFS integration *Tech Debt* ======== *Eventlet removal* ---------------------- Our main concerns: - Performance should not be degraded with the default configuration when we switch. - Synchronous calls do not take a big hit and become asynchronous. - Impact to the SSH Pool (used by many drivers) should be minimal. *Action items for 2025.1 Epoxy:* - Tackle the low-hanging-fruit changes. - Participating in the pop-up team discussions. - Removing the affected console scripts in Manila. - Working on performance tests to understand what will be the impact on the SSH pool that is used by some drivers. - Look into enhancing our rally/browbeat test coverage. *CI and testing images* ------------------------------- We started working on the migration of the CI to Ubuntu 24.04 in all of the manila repositories (manila-image-elements, python-manilaclient, manila-ui, manila, manila-specs). Currently, the Ceph job is broken [3]. *Action items:* - We should clean up our CI job variants, as they have a lot of workarounds and we can start moving away from them. *Stable branches* ---------------------- We currently have 5 "unmaintained" branches, so we should be looking at sunsetting them. *Action items:* - Carlos (carloss) will start the conversation for the transition of some of these branches in the openstack-discuss mailing list. *Allowing force delete of a share network subnet* ======================================= We currently can add subnets (which translates to adding new network interfaces) to a share server but we can't remove them. This is a proposal to add this removal feature and being able to detach a network interface of a share server. We agreed that: - This is a good use case and something that can be enhanced. - The enhancement should add a force-delete API. - We should not allow the last subnet to be deleted, otherwise the shares won't have an export path. - A bug should be filed for a tangential issue that the NetApp driver is using "neutron_net_id" (and possibly "neutron_subnet_id" to name resources on the backend: ipspaces, broadcast domains, and possibly concurrency control / locks. *Action Items:* - sylvanld will look into proposing a spec to document this behavior *NetApp Driver: Enforcing lifs limit per HA pair* ===================================== - The NetApp ONTAP storage has a limit of network interfaces per node in a HA pair. In case the sum of allocated network interfaces in the two nodes of the HA pair is bigger than the limit of the single node, then the failover operation is compromised and will fail. - NetApp maintainers would like to fix this issue, and we agreed that: - The fix should be as dynamic as possible, not relying on users/admin input or configuration. - The ONTAP driver must look up all of the interfaces already created and allow/deny the request in case it would compromise the failover. - The NetApp ONTAP driver should keep an updated capability with the max network interfaces support number, and possible the number of allocated network interfaces at the moment. *NetApp Driver: Implement Certificate based authentication* ================================================ - The NetApp ONTAP driver currently handles only user/password authentication, but in an environment that password should change quarterly, this means updating the local.conf at least every three months. This enhancement proposes also adding the possibility of adding certificate based authentication. - We agreed that this is something that is going to be important for operators and will allow them to add their certificates with a longer expiration date, avoiding the disruptions caused by needing to update the user/password. *Manage Share affinity relationships by annotation/label* ============================================= Currently the manila scheduler uses affinity/anti-affinity hints and we base ourselves on share IDs. The idea now would be to have the affinite hints to be based in an affinity policy, as possible with Nova. We considered the proposed approaches, and agreed that: - If we are adding new policies, they should end up becoming a new resource/entity within the manila database - If there is a way to reuse the share groups mechanism, we should prioritize it *Action items:* - Chuan (chuanm) will propose a design spec to document this new behavior. *Share encryption* ============== This feature is currently waiting for more reviews and testing on gerrit. In the Dalmatian release mid-cycle we talked about the importance of testing this feature against a first party driver, to ensure that the APIs and integration with Barbican and Castellan work. We agreed that: - We should do some research on how to do this testing with the generic driver (which uses Cinder and Nova) - The testing will focus on the APIs and behavior of this feature, not the encryption of the shares. *Action items:* - gouthamr will help with some research on how to test this with the generic driver - The manila team will discuss this again in the upcoming manila weekly meetings. [0] https://www.youtube.com/watch?v=8UxrjEr6yik&list=PLnpzT0InFrqDHGfSDPhiGtSeX… [1] https://etherpad.opendev.org/p/epoxy-ptg-manila [2] https://review.opendev.org/c/openstack/manila/+/907983 [3] https://www.spinics.net/lists/ceph-users/msg83201.html

1 0

[qa][all] stable/2023.1 gate is blocked (hold the recheck)
by Ghanshyam Mann 31 Oct '24

31 Oct '24

Hello Everyone, Yatin pointed out that the python 3.8 drops impacted the stable/2023.1 focal jobs also. As per the stable/2023.1 testing runtime[1], we run at least one job on focal and the rest all on jammy. Python 3.8 drops broke the focal job[2]. To solve this, we need to pin the Tempest on stable/2023.1 testing (Yatin has proposed it[3]) We are testing it on neutron gate - https://review.opendev.org/c/openstack/neutron/+/933705 Until the fixes[3] are merged, please hold off on rechecking stable/2023.1 focal job failure. [1] https://governance.openstack.org/tc/reference/runtimes/2023.1.html#addition… [2] https://bugs.launchpad.net/devstack/+bug/2085886 [3] https://review.opendev.org/q/topic:%22bug/2085886%22 -gmann

1 1

[python-openstackclient] List security group IDs used by a VM with "openstack server show"
by Sebastian Luna Valero 31 Oct '24

31 Oct '24

Hello, Currently "openstack server show" provides the list of security groups used by the VM. However, this is a list of names only [1]. Therefore a VM may contain multiple security groups with the same name but different IDs. However, getting the IDs of the security groups used by a VM seems possible according to the API [2]. Would it be possible to make this available via the python-openstackclient CLI? Many thanks, Sebastian [1] https://docs.openstack.org/api-ref/compute/#show-server-details [2] https://docs.openstack.org/api-ref/compute/#list-security-groups-by-server

2 1

[barbican] KMIP integration
by przemyslaw.kuczynski＠ochk.pl 31 Oct '24

31 Oct '24

Why KMIP integration always create OPAQUE secret with name Opaque Object . There can be created only 1 entry Is there bug ? https://bugs.launchpad.net/barbican/+bug/2033620

2 1