April 2023 - openstack-discuss - lists.openstack.org

[nova] 2023.2 Bobcat PTG summary
by Sylvain Bauza 03 Apr '23

03 Apr '23

Hey folks, that's a wrap ! I'm happy to say we had a very productive PTG week with around more than 12 contributors each day. It was nice to see new faces. As a reminder, please don't open the main etherpad link [1] if you use a Web browser with an automatic translation modification feature. If you do it, then it would internally translate all the phrases directly in the etherpad as if you were directly modifying the etherpad by yourself. Please also make sure you don't accidentally remove or modify lines . In order to prevent any misusage, this etherpad is a read-only copy of the main one : https://etherpad.opendev.org/p/r.22d95e20f7133350dcad573c015ed7da Last point but not the least, I'm a human who usually writes bugs [2], including when it's about writing a report of the past week's discussions and the actions taken. Don't feel offended or irritated if you read silly notes of the very beloved chat you had, please rather fix it the opensource way by correcting my errors with a reply on that thread. Enough chat, please grab a coffee [3] (as it's a quite verbose email) and jump straight on below. ### Operator hour ### (specific and distinct read-only etherpad here : https://etherpad.opendev.org/p/r.f94e6dda608450b6a4a5c9afc76e8a88 ) Yet again, we missed a few more operators in this room (at least we had 3 of them, thanks for those that were there !). I'm still wondering why they are not joining a free session where they can discuss their concerns or some wanted features. Anyway, that's what we discussed : - for the moment, no operators have tried to use the Yoga unified limits feature. - operators are happy with the new default RBAC scopes. We could create a small and quick admin-guide section for explaining that and pointing to the oslo.policy documentation. - operators would love to use the (possibly) new virtiofs feature from Bobcat for Manila shares (if we are able to merge it). That being said, they think they couldn't provide it for their environments until Nova can live-migrate the instances using the shares (which is not possible as for the moment, even QEMU doesn't support it) - we also discussed a pain point from an operator and then we asked him to provide a bug report (related to the vm state continuing to be ACTIVE after shelving). For the moment, we can't reproduce the problem. ### Cross-project discussions ### ## Manila/Nova ### # Prevent shares deletion while it's attached to an instance - Nova could add some traits to reflect the share type. - the Manila team will create a manila spec for adding a service-level (or admin-level) lock API with a 'locked-by' parameter (or using service tokens if set) ### Neutron/Nova ### Read Neutron summary email [4] if you want more details. - We agreed on a delete_on_termination feature for ports. We need to see if we can use some already existing Neutron API (tags) or if we need a whole new API. - We are quite OK with the Napatech LinkVirt SmartNics feature, but we need to review the spec and we also want some third-party CI support for it. - the Neutron team accepted to return a HTTP409 Conflict return code if a duplicated port binding action is done by Nova and the Nova team agreed on handling it. - About the SRIOV port tracking in Placement, the Nova team needs to help the spec owner on how to work on it as it's a large and difficult design (but we agreed on the first part of the design). ### Glance/Nova ### Glance wants to add a new API for Image Direct URL access. Nova was quite OK with this new API, but the Nova team was having some concerns about upgrades and client calls to the new API. The Manila team agreed to create a nova spec for discussing those. ### Cinder/Nova ### - The Nova team is quite OK with the usecase for NFV encryption. Since the design is quite simple, maybe this could be a specless feature, just using traits for telling to Placement which computes support NFV encryption. We're OK to test this with cinder NFV jobs and maybe also testing it with a Nova periodic job depending on the implementation. - The Cinder team agreed on using its volume metadata API for tracking the cinder volumes hardware models. ### Horizon/Nova ### Horizon wanted to call the Placement API. We agreed on using the openstackSDK for this as Placement doesn't have python bindings directly. Some new methods may be added against the SDK for calling specific Placement APIs if we think we may need those (instead of directly calling Placement with the standard HTTP verbs and the API URIs) ### Nova-specific topics ### # Antelope retrospective and process-related discussions Shorter cycle than Zed, 6 features landed (same as Zed), 46 contributors (down by 8 compared to Zed), 27 bugfixes (vs. 54 for Zed). We were happy to not have a RC2 for the first time in the Nova history. The bug triage backlog kept small. - We want to discuss on a better review cadence to not look at all the implementations by the last weeks. - We want to get rid of Storyboard stories in Placement. We found and agreed on a way forward. - We will set 5-years-old-and-more bug reports as Incomplete, so bug reporters have 90 days to confirm that the bug is still present on master. - We will add more contrib documentation for explaining more how to find Gerrit changes for one-off reviewers and how to create some changes - We agreed on no longer automatically accepting an already-approved spec from a previous cycle if no code is still not existing. - All efforts on porting our client calls to the SDK should be tagged in Launchpad with the 'sdk' tag. - A Bobcat schedule will be proposed on next Tuesday weekly meeting with feature and spec deadlines + some review days. - We also discussed what we should discuss in the Vancouver PTG. # CI failures continuous improvement - We want to test some alternative image but cirros (alpine as a potential alternative) with a small footprint in some specific nova jobs - We need to continue investigating on the volume detach/attach failures. We may create a specific job that would do serialized volume checks as a canary job for forcing the job to fail more frequently in order to dig down further. - (related) Sylvain will propose a Vancouver PTG cross-project session for a CI debugging war room experience. # The new release cadence We had long arguments whether we should hold deprecations and removals this cycle, given the releasenotes tooling isn't in place yet. As a reminder, since Bobcat is a non-SLURP release, operators are able to skip it and just look at the C releasenotes so we want to make sure we forward-port upgrade notes for them. For the moment, no consensus in place, we defer the outcome into a subsequent nova meeting and in the meantime, we raised the point to the TC for guidance needs. # The problem with hard affinity group policies - We agreed on the fact that hard affinity/anti-affinity policies aren't ideal (operators, if you read me, please prefer soft affinity/anti-affinity policies for various reasons I won't explain here) but since we support those policies and the use case is quite understandable, we need to find a way to unblock instances that can't be easily-migrated from those kind of groups (eg. try to move your instances off a host with hard Affinity between them, good luck) - A consensus has been reached to propose a new parameter for live-migrate, evacuate and migrate that would only skip the hard affinity checks. A backlog spec will be written capturing this consensus. - Since a group policy may be violated, we'll also propose a new server group API modification that would show if a group has its policy violated. # Unified limits next steps - We agreed on the fact we should move on and enable by default unified limits in a close future, but we first need to provide a bit of tooling to help the migration. - tool could be a nova-manage command that would provide the existing quota limits as a yaml file so we could inject those into keystone (to be further defined during Bobcat) - a nova-status upgrade check may be nice for yelling if operators forgot to define their limits in keystone before upgrading to the release (potentially C) that would default to unified limits - testing in the grenade job is a feat. # Frequently written objects may lead to exhaustion of DB primary keys - We agreed on having all new primary keys to have BigInteger as data type. Existing keys should use BigInt but there is a huge upgrade impact to mitigate. - Operators will be sollicited to know their preferences between a costly-at-once data migration vs. a rolling data online migration for existing tables. - We will document which DB tables are safe to reindex (since their PKs isn't normalized as a FK somewhere else) - A backlog spec will capture all of the above # Leftover volumes relationship when you directly call Cinder instead of Nova for detaching - Yup, this is a known issue and we should discuss this with the Cinder team (unfortunately, we didn't did it this week) - We maybe should have a nova-manage script that would cleanup the BDMs but we're afraid of leaking some volume residues in the compute OS - Anyway, we need to understand the main scope and we also need to discuss about the preconditions ('when' should we call this tool and 'what' this tool should be doing ?). Probably a spec. # Misc but not the leasc (heh) - we will fill a rfe bug for lazy-loading instance name from system metadata or update the nested field in the instance object (no additional DB write) - we will add a new policy for cold-migrate specifically when a host is passed as a parameter (admin-only as a default) - We agreed on continuing the effort on compute node hostname robustification (the Bobcat spec seems not controversial) - We agreed on a few things about the server show command and the use of the SDK. Some existing OSC patch may require a bit of rework but it would cover most of the concerns we discussed. - we should fix the post-copy issue when you want to live-migrate a paused instance by removing the post-copy migration flag if so. - we could enable virtio-blk trim support by default. It's both a bugfix (we want to remove an exception) and a new feature (we want to enable it by default) so we'll discuss whether we need a specless blueprint by a next meeting. - we also discuss about a Generic VDPA support feature for Nova, and we asked the owner to provide us a spec for explaining the usecase. ### That's it. ### Yet again, I'm impressed. If you read that phrase, you're done with reading. I just hope your coffee was good and the time you took reading that email was worth it. If you have questions or want to reach me directly on IRC, that's simple : I'm bauzas on #openstack-nova. HTH, -Sylvain (on behalf of the whole Nova community) [1] https://etherpad.opendev.org/p/nova-bobcat-pt g (please remove the empty char between 'pt' and 'g') [2] Unfortunately, I'm very good at creating bugs as I discovered that based on my 10-year unfortunate contributions to OpenStack. [3] Or a tea, or whatever you prefer. [4] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033115.h…

1 1

[kolla-ansible][zed][15.1.0] Horizon failed to load trunk and QoS
by Stefan Bryzga 03 Apr '23

03 Apr '23

Hi all, Some time ago I deployed a small cluster with kolla-ansible v15.1.0. Before deployment I enabled neutron_qos and neutron_trunk flags in /etc/kolla/globals.yml. Both functions work fine via cli and I can create new entries via dashboard however horizon won't display created trunks/qos on the page (It shows "No items to display"). In web browser console I get this error while loading page: TypeError: Cannot read properties of undefined (reading 'data') at addTrackBy (output.0a9483334bbd.js:1115:59) at processQueue (output.43a09e34a303.js:1581:786) at output.43a09e34a303.js:1585:110 at Scope.$digest (output.43a09e34a303.js:1636:200) at Scope.$apply (output.43a09e34a303.js:1643:269) at done (output.43a09e34a303.js:1356:123) at completeRequest (output.43a09e34a303.js:1375:20) at XMLHttpRequest.requestLoaded (output.43a09e34a303.js:1368:1) 'Possibly unhandled rejection: {}' Does anyone knows how to fix this? Should I modify any configs to display new entries in horizon? Best Regards, Stefan

1 0

Opensearch service upgrade
by Daniel Pawlik 03 Apr '23

03 Apr '23

Hello, We would like to notify you that the Opensearch service [1] would be updated to a newer version on 03.04.2023 at 12:00 PM UTC. This procedure might take a while, depending of the cluster size. During that time, the Opensearch service would not be available. If anyone has any doubts, please reply to the email. Dan [1] - https://opensearch.logs.openstack.org/_dashboards/app/home

1 1

Propose Dale Smith as Magnum Caore
by Jake Yip 03 Apr '23

03 Apr '23

Hi all, I would like to propose Dale Smith as a Magnum Core Reviewer. Dale is a contributor to Magnum, and also takes care of Magnum at his place of employment - Catalyst Cloud NZ. As many Magnum contributors are ourselves operators, I think it is important that we have representation from different cloud providers to help guide and shape Magnum going forward. He will replace Feilong, who is the former PTL and used to work in Catalyst Cloud, in the Core Team. We thank Feilong for his contributions and welcome Dale. Regards, Jake -- Jake Yip Technical Lead, ARDC Nectar Research Cloud

1 0

[nova][heat] The next steps to "fix" libvirt problems in Ubuntu Jammy
by Takashi Kajinami 03 Apr '23

03 Apr '23

Hello, Since we migrated our jobs from Ubuntu Focal to Ubuntu Jammy, heat gate jobs have become very flaky. Further investigation revealed that the issue is related to something in libvirt from Ubuntu Jammy and that prevents detaching devices from instances[1]. The same problem appears in different jobs[2] and we workaround the problem by disabling some affected jobs. In heat we also disabled some flaky tests but because of this we no longer run basic scenario tests which deploys instance/volume/network in a single stack, which means we lost the quite basic test coverage. My question is, is there anyone in the Nova team working on "fixing" this problem ? We might be able to implement some workaround (like checking status of the instances before attempting to delete it) but this should be fixed in libvirt side IMO, as this looks like a "regression" in Ubuntu Jammy. Probably we should report a bug against the libvirt package in Ubuntu but I'd like to hear some thoughts from the nova team because they are more directly affected by this problem. I'm now trying to set up a centos stream 9 job in Heat repo to see whether this can be reproduced if we use centos stream 9. I've been running that specific scenario test in centos stream 9 jobs in puppet repos but I've never seen this issue, so I suspect the issue is really specific to libvirt in Jammy. [1] https://bugs.launchpad.net/nova/+bug/1998274 [2] https://bugs.launchpad.net/nova/+bug/1998148 Thank you, Takashi

3 5

[horizon]display quota volume type on horizon
by Nguyễn Hữu Khôi 03 Apr '23

03 Apr '23

Hello. I see that blueprint is very real case but it has not deployed yet. https://blueprints.launchpad.net/horizon/+spec/quotas-per-cinder-volume-type I am curious why it is abanddoned .

2 2

[designate][ptg] Designate Bobcat (2023.2) PTG summary
by Michael Johnson 03 Apr '23

03 Apr '23

We accomplished some big tasks in Antelope: Finishing the shared zones feature, adding support for SQL Alchemy 2.x, and hard zone deletes. For the Bobcat series, Designate had a pretty straightforward PTG session with good discussion. The focus areas for Bobcat are going to be on the batch serial number update patch[1] (deferred from Antelope) and improving the multiple pools feature[2][3]. Multiple DNS server pools have been a feature in Designate for a number of years now, but we need to improve the test coverage and documentation. Other minor work anticipated for Bobcat is to update the RBAC testing to account for the RBAC "Direction Change"[4] in support of enabling the new Keystone default roles as the default RBAC configuration. We also discussed the new specification that merged during Antelope[5] to add support for catalog zones. This specification is merged and ready for development, but it was not clear that there are developers planning to work on it during Bobcat. We hope this feature can at least get started during the Bobcat series. Thank you to everyone that participated! Michael [1] https://review.opendev.org/c/openstack/designate/+/871255 [2] https://review.opendev.org/c/openstack/designate/+/857978 [3] https://review.opendev.org/c/openstack/designate-tempest-plugin/+/871184 [4] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… [5] https://specs.openstack.org/openstack/designate-specs/specs/antelope/catalo…

1 0

[ptls] PyPI maintainer cleanup - Action needed: Contact extra maintainers
by Jay Faulkner 03 Apr '23

03 Apr '23

Hi PTLs, The TC recently voted[1] to require humans be removed from PyPI access for OpenStack-managed projects. This helps ensure all releases are created via releases team tooling and makes it less likely for a user account compromise to impact OpenStack packages. Many projects have already updated https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L33 with a list of packages that contain extra maintainers. We'd like to request that PTLs, or their designate, reach out to any extra maintainers listed for projects you are responsible for and request they remove their access in accordance with policy. An example email, and detailed steps to follow have been provided at https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup-email-tem… . Thank you for your cooperation as we work to improve our security posture and harden against supply chain attacks. Thank you, Jay Faulkner TC Vice-Chair 1: https://opendev.org/openstack/governance/commit/979e339f899ef62d2a6871a99c9…

3 6

[barbican] database is growing and can not be purged
by Pavlo Shchelokovskyy 03 Apr '23

03 Apr '23

Hi all, we are observing the following behavior in Barbican: - OpenStack environment is using both encrypted Cinder volumes and encrypted local storage (lvm) for Nova instances - over the time, the secrets and orders tables are growing - many soft-deleted entries in secrets DB can not be purged by the db cleanup script As I understand what is happening - both Cinder and Nova create secrets in Barbican on behalf of the user when creating an encrypted volume or booting an instance with encrypted local storage. They both do it via castellan library, that under the hood creates orders in Barbican, waits for them to become active and returns to the caller only the ID of the generated secret. When time comes to delete the thing (volume or instance) Cinder/Nova again use castellan, but only delete the secret, not the order (they are not aware that there was any 'order' created anyway). As a result, the orders are left in place, and DB cleanup procedure does not delete soft-deleted secrets when there's an ACTIVE order referencing such secret. This is troublesomes on many levels - users who use Cinder or Nova may not even be aware that they are creating something in Barbican. Orders accumulating like that may eventually result in cryptic errors when e.g. when you run out of quota for orders. And what's more, default Barbican policies do allow 'normal' (creator) users to create an order, but not delete it (only project admin can do it), so even if the users are aware of Barbican involvement, they can not delete those orders manually anyway. Plus there's no good way in API to determine outright which orders are referencing deleted secrets. I see several ways of dealing with that and would like to ask for your opinion on what would be the best one: 1. Amend Barbican API to allow filtering orders by the secrets, when castellan deletes a secret - search for corresponding order and delete it as well, change default policy to actually allow order deletion by the same users who can create them. 2. Cascade-delete orders when deleting secrets - this is easy but probably violates that very policy that disallowed normal users to delete orders. 3. improve the database cleanup so it first marks any order that references a deleted secret also as deleted, so later when time comes both could be purged (or something like that). This also has a similar downside to the previous option by not being explicit enough. I've filed a bug for that https://storyboard.openstack.org/#!/story/2010625 and proposed a patch for option 2 (cascade delete), but would like to ask what would you see as the most appropriate way or may be there's something else that I've missed. Btw, the problem is probably even more pronounced with keypairs - when castellan is used to create those, under the hood both order and container are created besides the actual secrets, and again only the secret ids are returned to the caller. When time comes to delete things, the caller only knows about secret IDs, and can only delete them, leaving both container and order behind. Luckily, I did not find any place across OpenStack that actually creates keypairs using castellan... but the problem is definitely there. Best regards, -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

2 2

[SWIFT] ssync receiver crash
by Reid Guyett 03 Apr '23

03 Apr '23

Hello, We recently upgraded our swift servers to 20.04 from 18.04. After the upgrades were completed we are seeing 1 server where the ssync receiver keeps crashing. The crash is preventing hand offs from being cleanup up in the cluster (50 servers). Looking for some advice to fix/workaround the issue. We have 20 clusters but are only seeing this issue in 1 cluster. It is strange to me that we only see this ssync receiver error in one server out of the ~1000 we are running. OS: Ubuntu 20.04.5 Swift version: 2.25.2 On the receiver side we are seeing the following trace: object-server: senderIp/d1/2166 EXCEPTION in ssync.Receiver: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 209, in _chunked_read self.chunk_length = int(rfile.readline().split(b";",1)[0], 16) ValueError: invalid literal for int() with base 16: b'' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 166, in __call__ for data in self.missing_check(): File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 340, in missing_check line = self.fp.readline(self.app.network_chunk_size) File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 226, in readline return self._chunked_read(self.rfile, size, True) File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 211, in _chunked_read raise ChunkReadError(err) eventlet.wsgi.ChunkReadError: invalid literal for int() with base 16: b'' On the sender side we are seeing this message when I try to run the reconstructor manually with debug logging: root@use1-saas-p6-paco-9:~# swift-object-reconstructor object-server.conf -p 27999 -v -o object-reconstructor: Starting 271723 object-reconstructor: Spawned worker 271755 with {'override_partitions': [27999], 'override_devices': ['d10', 'd17', 'd19', 'd9', 'd29', 'd15', 'd2', 'd24', ' d32', 'd1', 'd8', 'd14', 'd11', 'd28', 'd27', 'd6', 'd7', 'd16', 'd13', 'd25', 'd26', 'd20', 'd18', 'd12', 'd4', 'd0', 'd22', 'd3', 'd5', 'd21', 'd33', 'd23', 'd30', 'd34', 'd31'], 'multiprocess_worker_index': 0} object-reconstructor: [worker 1/1 pid=271755] Running object reconstructor in script mode. object-reconstructor: [worker 1/1 pid=271755] Run listdir on /srv/node/d22/objects-4/27999 object-reconstructor: [worker 1/1 pid=271755] recieverIP:6200/d31/27999 10.0 seconds: connect receive object-reconstructor: [worker 1/1 pid=271755] 1/26820 (0.00%) partitions reconstructed in 10.14s (0.10/sec, 75h remaining) object-reconstructor: [worker 1/1 pid=271755] Object reconstruction complete (once). (0.17 minutes) object-reconstructor: Forked worker 271755 finished object-reconstructor: Worker 271755 exited object-reconstructor: Finished 271723 object-reconstructor: Exited 271723 Thanks! Reid Guyett Staff Storage Engineer Datto, A Kaseya Company www.datto.com<http://www.datto.com/> Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at legal(a)kaseya.com, and then delete it. To opt-out of receiving emails Please click here<https://info.kaseya.com/email-subscription-center.html>. The term 'this e-mail' includes any and all attachments.

1 0