- openstack-discuss - lists.openstack.org

[neutron] Bug Deputy Report Oct 27 - Nov 02
by Yatin Karel 04 Nov '25

04 Nov '25

Hi Neutrinos, Below is the bug deputy report for last week:- Critical ======== - https://bugs.launchpad.net/neutron/+bug/2130380 - OOM called from keepalived, "neutron-tempest-plugin-openvswitch" job Unassigned - https://bugs.launchpad.net/neutron/+bug/2130416 - [UT] CI job ``openstack-tox-py310`` and ``openstack-tox-py311`` frequently timing out Assigned to Rodolfo Alonso High ==== - https://bugs.launchpad.net/neutron/+bug/2129926 - DNS integration: port deletion fails when reverse DNS zone doesn't exist in Designate Assigned to Helen Chen - https://bugs.launchpad.net/neutron/+bug/2130405 - Neutron and Designate is not properly cleaned when a VM with a FIP is deleted Assigned to Helen Chen - https://bugs.launchpad.net/neutron/+bug/2130541 - [FT] test failure with no child process missing uuid Assigned to Rodolfo Alonso Medium ====== - https://bugs.launchpad.net/neutron/+bug/2129959 - Semantic violated with bulk creation of security groups Unassigned - https://bugs.launchpad.net/neutron/+bug/2130340 - [CI] functional test test__cleanup_previous_tags failure since Oct. 28, 2025 Assigned to Rodolfo Alonso - https://bugs.launchpad.net/neutron/+bug/2130564 - [FT] Error in ``test_initialize_nsrange_support_parallel_exec_diff_init_time`` Assigned to Rodolfo Alonso Wishlist ======== - https://bugs.launchpad.net/neutron/+bug/2130453 - [RFE] Support provider routed networks with multiple segments per host in the ML2/OVN driver Assigned to Miguel Lavalle - https://bugs.launchpad.net/neutron/+bug/2130570 - [RFE] Make "ovn-db-sync" to be pluggable Assigned to Slawek Kaplonski - https://bugs.launchpad.net/neutron/+bug/2130571 - [RFE] Make ovn maintenance tasks to be pluggable Assigned to Slawek Kaplonski New === - https://bugs.launchpad.net/neutron/+bug/2130211 ovn octavia provider related, Checking with froyo on this -- Thanks and Regards Yatin Karel

1 0

[tc][all] OpenStack Technical Committee Weekly Meeting Agenda (2026.1/R-21)
by Goutham Pacha Ravi 04 Nov '25

04 Nov '25

Hello Stackers, After the action-filled OpenInfra PTG week, The OpenStack Technical Committee will resume its regular meeting schedule this week. The next weekly meeting is scheduled to take place today, November 4, 2025, at 1700 UTC. Please find the meeting agenda and details for joining the meeting on the meeting wiki [1]. On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] TC Meeting Agenda, 2025-11-04: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[kolla-ansible][keystone] Could not recognize Fernet token
by Satish Patel 04 Nov '25

04 Nov '25

I am running 3 node controllers in multi-node deployment using the 2023.1 release and life was good until one of my controller nodes died. after losing one of the controller node I have started seeing following error in keystone logs 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application [None req-95a83b6e-dacc-45d4-8d1f-4d5c979f79e6 d58d8e432da74d8ba4416054d1e36a84 d1ef63c18c6648faa56643a942216311 - - default default] Could not recognize Fernet token: keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 89, in unpack 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.crypto.decrypt(token.encode('utf-8')) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/cryptography/fernet.py", line 210, in decrypt 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application raise InvalidToken 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application cryptography.fernet.InvalidToken 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 99, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.token_formatter.validate_token(token_id) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 173, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application serialized_payload = self.unpack(token) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 91, in unpack 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application raise exception.ValidationError( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application keystone.exception.ValidationError: Could not recognize Fernet token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application rv = self.dispatch_request() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application resp = resource(*args, **kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 107, in view 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application resp = meth(*args, **kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 145, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application token = self._validate_token(token_id) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.get_or_create( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application with Lock( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self._enter() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.creator() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application created_value = creator( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 158, in _validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application issued_at, expires_at) = self.driver.validate_token(token_id) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 101, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application raise exception.TokenNotFound(e) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application I have rotated the fernet key and also verified in docker container and md5sum is match and date/time also very accurate. I am not sure what else I should look for. (keystone)[root@os1-ctrl01 fernet-keys]# md5sum /etc/keystone/fernet-keys/* 4874cf812b73f18cde82f1d00ea681e1 /etc/keystone/fernet-keys/0 e1a40f4cd58cfbc9b3b5af7305a16fc8 /etc/keystone/fernet-keys/214 0f45eeb318e3c4bfb97bb07da790329a /etc/keystone/fernet-keys/215 (keystone)[root@os1-ctrl03 /]# md5sum /etc/keystone/fernet-keys/* 4874cf812b73f18cde82f1d00ea681e1 /etc/keystone/fernet-keys/0 e1a40f4cd58cfbc9b3b5af7305a16fc8 /etc/keystone/fernet-keys/214 0f45eeb318e3c4bfb97bb07da790329a /etc/keystone/fernet-keys/215

1 0

[glance] Gazpacho PTG summary
by Cyril Roelandt 03 Nov '25

03 Nov '25

Hi, We have just had our virtual PTG. The full list of topics, along with notes, can be found on our Etherpad[1]. Here is a summary of the topics that will be targeted during this cycle. If you would like to view one of the recordings, please reach out to us so we can figure out a way to share it. # Configurable pr limits for qemu-img info We have agreed to make the pr limits for qemu-img info configurable, as is the case in Cinder. # Eventlet removal We will keep working towards moving our functional tests away from eventlet. # Download image from specific store We discussed the implementation details of this new feature, that will allow users to specify a specific store from which to download their images. # Eliminate redundant downloads We tried implementing this in the past and realized there were more challenges than we initially realized. We discussed the technical difficulties we faced during last cycle. # Improve image_size_total behavior in glance unified quota There are multiple scenarios that can lead to a user going slightly over quota in terms of the total size of images stored. We discussed ideas on how to mitigate the issue. # Replace glanceclient with OpenStack SDK We intend to have glanceclient go into maintenance mode in the I cycle. Our main concern is that some projects still use glanceclient as a library. We will be making sure they use the OpenStack SDK instead. # Add RBD Erasure-Coded Pools Support Using erasure-coded pools in Ceph used to cause a CPU overhead that was not acceptable for many operators. The situation has improved and the operators may now want to use this feature to save space. # Filesystem store improvement for NFS We will be improving the filesystem store so that operators face fewer issues when an NFS share is disconnected. # S3 Store Boto3 Checksum Behavior Configuration Some users are facing issues with S3-compatible storage if they are running the latest version of boto3. We discussed exactly how to work around the issue. # Image Encryption and disk format enforcement Image encryption has an impact on multiple OpenStack projects: Glance, Cinder and Nova, at least. Happy Hacking! Cyril Roelandt [1] https://etherpad.opendev.org/p/oct2025-ptg-glance

1 0

[neutron] Team and Drivers meeting canceled this week
by Brian Haley 03 Nov '25

03 Nov '25

Hello Neutrinos: As mentioned at the PTG, I am going to cancel the Neutron Team and Drivers meetings this week as I am out of the office. Have a nice week! -Brian

1 0

[tacker] Cancelling weekly IRC meeting
by Yasufumi Ogawa 03 Nov '25

03 Nov '25

Hi team, Due to a holiday in Japan and most of team members aren't join, I'm cancelling today's meeting. Regards, Yasufumi

2 1

[ironic] Stop maintaining sushy as a separate project, let's move it into ironic
by Iury Gregory 01 Nov '25

01 Nov '25

Hello everyone! During the PTG yesterday, we discussed stop maintaining sushy as a separate project. For those who don't know what sushy is: a Python library to communicate with Redfish based systems and is limited to what Ironic supports. The library was never intended for others to consume, yet that has happened, and it being a library has been a noted hurdle to quickly resolving user issues, because it requires releasing a new version. But before we move this forward, we would like to hear from you what are your thoughts. Thanks! -- *Att[]'s* *Iury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Ironic Core - Ironic Event Liaison (DPL)* *Senior Software Engineer at Red Hat Brazil* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

6 6

[nova][glance] Image corruption during shelving
by Vladimir Prokofev 31 Oct '25

31 Oct '25

Hello everyone. I've discovered a bug in nova/glanceclient interoperation that leads to image corruption during shelving under some circumstances involving connection disruption. I was able to reproduce it in two different production environments running on xena and 2024.2 respectively, I'm also able to reproduce it in devstack though without image corruption - there it just ends up in a shelve failure due to detected corruption. My setup assumes LVM-backed QEMU VMs with backend in CEPH for Glance, but I believe this is applicable to a variety of nova/glance backends. When shelve is triggered, nova-compute creates an image file locally, and then initiates upload of said image file into Glance[0]. If something happens to the connection during the upload("broken pipe", "connection timeout") - nova-compute retries upload operation[1] while image object is removed from CEPH backend by Glance[2] Problem here is that the image_file object that is passed to glanceclient by nova-compute is a byte-stream created with an open()[3] call, and upon retry it resumes the upload from the point where it was interrupted. This is easily confirmed by calling image_data.tell() in glanceclient.v2.images.Controller.upload() function - it will be at zero initially and non-zero on retry. I haven't thoroughly checked protection offered in Glance master, I believe it may have been significantly improved, but in older releases such as 2024.2, this leads to a corrupted shelved image, because upon retry only part of an image object is uploaded, after which original VM is offloaded(removed) and you end up with lost data. Now, my issue here is where to submit this bug: glanceclient or nova? This problem is easily fixable in glanceclient by calling image_data.seek(0) in glanceclient.v2.images.Controller.upload(): it makes sense to always point byte-stream to the beginning before initiating upload, but should it really be responsibility of a client to perform such a sanity check? I'm also not sure if there're cases that this will break, for example if a non-seekable object is passed into glanceclient, but I'm not sure if this is even possible? [0] https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driv… [1] https://opendev.org/openstack/nova/src/branch/master/nova/image/glance.py#L… [2] https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_… [3] https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driv…

2 2

[ptg] Image Encryption Session, Thursday 30th @ 16:00 UTC
by Markus Hentsch 30 Oct '25

30 Oct '25

Dear OpenStack community, in preparation for the vPTG discussion slot for the image encryption (Thursday 30th at 16:00 UTC) I collected a summary explaining the key points and state of the contribution and gathered some discussion topics below. Since this spans multiple projects (mainly Glance, Cinder and Nova) and has been ongoing for quite some years, I tried my best to briefly summarize the most important aspects of the current state so that all those who wish to participate have the latest information. The text below can also be found on the etherpad: https://etherpad.opendev.org/p/2026.1-ptg-glance-planning#L128 Summary: - historically, only Cinder has been able to create and use encrypted images (as a side effect) by dumping LUKS-encrypted volumes 1:1 into images, for exclusive use in Cinder - this contribution aims to introduce encrypted image functionality for the end user: images can be encrypted/decrypted outside of OpenStack and resulting (encrypted) image is stored in Glance and the key in Barbican - goal: standardize encrypted image formats and handling across Glance, Cinder and Nova - it introduces two formats: raw LUKS and qcow2-wrapped LUKS (native qcow2 feature) - it introduces new key handling: if Barbican secret is of type 'passphrase' it is used as the LUKS key as-is; if it is of any other secret type, it is assumed to be binary and converted via 'binascii.hexlify()' (preserves Cinder behavior and compatibility) - it makes use of the new secret consumer API in Barbican to prevent accidental key deletion (since keys may be managed by users, not OpenStack), registering images as consumers to their keys - note: the image encryption feature focuses on Glance and Cinder for now, Nova has been largely excluded so far due to ephemeral storage encryption not being ready yet Implementation State (note: aside from os-brick this is not merged yet!): - os-brick: implemented outsourced utility function to discern encryption between passphrase and binary (hexlify conversion) for shared usage by Nova and Cinder - Glance: implemented new encryption-specific image formats and properties as well as secret consumer handling - Cinder: implemented encrypted image usage for volumes: LUKS is handled as raw (like before) and qcow2+LUKS is converted; image creation from volume stays limited to raw LUKS - Nova: implemented only compatibility with os-brick and Cinder changes so far - barbican-tempest-plugin: full scenario tests (image > volume > server > SSH) for 4 permutations of the image encryption Patchsets: https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 Discussion Topics: 1) Image inspection: related to CVE-2024-32498 concerns were brought up about encrypted image payload containing malicious formats triggering unintended QEMU behavior; how to screen encrypted image payload and reject undesired formats? 1a) limiting to specific formats has consequences for Cinder: attached (secondary) encrypted volumes might contain arbitrary binary data in VMs (user choice), when uploading as image to Glance it might be rejected as prohibited format; additionally, such image vanishes without trace because there is no error state for images (bad Glance <-> Cinder interaction) - how to address this? 1b) how deep do we want to look into encrypted image payload? decrypting a 1TB image might be very costly (CPU load, disk space) - problem: "qemu-img convert" cannot stream to stdout (see https://blogs.igalia.com/berto/2025/07/15/converting-qemu-qcow2-images-dire… ); in the worst case we have to wait for the full 1TB to be decrypted, consuming almost twice the amount of disk space in the process, instead of being able to only read the payload header and abort 2) Nova integration: how to approach future interoperability with ephemeral storage encryption? 2a) can we handle this as separate future expansion to this feature, i.e., merging Glance+Cinder functionality short-term with only compatibility changes in Nova for now and addressing full Nova integration as a next step with dedicated patchsets based on the current work? 3) Better organization: would there be any interest in reviving the image encryption popup-team meeting? https://meetings.opendev.org/#Image_Encryption_Popup-Team_Meeting Best regards, Markus Hentsch -- Markus Hentsch DevOps Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 | 01099 Dresden +49 351 479 367 00 markus.hentsch(a)cloudandheat.com | www.cloudandheat.com Green, Open, Efficient. Ihr Cloud-Service- und Cloud-Technologie-Provider aus Dresden. https://www.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann

2 3

[skyline] possible small bug in skyline
by Taavi Ansper 30 Oct '25

30 Oct '25

Hi I have found a possible small bug in skyline. I didn't know where to post this in bugzilla. https://bugs.launchpad.net/skyline-apiserver/+bug/2129915 The bug comes from the placement API request to: //resource_providers/1ab32e69-ee87-4ccb-bd8a-b39abfa4a980/inventories The same UI view in horizon works nicely and it doesn't append the the double slash in the API request. Where might this come from the code? Something related to this is defined here, but it probably comes from the "Base" stuff: https://github.com/openstack/skyline-console/blob/stable/2025.1/src/client/… https://github.com/openstack/skyline-console/blob/stable/2025.1/src/client/… This issue appeared after upgrading from 2024.2 to 2025.1 Best regards -- ---- Taavi Ansper taavi.ansper(a)cyber.ee

1 0