October 2025 - openstack-discuss - lists.openstack.org

[nova][glance] Image corruption during shelving
by Vladimir Prokofev 31 Oct '25

31 Oct '25

Hello everyone. I've discovered a bug in nova/glanceclient interoperation that leads to image corruption during shelving under some circumstances involving connection disruption. I was able to reproduce it in two different production environments running on xena and 2024.2 respectively, I'm also able to reproduce it in devstack though without image corruption - there it just ends up in a shelve failure due to detected corruption. My setup assumes LVM-backed QEMU VMs with backend in CEPH for Glance, but I believe this is applicable to a variety of nova/glance backends. When shelve is triggered, nova-compute creates an image file locally, and then initiates upload of said image file into Glance[0]. If something happens to the connection during the upload("broken pipe", "connection timeout") - nova-compute retries upload operation[1] while image object is removed from CEPH backend by Glance[2] Problem here is that the image_file object that is passed to glanceclient by nova-compute is a byte-stream created with an open()[3] call, and upon retry it resumes the upload from the point where it was interrupted. This is easily confirmed by calling image_data.tell() in glanceclient.v2.images.Controller.upload() function - it will be at zero initially and non-zero on retry. I haven't thoroughly checked protection offered in Glance master, I believe it may have been significantly improved, but in older releases such as 2024.2, this leads to a corrupted shelved image, because upon retry only part of an image object is uploaded, after which original VM is offloaded(removed) and you end up with lost data. Now, my issue here is where to submit this bug: glanceclient or nova? This problem is easily fixable in glanceclient by calling image_data.seek(0) in glanceclient.v2.images.Controller.upload(): it makes sense to always point byte-stream to the beginning before initiating upload, but should it really be responsibility of a client to perform such a sanity check? I'm also not sure if there're cases that this will break, for example if a non-seekable object is passed into glanceclient, but I'm not sure if this is even possible? [0] https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driv… [1] https://opendev.org/openstack/nova/src/branch/master/nova/image/glance.py#L… [2] https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_… [3] https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driv…

2 2

[ptg] Image Encryption Session, Thursday 30th @ 16:00 UTC
by Markus Hentsch 30 Oct '25

30 Oct '25

Dear OpenStack community, in preparation for the vPTG discussion slot for the image encryption (Thursday 30th at 16:00 UTC) I collected a summary explaining the key points and state of the contribution and gathered some discussion topics below. Since this spans multiple projects (mainly Glance, Cinder and Nova) and has been ongoing for quite some years, I tried my best to briefly summarize the most important aspects of the current state so that all those who wish to participate have the latest information. The text below can also be found on the etherpad: https://etherpad.opendev.org/p/2026.1-ptg-glance-planning#L128 Summary: - historically, only Cinder has been able to create and use encrypted images (as a side effect) by dumping LUKS-encrypted volumes 1:1 into images, for exclusive use in Cinder - this contribution aims to introduce encrypted image functionality for the end user: images can be encrypted/decrypted outside of OpenStack and resulting (encrypted) image is stored in Glance and the key in Barbican - goal: standardize encrypted image formats and handling across Glance, Cinder and Nova - it introduces two formats: raw LUKS and qcow2-wrapped LUKS (native qcow2 feature) - it introduces new key handling: if Barbican secret is of type 'passphrase' it is used as the LUKS key as-is; if it is of any other secret type, it is assumed to be binary and converted via 'binascii.hexlify()' (preserves Cinder behavior and compatibility) - it makes use of the new secret consumer API in Barbican to prevent accidental key deletion (since keys may be managed by users, not OpenStack), registering images as consumers to their keys - note: the image encryption feature focuses on Glance and Cinder for now, Nova has been largely excluded so far due to ephemeral storage encryption not being ready yet Implementation State (note: aside from os-brick this is not merged yet!): - os-brick: implemented outsourced utility function to discern encryption between passphrase and binary (hexlify conversion) for shared usage by Nova and Cinder - Glance: implemented new encryption-specific image formats and properties as well as secret consumer handling - Cinder: implemented encrypted image usage for volumes: LUKS is handled as raw (like before) and qcow2+LUKS is converted; image creation from volume stays limited to raw LUKS - Nova: implemented only compatibility with os-brick and Cinder changes so far - barbican-tempest-plugin: full scenario tests (image > volume > server > SSH) for 4 permutations of the image encryption Patchsets: https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 Discussion Topics: 1) Image inspection: related to CVE-2024-32498 concerns were brought up about encrypted image payload containing malicious formats triggering unintended QEMU behavior; how to screen encrypted image payload and reject undesired formats? 1a) limiting to specific formats has consequences for Cinder: attached (secondary) encrypted volumes might contain arbitrary binary data in VMs (user choice), when uploading as image to Glance it might be rejected as prohibited format; additionally, such image vanishes without trace because there is no error state for images (bad Glance <-> Cinder interaction) - how to address this? 1b) how deep do we want to look into encrypted image payload? decrypting a 1TB image might be very costly (CPU load, disk space) - problem: "qemu-img convert" cannot stream to stdout (see https://blogs.igalia.com/berto/2025/07/15/converting-qemu-qcow2-images-dire… ); in the worst case we have to wait for the full 1TB to be decrypted, consuming almost twice the amount of disk space in the process, instead of being able to only read the payload header and abort 2) Nova integration: how to approach future interoperability with ephemeral storage encryption? 2a) can we handle this as separate future expansion to this feature, i.e., merging Glance+Cinder functionality short-term with only compatibility changes in Nova for now and addressing full Nova integration as a next step with dedicated patchsets based on the current work? 3) Better organization: would there be any interest in reviving the image encryption popup-team meeting? https://meetings.opendev.org/#Image_Encryption_Popup-Team_Meeting Best regards, Markus Hentsch -- Markus Hentsch DevOps Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 | 01099 Dresden +49 351 479 367 00 markus.hentsch(a)cloudandheat.com | www.cloudandheat.com Green, Open, Efficient. Ihr Cloud-Service- und Cloud-Technologie-Provider aus Dresden. https://www.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann

2 3

[skyline] possible small bug in skyline
by Taavi Ansper 30 Oct '25

30 Oct '25

Hi I have found a possible small bug in skyline. I didn't know where to post this in bugzilla. https://bugs.launchpad.net/skyline-apiserver/+bug/2129915 The bug comes from the placement API request to: //resource_providers/1ab32e69-ee87-4ccb-bd8a-b39abfa4a980/inventories The same UI view in horizon works nicely and it doesn't append the the double slash in the API request. Where might this come from the code? Something related to this is defined here, but it probably comes from the "Base" stuff: https://github.com/openstack/skyline-console/blob/stable/2025.1/src/client/… https://github.com/openstack/skyline-console/blob/stable/2025.1/src/client/… This issue appeared after upgrading from 2024.2 to 2025.1 Best regards -- ---- Taavi Ansper taavi.ansper(a)cyber.ee

1 0

[openstack][kolla-ansible]about group node config overwrite
by Nguyễn Hữu Khôi 29 Oct '25

29 Oct '25

Hello Koller. I want to ask if we can overwrite config for a group of nodes. For example, compute01 >> compute10 will have the same nova configuration compute11 >> compute20 will have the same nova configuration. It will be very nice if we support this way. Thank you. Regards Nguyen Huu Khoi

4 7

Opensearch Service Update
by Daniel Pawlik 29 Oct '25

29 Oct '25

Hello, We would like to notify you that the Opensearch service [1] would be updated to a newer version on 13 November 2025 at 12:00 PM UTC. This procedure might take a while, depending on the cluster size. During that time, the Opensearch service would not be available. If anyone has any doubts, please reply to the email. Dan [1] - https://opensearch.logs.openstack.org/_dashboards/app/home

2 1

[nova][glance] Server rescue falls back to default image
by Francesco Di Nucci 29 Oct '25

29 Oct '25

Hi all, I am on OpenStack Caracal, trying to rescue a VM that's not booting: * I previously created a SystemRescueCD rescue image (with openstack image create --file ./systemrescue-12.02-amd64.iso --disk-format raw --container-format bare --property hw_rescue_bus=virtio --property hw_rescue_device=disk systemrescuecd_1202) - it works on VMs that still have their base images in Glance * tried to rescue the VM (with openstack server rescue --image systemrescuecd_1202 my-vm-id-here) * but the process fails as it tries to also use the base image used to create that VM, instead of the rescue image only (Instance my-vm-id-here cannot be rescued: Driver Error: Image b78cf456-b395-434f-bf50-a1f2f3684a37 could not be found. ; where b78cf456-b395-434f-bf50-a1f2f3684a37 was a Ubuntu cloud image used to create that VM, since deleted as it's an EOL version) Is it supposed to also try to use the base image or should the --image bypass it? In case, what could I do? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email:francesco.dinucci@na.infn.it

3 2

[kolla-ansible] Rabbitmq queue creation issues in "fresh" 2025.1 deploy
by Jonathan Proulx 28 Oct '25

28 Oct '25

Hi All, I'm putting "fresh" in quotes becuase I expect the issue is some home my deploy isn't as fresh as I belive it to be. I did have and older test environment which was serveral steps behind (and not quite workign for other reasons). The presenting issues is this error (and other similar ones for other services): 2025-10-24 11:44:07.840 1076 WARNING oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Queue conductor could not be declared probably because of conflicting configurations: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false' 2025-10-24 11:44:07.841 1076 INFO oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Retrying to declare the exchange (nova) as non durable I'm puzzled since the services are deploying to recently reinstalled nodes (I'm sure that part is fresh) how the queues can be created with flags the services don't expect. my shell history tells me I'm using these versions (in a venev): pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2025.1 pip3 install --upgrade git+https://opendev.org/openstack/kolla@stable/2025.1 pip3 install --upgrade 'ansible-core>=2.17,<2.18.99' The only bits I know are left over and "upgraded" from previous attempts are my configs in /etc/kolla. My deploy host wasn't reinstalled but I did delete and recreate the venv. Using Ubuntu 24.04 as the base OS Where do I go from here to find my wrong turn? Thanks, -Jon -- Jonathan "Some Day I'll Upgrade From Mitaka" Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL

2 4

Re: [tc][all][security] Supporting Post-Quantum Cryptography in OpenStack code (all projects)
by Jean-Philippe Jung 28 Oct '25

28 Oct '25

Hi, Thanks for the comments, and let me try to group some & reply ahead of the PTG session. 1- Quantum computers are no threat, they are underpowered, so what’s the risk & why should we do anything now? * The issue is time. The day a quantum computer is powerful enough to break traditional cryptography (known as Q-day), it is too late. OpenStack lays naked and vulnerable. * We can argue about dates, and my current opinion is that to be on the safe side, any software product should implement PQC cryptography by 2029. * We have a large code base, and we need to give time to the community to review & adjust it then to openstack operators to implement quantum-safe cryptography. * There is no snake oil/panic-driven agenda here, either we protect access to OpenStack infrastructure & between services, preferably by 2028. Or we don’t. We have time, but we should start work now. 2- It’s a problem for Operating System vendors, we just consume their cryptography. * I wish it were true. But the initial analysis I made demonstrates we have libraries that do cryptographic operations, and maybe these libraries end up properly calling an OS cryptographic module. Maybe not. I know of paramiko, that does not call any OS layer, but its own copy of OpenSSL code. We cannot assume OpenStack has nothing to do until we analyse our cryptographic usage. * I totally support the idea of using OS cryptography as much as possible, because we offload the maintenance problem to OS vendors. * Then because new fancy cryptographic algorithms have been finalized (key encapsulation, ML-KEM), or near finalized (digital signature, ML-DSA) does not indeed mean they are available in all the protocols that uses them (IPsec, kerberos, dnssec…) in operating systems. 3- Some of the libraries listed might no longer be in used. * The code should be removed from OpenStack. We have customers doing Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST): these will flag those unsafe modules. * Other side of that point: some of these libraries are still used. Do they rely on an operating system cryptography, or do they pull their own cryptographic call? Are they maintained? Should we reduce the amount of cryptographic modules in OpenStack? 4- There is no plan proposed, what’s next? * Initial email was to express my concern about supporting PQC in OpenStack, understanding if my concern was shared, and raise awareness on this topic right ahead of the PTG. * The first step to solve a problem is to know there is a problem, I’d start with a per project effort starting with a simple question for projects: "Does your project do anything in relation to keys, encryption, or interaction of any encrypted data either at rest or while in transit?" * Next would be to use those results to plan for protecting OpenStack against the store-now, decrypt-later attacks, meaning protecting key exchanges relaying on ML-KEM (well hybrid X25519MLKEM768 to start with), ensuring symmetric keys are long enough (256 bits minimum), then deal later with digital signatures code (if any). 5- The goal is premature * I consider this a community goal because all OpenStack components must achieve support pretty much in the same time frame. I am not saying it should take precedence over any other critical work (e.g., eventlet removal), I think now is the time to scope the work and plan to meet the 2027/2028 deadline. * OpenStack customers planning to run it well into the 2030s are asking about the plans for PQC support. And new customers considering OpenStack are asking about PQC support plans. Either we have an answer, a roadmap, or OpenStack will not be part of the discussion, at some point (ask me how I know :-). Regards, JP

1 0

pip 25.3 removes support for legacy editable installs
by Stephen Finucane 28 Oct '25

28 Oct '25

o/ pip 25.3 was released on Friday (who does Friday releases?!) and had removed support for the legacy `setup.py develop` editable install method in setuptools [1]. Now, thankfully most services seem to have listened to me [2][3] and the shouty warnings from setuptools/pip and migrated away from 'wsgi_scripts'. However, there is a good chance that older stable branches are missing these migration patches and consequently now broken. For example, it seems keystone is missing these changes on stable/2024.2, and attempts to deploy that branch currently fail with: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Internal Server Error (HTTP 500) which, upon further inspection, will turn out to be a missing wsgi script: devstack(a)keystone.service[40466]: failed to open python file /opt/stack/data/venv/bin/keystone-wsgi-public We have already seen this pop up on the cluster-api-provider-openstack (CAPO) project [4], which is using the old stable/2024.2 (Dalmatian) release. I have proposed fixes to stable/2025.1 [5] and stable/2024.2 [6] to constrain pip on those older branches. This should fix the issue for everyone, provided you're not using a devstack plugin that re-updates pip. Individual services may also wish to backport the patches that added '$service.wsgi' modules if possible, to fix this for non-devstack use cases, but this won't be possible in all cases. Hope this helps, Stephen [1] https://pip.pypa.io/en/stable/news/#v25-3 [2] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/28… [5] https://review.opendev.org/c/openstack/devstack/+/965092 [6] https://review.opendev.org/c/openstack/devstack/+/965093

2 1

[keystone][ptg] Keystone PTG schedule is available (Tue and Wed)
by Artem Goncharov 28 Oct '25

28 Oct '25

Hey guys, The Keystone team is meeting today and tomorrow for the PTG. Today's focus are the federation topics, passkeys, service accounts and discussion of feature deprecation that depend on the software abandoned upstream (i.e. saml2). Tomorrow we are going to focus more on the cryptography used in Keystone, Horizon cross-session and certain cli aspects. You are very welcome to join us discussing the future of Keystone. TECHs of today put big challenges in front of us and we need to do rework no matter how hard and painful it is. https://etherpad.opendev.org/p/oct2025-ptg-keystone Regards and see you in already a bit Artem

1 0