October 2025 - openstack-discuss - lists.openstack.org

[ops][largescale-sig][scientific-sig] Ops Radio Hour on October 31, 2025 at the PTG
by Ildiko Vancsa 04 Nov '25

04 Nov '25

Hi OpenStack community, I'm reaching out to you with a friendly reminder that the next OpenStack Ops Radio Hour[1] is scheduled to happen on __October 31st, 2025 at 1300UTC,__ which is during the PTG! The call series is a casual setting for operators, engineers, and contributors to come together, share knowledge, and collaborate on the complexities of managing and deploying OpenStack environments. The main topic of the next gathering will be to discuss how to refactor Ops-related activities. See Thierry's email for more details: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… If you have further topics you would like to dive into during the call, please add them to this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour A few reminders: - The group on the call agreed to try out Matrix as a chat platform for more synchronous communication outside calls + You can access the Matrix room here: https://matrix.to/#/#openstack-ops:opendev.org<https://matrix.to/#/%23openstack-ops:opendev.org> + Further details are here: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - If you would like to read and catch up on notes from previous calls, you can find them in this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour See you on the call on October 31st! Bets Regards, Ildikó [1] https://wiki.openstack.org/wiki/Operations/Meetups

1 1

[ironic][all] Making core teams more accessible
by Jay Faulkner 04 Nov '25

04 Nov '25

I'm going to preface this with a comment that I'm not completely sold on this idea yet, but I think it's worth a larger discussion and might be a seed to help with discussion. I've tagged Ironic as this proposal is for Ironic specifically, but I'm very, very interested in the larger community opinion for this kind of approach so I also tagged all. One of the toughest parts about walking the path to becoming core is how fuzzy the targets are. With Ironic recently having added a two-tier review system, with reviewers and approvers separate, we have an opportunity to allow a much, much bigger tent for the ironic-reviewers group without risking code getting merged without approval by one of our folks with more experience and context in OpenStack and Ironic as a whole. So here's the suggestion -- we come up with a static criteria and period with a clear bar to "if you do this, you are an ironic-reviewer". For instance, we could say (I'm not sure about these numbers): if you do an average of Y (5?) reviews/week and have merged a major Ironic feature or multiple bugfix patches, you would automatically be a reviewer. Obviously, we'd have to keep safeguards to prevent abuse; in our community in the past we had a rule that changes required review from two separate corporate entities -- in this case, I'd suggest a waiting period to prevent quick-merging by someone with a single perspective. Exceptions would be, as in the current system, for emergency CI fixes and documentation improvements. To be clear: the safeguards shouldn't be seen as a lack of trust, but maybe instead as an acknowledgement that structural controls hold value even if there's no reason to assume mistrust. Full proposal: * ironic-approvers (+2 and Workflow +1) continues to operate as usual, with admission being based on technical merit, trust, and review frequency. * ironic-reviewers (+2 only) becomes a data-driven group, where reviewers are added/removed based on individual performance metrics TBD (likely reviews/commits/bugs fixed). -- I'll note that today, there's not really any set of reasonable metrics that would add additional reviewers to this group. * Code changes would either need a review from two separate company-representatives, or to have sat for a short waiting period to ensure other folks had a chance to look at it. * Urgent CI fixes, trivial changes, and documentation updates would still only need one review to land. What do folks think? Would this be a good carrot for existing non-cores to up their review and upstream contribution commitments? - Jay Faulkner G-Research OSS

6 6

Kolla-Ansible deployment results in broken dashboard
by Ricardo Matheus 04 Nov '25

04 Nov '25

Hi all I just deployed a multinode OpenStack using Kolla-Ansible, but it seems that the Horizon image (ubuntu / master) is somewhat broken. Horizon runs and the dashboard is accessible, but every file in the "static" part of the web server is not available (no logo images, no css, nothing... just plain unformatted html). Is this a known bug? (btw, there was a guy asking about the same problem in irc a few days ago, it is in the logs for 2025-10-23 (Thursday) <https://meetings.opendev.org/irclogs/%23openstack/%23openstack.2025-10-23.l…> ) Cheers! RDM

2 2

[ironic] Stop maintaining sushy as a separate project, let's move it into ironic
by Iury Gregory 01 Nov '25

01 Nov '25

Hello everyone! During the PTG yesterday, we discussed stop maintaining sushy as a separate project. For those who don't know what sushy is: a Python library to communicate with Redfish based systems and is limited to what Ironic supports. The library was never intended for others to consume, yet that has happened, and it being a library has been a noted hurdle to quickly resolving user issues, because it requires releasing a new version. But before we move this forward, we would like to hear from you what are your thoughts. Thanks! -- *Att[]'s* *Iury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Ironic Core - Ironic Event Liaison (DPL)* *Senior Software Engineer at Red Hat Brazil* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

6 6

[nova][glance] Image corruption during shelving
by Vladimir Prokofev 31 Oct '25

31 Oct '25

Hello everyone. I've discovered a bug in nova/glanceclient interoperation that leads to image corruption during shelving under some circumstances involving connection disruption. I was able to reproduce it in two different production environments running on xena and 2024.2 respectively, I'm also able to reproduce it in devstack though without image corruption - there it just ends up in a shelve failure due to detected corruption. My setup assumes LVM-backed QEMU VMs with backend in CEPH for Glance, but I believe this is applicable to a variety of nova/glance backends. When shelve is triggered, nova-compute creates an image file locally, and then initiates upload of said image file into Glance[0]. If something happens to the connection during the upload("broken pipe", "connection timeout") - nova-compute retries upload operation[1] while image object is removed from CEPH backend by Glance[2] Problem here is that the image_file object that is passed to glanceclient by nova-compute is a byte-stream created with an open()[3] call, and upon retry it resumes the upload from the point where it was interrupted. This is easily confirmed by calling image_data.tell() in glanceclient.v2.images.Controller.upload() function - it will be at zero initially and non-zero on retry. I haven't thoroughly checked protection offered in Glance master, I believe it may have been significantly improved, but in older releases such as 2024.2, this leads to a corrupted shelved image, because upon retry only part of an image object is uploaded, after which original VM is offloaded(removed) and you end up with lost data. Now, my issue here is where to submit this bug: glanceclient or nova? This problem is easily fixable in glanceclient by calling image_data.seek(0) in glanceclient.v2.images.Controller.upload(): it makes sense to always point byte-stream to the beginning before initiating upload, but should it really be responsibility of a client to perform such a sanity check? I'm also not sure if there're cases that this will break, for example if a non-seekable object is passed into glanceclient, but I'm not sure if this is even possible? [0] https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driv… [1] https://opendev.org/openstack/nova/src/branch/master/nova/image/glance.py#L… [2] https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_… [3] https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driv…

2 2

[ptg] Image Encryption Session, Thursday 30th @ 16:00 UTC
by Markus Hentsch 30 Oct '25

30 Oct '25

Dear OpenStack community, in preparation for the vPTG discussion slot for the image encryption (Thursday 30th at 16:00 UTC) I collected a summary explaining the key points and state of the contribution and gathered some discussion topics below. Since this spans multiple projects (mainly Glance, Cinder and Nova) and has been ongoing for quite some years, I tried my best to briefly summarize the most important aspects of the current state so that all those who wish to participate have the latest information. The text below can also be found on the etherpad: https://etherpad.opendev.org/p/2026.1-ptg-glance-planning#L128 Summary: - historically, only Cinder has been able to create and use encrypted images (as a side effect) by dumping LUKS-encrypted volumes 1:1 into images, for exclusive use in Cinder - this contribution aims to introduce encrypted image functionality for the end user: images can be encrypted/decrypted outside of OpenStack and resulting (encrypted) image is stored in Glance and the key in Barbican - goal: standardize encrypted image formats and handling across Glance, Cinder and Nova - it introduces two formats: raw LUKS and qcow2-wrapped LUKS (native qcow2 feature) - it introduces new key handling: if Barbican secret is of type 'passphrase' it is used as the LUKS key as-is; if it is of any other secret type, it is assumed to be binary and converted via 'binascii.hexlify()' (preserves Cinder behavior and compatibility) - it makes use of the new secret consumer API in Barbican to prevent accidental key deletion (since keys may be managed by users, not OpenStack), registering images as consumers to their keys - note: the image encryption feature focuses on Glance and Cinder for now, Nova has been largely excluded so far due to ephemeral storage encryption not being ready yet Implementation State (note: aside from os-brick this is not merged yet!): - os-brick: implemented outsourced utility function to discern encryption between passphrase and binary (hexlify conversion) for shared usage by Nova and Cinder - Glance: implemented new encryption-specific image formats and properties as well as secret consumer handling - Cinder: implemented encrypted image usage for volumes: LUKS is handled as raw (like before) and qcow2+LUKS is converted; image creation from volume stays limited to raw LUKS - Nova: implemented only compatibility with os-brick and Cinder changes so far - barbican-tempest-plugin: full scenario tests (image > volume > server > SSH) for 4 permutations of the image encryption Patchsets: https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 Discussion Topics: 1) Image inspection: related to CVE-2024-32498 concerns were brought up about encrypted image payload containing malicious formats triggering unintended QEMU behavior; how to screen encrypted image payload and reject undesired formats? 1a) limiting to specific formats has consequences for Cinder: attached (secondary) encrypted volumes might contain arbitrary binary data in VMs (user choice), when uploading as image to Glance it might be rejected as prohibited format; additionally, such image vanishes without trace because there is no error state for images (bad Glance <-> Cinder interaction) - how to address this? 1b) how deep do we want to look into encrypted image payload? decrypting a 1TB image might be very costly (CPU load, disk space) - problem: "qemu-img convert" cannot stream to stdout (see https://blogs.igalia.com/berto/2025/07/15/converting-qemu-qcow2-images-dire… ); in the worst case we have to wait for the full 1TB to be decrypted, consuming almost twice the amount of disk space in the process, instead of being able to only read the payload header and abort 2) Nova integration: how to approach future interoperability with ephemeral storage encryption? 2a) can we handle this as separate future expansion to this feature, i.e., merging Glance+Cinder functionality short-term with only compatibility changes in Nova for now and addressing full Nova integration as a next step with dedicated patchsets based on the current work? 3) Better organization: would there be any interest in reviving the image encryption popup-team meeting? https://meetings.opendev.org/#Image_Encryption_Popup-Team_Meeting Best regards, Markus Hentsch -- Markus Hentsch DevOps Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 | 01099 Dresden +49 351 479 367 00 markus.hentsch(a)cloudandheat.com | www.cloudandheat.com Green, Open, Efficient. Ihr Cloud-Service- und Cloud-Technologie-Provider aus Dresden. https://www.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann

2 3

[skyline] possible small bug in skyline
by Taavi Ansper 30 Oct '25

30 Oct '25

Hi I have found a possible small bug in skyline. I didn't know where to post this in bugzilla. https://bugs.launchpad.net/skyline-apiserver/+bug/2129915 The bug comes from the placement API request to: //resource_providers/1ab32e69-ee87-4ccb-bd8a-b39abfa4a980/inventories The same UI view in horizon works nicely and it doesn't append the the double slash in the API request. Where might this come from the code? Something related to this is defined here, but it probably comes from the "Base" stuff: https://github.com/openstack/skyline-console/blob/stable/2025.1/src/client/… https://github.com/openstack/skyline-console/blob/stable/2025.1/src/client/… This issue appeared after upgrading from 2024.2 to 2025.1 Best regards -- ---- Taavi Ansper taavi.ansper(a)cyber.ee

1 0

[openstack][kolla-ansible]about group node config overwrite
by Nguyễn Hữu Khôi 29 Oct '25

29 Oct '25

Hello Koller. I want to ask if we can overwrite config for a group of nodes. For example, compute01 >> compute10 will have the same nova configuration compute11 >> compute20 will have the same nova configuration. It will be very nice if we support this way. Thank you. Regards Nguyen Huu Khoi

4 7

[nova][glance] Server rescue falls back to default image
by Francesco Di Nucci 29 Oct '25

29 Oct '25

Hi all, I am on OpenStack Caracal, trying to rescue a VM that's not booting: * I previously created a SystemRescueCD rescue image (with openstack image create --file ./systemrescue-12.02-amd64.iso --disk-format raw --container-format bare --property hw_rescue_bus=virtio --property hw_rescue_device=disk systemrescuecd_1202) - it works on VMs that still have their base images in Glance * tried to rescue the VM (with openstack server rescue --image systemrescuecd_1202 my-vm-id-here) * but the process fails as it tries to also use the base image used to create that VM, instead of the rescue image only (Instance my-vm-id-here cannot be rescued: Driver Error: Image b78cf456-b395-434f-bf50-a1f2f3684a37 could not be found. ; where b78cf456-b395-434f-bf50-a1f2f3684a37 was a Ubuntu cloud image used to create that VM, since deleted as it's an EOL version) Is it supposed to also try to use the base image or should the --image bypass it? In case, what could I do? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email:francesco.dinucci@na.infn.it

3 2

[kolla-ansible] Rabbitmq queue creation issues in "fresh" 2025.1 deploy
by Jonathan Proulx 28 Oct '25

28 Oct '25

Hi All, I'm putting "fresh" in quotes becuase I expect the issue is some home my deploy isn't as fresh as I belive it to be. I did have and older test environment which was serveral steps behind (and not quite workign for other reasons). The presenting issues is this error (and other similar ones for other services): 2025-10-24 11:44:07.840 1076 WARNING oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Queue conductor could not be declared probably because of conflicting configurations: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false': amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'true' but current is 'false' 2025-10-24 11:44:07.841 1076 INFO oslo.messaging._drivers.impl_rabbit [None req-4eff1f78-c16a-4bc9-b0da-9ce0a80b66ba - - - - - -] [83941886-4e9c-4d0c-b3b0-53c107f018e9] Retrying to declare the exchange (nova) as non durable I'm puzzled since the services are deploying to recently reinstalled nodes (I'm sure that part is fresh) how the queues can be created with flags the services don't expect. my shell history tells me I'm using these versions (in a venev): pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/2025.1 pip3 install --upgrade git+https://opendev.org/openstack/kolla@stable/2025.1 pip3 install --upgrade 'ansible-core>=2.17,<2.18.99' The only bits I know are left over and "upgraded" from previous attempts are my configs in /etc/kolla. My deploy host wasn't reinstalled but I did delete and recreate the venv. Using Ubuntu 24.04 as the base OS Where do I go from here to find my wrong turn? Thanks, -Jon -- Jonathan "Some Day I'll Upgrade From Mitaka" Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL

2 4