Dear Sean,
so this si the disconnect when moving forward with the integration into nova you need to actually reach out again to agree that the design is still correct. all feature need to be re-appoved for each release by each team affected in a given release. we di not ask you to never talk to us again we asked you to come back to use when the depencies in cinder/glance were resolved and it was time to work on the nova part.
Thanks for clarifying your point of view on this! Let me share our perspective so that we can shed some light on why we didn't approach the Nova team with a dedicated blueprint/spec for the current patchset that we propose and figure out what exactly is missing. Outside of unittests, our patchset only changes about 20 lines in Nova as it is very tiny and addresses only two key points: 1. Correctly detect the newly established "os_encrypt_*" image properties in addition to the already existing "cinder_encryption_*" ones for the sole purpose of blocking those properties from getting inherited or ephemeral storage from getting created based on those images (which would fail since we have no compatible ephemeral storage encryption yet). 2. Switch to os-brick for converting Barbican keys to LUKS passphrases when attaching encrypted volumes. This is to support proper decryption of volume data which was created from encrypted images in Cinder with the new mechanism. It also fully retains the old key conversion behavior (hexlify), so unless it encounters the corresponding new "os_encrypt_*" volume metadata, the behavior in Nova is the exact same as before. This is all. There is no new feature for Nova in the patchset. It only ensures it correctly prepares keys for encrypted Cinder volumes which make use of a new passphrase conversion stemming from a new form of encrypted image. All the image handling happens outside of Nova, in either Cinder or Glance. The current proposed Nova patchset in addition to the overarching Glance and Cinder ones is the first step of getting standardized image encryption adopted across services, focusing only on images and volumes (i.e. Glance and Cinder) for now and not yet addressing ephemeral storage (as agreed with Nova). As such, this patchset is completely unrelated to any kind of ephemeral storage encryption or image handling in Nova except for the fact that the properties established with the new image encryption standardization in Glance may (and imo should) also be reused by such implementation in the future, which would consist of dedicated blueprints/specs and patchsets, some of which are already WIP as you already linked earlier. Based on this fact, we didn't consider a dedicated Nova spec or blueprint to be necessary for this minor adjustment to Nova for supporting changes in Cinder and Glance. It was no ill intent from our side to skip any of the required steps for contributions. Sorry if this was a misjudgment on our side. We'd appreciate your guidance if any form of documentation is still required, such as a specless blueprint. For the remainder of this message, let me address each technical point you raised in your last two mails in chronological order individually to make sure that we do not miss anything crucial.
Since you are using some of the same image properties that were created for local storage encryption, we need to ensure that they do not conflict with the usage described in
https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem... I think it is actually the other way around: for the standardized image encryption, we defined those new properties in our specs early on [1] [2] and then were adopted on your side over time [3]. Which is good, because fragmentation would hurt the whole interoperability that we had in mind when we initially proposed this standardization. Our Glance spec [2] is already merged, which describes the properties and their purpose in detail. So as long as we don't deviate from that in Nova in the future, I don't think there is any conflict. [1] https://review.opendev.org/c/openstack/nova-specs/+/608696 [2] https://review.opendev.org/c/openstack/glance-specs/+/915726 [3] https://review.opendev.org/c/openstack/nova-specs/+/907654/comment/a5209786_...
In any case, the patch is incorrect as it is using new image properties without extending https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L15... Nova does not support arbitrary image properties. We have a defined set of standard image properties in code, and any other image property is considered invalid, so you should extend that with the new os_* ones.
Any image with the new encryption parameters is not intended to be directly used in Nova (in the scope of our current patchset for now, that is). We explicitly bail out with an error if we encounter such an image while ephemeral storage is being created. Our current image encryption patchset is only meant for Cinder and Glance for use between images and volumes. Any integration with ephemeral storage encryption would be future work, based on the Glance spec.
I'm not seeing any consideration for scheduling based on this capability (there is no compute service bump or traits), so we can't select a host that supports this when creating a VM or moving it.
Our current implementation only concerns (attaching) Cinder volumes which are LUKS encrypted. The only change we introduce is to add an additional method to convert Barbican keys to LUKS passphrases before passed to Libvirt, in case they are direct passphrases and don't need binascii.hexlify conversion. All this happens in os-brick. There is no functional change in Nova or any new requirement to compute hosts. The encryption is the exact same as always has been, since LUKS-encrypted Cinder volumes already existed for a long time. It's only the key handling that has slight changes (calling or not calling `binascii.hexlify()` on it). All the actual image handling happens only in Cinder, way before it gets to Nova.
However, you have not raised the min version of os-brick, so that needs to be addressed.
The necessary changes in os-brick are already in since 6.9.0 [4] and currently Nova depends on 6.10.0. [4] https://github.com/openstack/os-brick/commit/2b42257eff03615816dc66d3171b87c...
Have you considered how this will work in a partly upgraded cloud where some nodes use the new approach and others use the old approach? 2025.2 is not a slurp, but we still need to support 2025.1 compute nodes, so it should not be possible to snapshot a VM on a 2025.2 host, then move it to a 2025.1 host and fail to restore from the snapshot via a rebuild.
I'm not sure if I'm understanding this completely the right way, so correct me if I'm wrong, but snapshots of VMs with attached volumes only lead to volume snapshots in Cinder directly for said volumes, right? (see also my last answer way down for more details) Since our changes are not related to any kind of ephemeral storage and image processing in Nova, I don't think we have any divergence with partly upgraded clouds and snapshots?
By the way, in addition to the work Melanie started for local image encryption, there was also
https://review.opendev.org/c/openstack/nova-specs/+/883516/3/specs/2023.2/ap...
to support NFS volumes with LUKS encryption. All three of these efforts need to work together in the long term although the other two are currently dormant but intended to be resumed in a future release.
We would be very glad to, as this was our intention from the very start. Let me quote from our Glance spec [5]: "That’s why we propose the introduction of a streamlined encrypted image format along with well-defined metadata specifications which will be supported across OpenStack services for the existing encryption implementations and increase interoperability as well as usability for users." [5] https://specs.openstack.org/openstack/glance-specs/specs/2025.1/approved/gla...
in 2024 in dalmatian Dan asked for change to the local ephermal encuypion spec specifically related to ensuring that when we take a snapshot of a guest with an encrypted root disk that we support specifying how the encyption is done for the snapshot image.
https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem...
this is a nova api change that default to using the same barbiacn key as the root image but can generate a new key or use an existing secret key provide by the user. this change did not merge [...] but form a nove presepctive i do nto see why this woudl be any differnt for cinder boot volumes.
I think the key difference here with regards to our proposed patchset is that any image (and key) handling concerning volumes happens outside of Nova and Nova is not involved in the image encryption but only the volume decryption (for attachment). Creating a snapshot (createImage action) of a VM that has volumes attached creates Cinder volume snapshots of affected volumes (in Cinder) and only actually produces image data for the ephemeral storage parts (if any). If you inspect the resulting image in Glance, you can see volume snapshot ID references to Cinder in the image metadata but the actual image payload blob will either contain only ephemeral storage dumps (of an ephemeral storage root disk) or nothing (in case the root disk was booted from volume). As such, Nova is not involved in the creation or consumption of encrypted images and their keys in case of volumes originating from such images, as all of this happens within Cinder. And for sake of completeness: in case of the createBackup Nova API action, volume-backed VMs are not supported anyway and any attached volumes on ephemeral-storage-based ones are ignored. I hope this addresses all of your technical concerns. Let me know if I missed anything! Kind regards, Markus Sean Mooney schrieb:
On 14/08/2025 14:43, Markus Hentsch wrote:
Dear Sean,
Procedurally, the problem is you do not have an approved spec or blueprint for this feature in Nova, and it has been several releases since we last discussed this feature. [...] One of the things that is not supported in your series is direct booting of an encrypted image. [...] Even though this is a small patch it is quite a lot to ask for Nova to review, especially when it was not discussed at the PTG with the Nova or Cinder teams.
With all due respect, I think this is misrepresenting the history of this whole contribution quite a bit. As we initially started with this initiative all the way back in 2019 with a lot of back and forth over time, the history and agreements may not be fully obvious to all parties involved, so let me recap the key points.
While talking to the TC about the challenges of starting a contribution in such cross-project manner in the beginning, we were advised to initiate a popup team for the image encryption. We organized a regular popup team IRC meeting from 2019 all the way into 2025 [1], asking all relevant parties (Nova, Cinder, Glance, Barbican) to participate.
We initially also did a Nova spec for this but very early on (2019) the Nova team said that they wanted to take care of LUKS support for ephemeral storage encryption on their side first before considering/adopting any local image encryption [2.1] [2.2] [2.3]. At this time we agreed with Nova to leave ephemeral storage out of it and focus on working with Glance, Cinder and Barbican to get the base image encryption format and interoperability volumes in Cinder landed.
Between 2019 and 2023 we did lots of iterations on the basic integration, wrote specs for Glance as well as Cinder and waited for the implementation of the Secret Consumers API in Barbican [3.1] [3.2] for the final workflow because it was crucial that when using Barbican secrets (image keys) cross-service between Glance and Cinder that usage is properly tracked and keys aren't deleted while still in-use by either.
In April 2024 we had a cross project session with Nova and Glance at the PTG [4]! There was a big discussion around the encryption format initiated by Dan Smith (Nova). He proposed to move away from GPG and use LUKS instead because this would streamline it with existing functionality and formats that are already available in Nova and Cinder. Due to this proposal from Nova, we agreed to discard our existing patchsets [5] and rewrite our image encryption feature with new patchsets [6] with LUKS as the encryption format, as suggested by Dan Smith (Nova). We also talked specifically about the cryptographic key differentiation (hexlify vs. non-hexlify) which materialized in the os-brick change that you mentioned.
From the beginning until the middle of 2025 we unfortunately had little resources to work on this from our side, so communication and progress took a bit of hit but we tried to continue the best we could. We intensified our work on this again over the last few months and got the patchsets focusing on images and volumes in good shape together with Glance and Cinder while also adding full Tempest scenario tests covering the full usage across Glance, Cinder and Nova.
We are now at a point where this is working for Glance and Cinder with favorable reviews from their side. The Nova team only requested to have the encryption format changed in the 2024 PTG, which we did as mentioned above but did not amend their stance on their involvement at any point. As we did agree in 2019 to leave Nova out of the implementation (as requested) and focus on Glance and Cinder, the patchset we are proposing to Nova is the bare minimum to get what we did for Glance and Cinder working in Nova but it does not touch anything else in Nova beyond that, especially ephemeral storage. And this is by design, because we were specifically asked not to and kept to the agreement.
so this si the disconnect when moving forward with the integration into nova you need to actually reach out again to agree that the design is still correct. all feature need to be re-appoved for each release by each team affected in a given release. we di not ask you to never talk to us again we asked you to come back to use when the depencies in cinder/glance were resolved and it was time to work on the nova part.
in 2024 in dalmatian Dan asked for change to the local ephermal encuypion spec specifically related to ensuring that when we take a snapshot of a guest with an encrypted root disk that we support specifying how the encyption is done for the snapshot image. https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem...
this is a nova api change that default to using the same barbiacn key as the root image but can generate a new key or use an existing secret key provide by the user. this change did not merge because of the CVEs related to backing files and data file in qcow2 and vmdk formats that were foudn during that cycle. but form a nove presepctive i do nto see why this woudl be any differnt for cinder boot volumes.
now i have not had time to dug into your patch in detail but looking at the comment form lee https://review.opendev.org/c/openstack/nova/+/926326/3/nova/compute/api.py#7... the code xispclity skip the check in the boot form volume case so that resolve the main issue i had with that.
these design issue evolve over time so what may have been ok in 2024 may not be ok in 2025 of ohter features have landed in between.
dan also responed so if dan has context and has time review given the size it might eb oke to proceed but this should have been approved as a specless bluering in the nova irc
meeting proir to milestoen 2 on july 3rd.
I hope this can clear up some of the misunderstanding around the topic and can help us identify what exactly is missing for Nova and how to proceed for the first iteration of this feature. Based on the discussions so far, I think ideally we would have image encryption compatibility for Cinder volumes only (in Nova) and have the base ready for an extension towards ephemeral storage encryption in the future. We will look at the technical points you raised in more detail and see if we can address them appropriately, either by adjusting a patchset or documenting this in dedicated spec(s).
Please let us know if you think we can discuss this more efficiently by joining the Nova IRC meetings again, creating a dedicated PTG slot with Nova and/or reviving the popup team slot.
Kind regards,
Markus
[1] https://meetings.opendev.org/meetings/image_encryption/
[2.1] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-a8642c... [2.2] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-e597b0... [2.3] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption...
[3.1] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption... [3.2] https://meetings.opendev.org/meetings/image_encryption/2023/image_encryption...
[4] https://etherpad.opendev.org/p/dalmatian-ptg-cinder#L382
[5] https://review.opendev.org/q/topic:%22image-encryption%22+status:abandoned
[6] https://review.opendev.org/q/topic:%22LUKS-image-encryption%22
-- Markus Hentsch DevOps Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 | 01099 Dresden +49 351 479 367 00 markus.hentsch@cloudandheat.com | www.cloudandheat.com Green, Open, Efficient. Ihr Cloud-Service- und Cloud-Technologie-Provider aus Dresden. https://www.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann