Hi nova folks, We worked on the LUKS-based image encryption and the patches Cinder and Glance are in a good state now + we also have tempest tests. Here you can see the overall state: https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 There is one patch in nova, which adresses an issue with keys for the encrypted image/volume being either used directly or being hexlified beforehand. We would be really happy if you could review this nova patch: https://review.opendev.org/c/openstack/nova/+/926326 So we can hopefully finish our work on the Image encryption. greetings Josephine (Luzi)
On 14/08/2025 09:25, Josephine Seifert wrote:
Hi nova folks,
We worked on the LUKS-based image encryption and the patches Cinder and Glance are in a good state now + we also have tempest tests. Here you can see the overall state:
https://review.opendev.org/q/topic:%22LUKS-image-encryption%22
There is one patch in nova, which adresses an issue with keys for the encrypted image/volume being either used directly or being hexlified beforehand. We would be really happy if you could review this nova patch: https://review.opendev.org/c/openstack/nova/+/926326
So we can hopefully finish our work on the Image encryption. Procedurally, the problem is you do not have an approved spec or blueprint for this feature in Nova, and it has been several releases since we last discussed this feature. So technically, this is not eligible to merge this cycle. One of the things that is not supported in your series is direct booting of an encrypted image.
I'm not sure if we are okay with that limitation. In the long term, we are not however it was already there when using the now deprecated options so its not making it any worse then the current state. We had already started implementing support form booting form encrypted media in Nova and had to pause it when we found the CVEs related to VMDK, etc., two years ago. Since you are using some of the same image properties that were created for local storage encryption, we need to ensure that they do not conflict with the usage described in https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem... In any case, the patch is incorrect as it is using new image properties without extending https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L15... Nova does not support arbitrary image properties. We have a defined set of standard image properties in code, and any other image property is considered invalid, so you should extend that with the new os_* ones. I'm not seeing any consideration for scheduling based on this capability (there is no compute service bump or traits), so we can't select a host that supports this when creating a VM or moving it. This would also need to work for all relevant lifecycle operations. Looking at the patch again, you mainly seem to be replacing existing functionality with delegation to os-brick https://review.opendev.org/c/openstack/nova/+/926326/3/nova/virt/libvirt/dri... However, you have not raised the min version of os-brick, so that needs to be addressed. Obviously, os-brick needs to be released with the new version, and that needs to be updated in upper-constraints before this can merge on the Nova side. Have you considered how this will work in a partly upgraded cloud where some nodes use the new approach and others use the old approach? 2025.2 is not a slurp, but we still need to support 2025.1 compute nodes, so it should not be possible to snapshot a VM on a 2025.2 host, then move it to a 2025.1 host and fail to restore from the snapshot via a rebuild. Based on https://review.opendev.org/c/openstack/nova/+/926326/3/nova/tests/unit/compu... I think you are doing that properly, but these are all the concerns that should have been addressed by a Nova spec. By the way, in addition to the work Melanie started for local image encryption, there was also https://review.opendev.org/c/openstack/nova-specs/+/883516/3/specs/2023.2/ap... to support NFS volumes with LUKS encryption. All three of these efforts need to work together in the long term although the other two are currently dormant but intended to be resumed in a future release. Even though this is a small patch it is quite a lot to ask for Nova to review, especially when it was not discussed at the PTG with the Nova or Cinder teams. https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d https://etherpad.opendev.org/p/r.c8d78881250998a7afc9b0929c99a8bf There was a meeting slot for this in the Glance room https://etherpad.opendev.org/p/r.cd74ef1b45e2f338a145173a65b98315#L96 but there are no comments or minutes related to that discussion if it happened. There is a lot of context not captured in the patch that would be needed to properly understand this change. So, procedurally, I'm tempted to say -2 on the Nova change because you have not followed the feature proposal process and got aproval do make this chagne for this cycle. This is a non-trivial thing to show up with three weeks before feature freeze, when we are ~6 weeks past the spec/blueprint approval deadline for this cycle(m2). The only reason I have not done that is this is a relatively small patch, and if all the concerns I raised have been addressed, it might be okay to proceed with. I am not going to have time to dig into this enough to prove that to myself, so really this is up to the other cores if they have time. Part of the reason we have the blueprint and spec freeze at M2 is to prevent late requests like this so we can complete the work that was planned for the release. We have quite a few planned enhancements still pending, so I want to spend most of my energy trying to land those.
greetings Josephine (Luzi)
Dear Sean,
Procedurally, the problem is you do not have an approved spec or blueprint for this feature in Nova, and it has been several releases since we last discussed this feature. [...] One of the things that is not supported in your series is direct booting of an encrypted image. [...] Even though this is a small patch it is quite a lot to ask for Nova to review, especially when it was not discussed at the PTG with the Nova or Cinder teams.
With all due respect, I think this is misrepresenting the history of this whole contribution quite a bit. As we initially started with this initiative all the way back in 2019 with a lot of back and forth over time, the history and agreements may not be fully obvious to all parties involved, so let me recap the key points. While talking to the TC about the challenges of starting a contribution in such cross-project manner in the beginning, we were advised to initiate a popup team for the image encryption. We organized a regular popup team IRC meeting from 2019 all the way into 2025 [1], asking all relevant parties (Nova, Cinder, Glance, Barbican) to participate. We initially also did a Nova spec for this but very early on (2019) the Nova team said that they wanted to take care of LUKS support for ephemeral storage encryption on their side first before considering/adopting any local image encryption [2.1] [2.2] [2.3]. At this time we agreed with Nova to leave ephemeral storage out of it and focus on working with Glance, Cinder and Barbican to get the base image encryption format and interoperability volumes in Cinder landed. Between 2019 and 2023 we did lots of iterations on the basic integration, wrote specs for Glance as well as Cinder and waited for the implementation of the Secret Consumers API in Barbican [3.1] [3.2] for the final workflow because it was crucial that when using Barbican secrets (image keys) cross-service between Glance and Cinder that usage is properly tracked and keys aren't deleted while still in-use by either. In April 2024 we had a cross project session with Nova and Glance at the PTG [4]! There was a big discussion around the encryption format initiated by Dan Smith (Nova). He proposed to move away from GPG and use LUKS instead because this would streamline it with existing functionality and formats that are already available in Nova and Cinder. Due to this proposal from Nova, we agreed to discard our existing patchsets [5] and rewrite our image encryption feature with new patchsets [6] with LUKS as the encryption format, as suggested by Dan Smith (Nova). We also talked specifically about the cryptographic key differentiation (hexlify vs. non-hexlify) which materialized in the os-brick change that you mentioned. From the beginning until the middle of 2025 we unfortunately had little resources to work on this from our side, so communication and progress took a bit of hit but we tried to continue the best we could. We intensified our work on this again over the last few months and got the patchsets focusing on images and volumes in good shape together with Glance and Cinder while also adding full Tempest scenario tests covering the full usage across Glance, Cinder and Nova. We are now at a point where this is working for Glance and Cinder with favorable reviews from their side. The Nova team only requested to have the encryption format changed in the 2024 PTG, which we did as mentioned above but did not amend their stance on their involvement at any point. As we did agree in 2019 to leave Nova out of the implementation (as requested) and focus on Glance and Cinder, the patchset we are proposing to Nova is the bare minimum to get what we did for Glance and Cinder working in Nova but it does not touch anything else in Nova beyond that, especially ephemeral storage. And this is by design, because we were specifically asked not to and kept to the agreement. I hope this can clear up some of the misunderstanding around the topic and can help us identify what exactly is missing for Nova and how to proceed for the first iteration of this feature. Based on the discussions so far, I think ideally we would have image encryption compatibility for Cinder volumes only (in Nova) and have the base ready for an extension towards ephemeral storage encryption in the future. We will look at the technical points you raised in more detail and see if we can address them appropriately, either by adjusting a patchset or documenting this in dedicated spec(s). Please let us know if you think we can discuss this more efficiently by joining the Nova IRC meetings again, creating a dedicated PTG slot with Nova and/or reviving the popup team slot. Kind regards, Markus [1] https://meetings.opendev.org/meetings/image_encryption/ [2.1] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-a8642c... [2.2] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-e597b0... [2.3] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption... [3.1] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption... [3.2] https://meetings.opendev.org/meetings/image_encryption/2023/image_encryption... [4] https://etherpad.opendev.org/p/dalmatian-ptg-cinder#L382 [5] https://review.opendev.org/q/topic:%22image-encryption%22+status:abandoned [6] https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 -- Markus Hentsch DevOps Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 | 01099 Dresden +49 351 479 367 00 markus.hentsch@cloudandheat.com | www.cloudandheat.com Green, Open, Efficient. Ihr Cloud-Service- und Cloud-Technologie-Provider aus Dresden. https://www.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann
One of the things that is not supported in your series is direct booting of an encrypted image.
I could be wrong, but I think this is just a simplistic read of the first addition in the patch. AFAIK, the direct-boot abort is already in the tree, and they are just adding an additional check for the new key id parameter to mirror the same (existing) behavior. That is, of course, fine.
In April 2024 we had a cross project session with Nova and Glance at the PTG [4]! There was a big discussion around the encryption format initiated by Dan Smith (Nova). He proposed to move away from GPG and use LUKS instead because this would streamline it with existing functionality and formats that are already available in Nova and Cinder. Due to this proposal from Nova, we agreed to discard our existing patchsets [5] and rewrite our image encryption feature with new patchsets [6] with LUKS as the encryption format, as suggested by Dan Smith (Nova). We also talked specifically about the cryptographic key differentiation (hexlify vs. non-hexlify) which materialized in the os-brick change that you mentioned.
Yep, this and the rest of your history summary matches my recollection as well. I know I've been on the hook to review this stuff and just keep getting pulled in different directions on more important stuff. My apologies, but there are some pretty important things up for review right now (like eventlet removal). Your patch to use brick for the passphrase extraction seems like a fine thing to merge at this point, especially because the earlier we merge it the better from the compatibility point of view. I'll try to make time today to look at it in detail. --Dan
On 14/08/2025 15:09, Dan Smith wrote:
One of the things that is not supported in your series is direct booting of an encrypted image. I could be wrong, but I think this is just a simplistic read of the first addition in the patch. AFAIK, the direct-boot abort is already in the tree, and they are just adding an additional check for the new key id parameter to mirror the same (existing) behavior. That is, of course, fine.
yes it is just an exteion of that but you shoudl be able to sue if for the "boot form voluem form inmage workflow" no? we had a very long converation about local iamge encryption and why you were not ok with breakign the workflow of creating a vm, modifying it to customisze it and sthen creating a snapshot and booting addtioanl vms. if the snapshot you are takign is of the boot volume and you don't support that work flow for that as well then we have a conflcit between the requrieemtn for both features. if something is taking a snapshot of a data volume and uploading it as an image that is diffent as that data volume is presumabel not marked as bootable anyway in cinder. booking using the encrypted image for local storage is totally valid as we have not implemented that in nova yet but i woudl expect the BFV case to work.
In April 2024 we had a cross project session with Nova and Glance at the PTG [4]! There was a big discussion around the encryption format initiated by Dan Smith (Nova). He proposed to move away from GPG and use LUKS instead because this would streamline it with existing functionality and formats that are already available in Nova and Cinder. Due to this proposal from Nova, we agreed to discard our existing patchsets [5] and rewrite our image encryption feature with new patchsets [6] with LUKS as the encryption format, as suggested by Dan Smith (Nova). We also talked specifically about the cryptographic key differentiation (hexlify vs. non-hexlify) which materialized in the os-brick change that you mentioned. Yep, this and the rest of your history summary matches my recollection as well.
that all fine and it more or less aligned with my recollection of this to however it misses the point that feature proposals be it tracked by a blueprint or spec are only accpted for a given release and need to be epxlictly propsoed again for the next cycle if tye are not complete. so even if it was acepted as a specless blueprint or an actul spec in 2024 on the nova side it would still need one for this cycle. appoval for dalmaion 2024.2 expired at the start of the 2025.1 cycle.
I know I've been on the hook to review this stuff and just keep getting pulled in different directions on more important stuff. My apologies, but there are some pretty important things up for review right now (like eventlet removal). Your patch to use brick for the passphrase extraction seems like a fine thing to merge at this point, especially because the earlier we merge it the better from the compatibility point of view. I'll try to make time today to look at it in detail.
--Dan
On 14/08/2025 14:43, Markus Hentsch wrote:
Dear Sean,
Procedurally, the problem is you do not have an approved spec or blueprint for this feature in Nova, and it has been several releases since we last discussed this feature. [...] One of the things that is not supported in your series is direct booting of an encrypted image. [...] Even though this is a small patch it is quite a lot to ask for Nova to review, especially when it was not discussed at the PTG with the Nova or Cinder teams.
With all due respect, I think this is misrepresenting the history of this whole contribution quite a bit. As we initially started with this initiative all the way back in 2019 with a lot of back and forth over time, the history and agreements may not be fully obvious to all parties involved, so let me recap the key points.
While talking to the TC about the challenges of starting a contribution in such cross-project manner in the beginning, we were advised to initiate a popup team for the image encryption. We organized a regular popup team IRC meeting from 2019 all the way into 2025 [1], asking all relevant parties (Nova, Cinder, Glance, Barbican) to participate.
We initially also did a Nova spec for this but very early on (2019) the Nova team said that they wanted to take care of LUKS support for ephemeral storage encryption on their side first before considering/adopting any local image encryption [2.1] [2.2] [2.3]. At this time we agreed with Nova to leave ephemeral storage out of it and focus on working with Glance, Cinder and Barbican to get the base image encryption format and interoperability volumes in Cinder landed.
Between 2019 and 2023 we did lots of iterations on the basic integration, wrote specs for Glance as well as Cinder and waited for the implementation of the Secret Consumers API in Barbican [3.1] [3.2] for the final workflow because it was crucial that when using Barbican secrets (image keys) cross-service between Glance and Cinder that usage is properly tracked and keys aren't deleted while still in-use by either.
In April 2024 we had a cross project session with Nova and Glance at the PTG [4]! There was a big discussion around the encryption format initiated by Dan Smith (Nova). He proposed to move away from GPG and use LUKS instead because this would streamline it with existing functionality and formats that are already available in Nova and Cinder. Due to this proposal from Nova, we agreed to discard our existing patchsets [5] and rewrite our image encryption feature with new patchsets [6] with LUKS as the encryption format, as suggested by Dan Smith (Nova). We also talked specifically about the cryptographic key differentiation (hexlify vs. non-hexlify) which materialized in the os-brick change that you mentioned.
From the beginning until the middle of 2025 we unfortunately had little resources to work on this from our side, so communication and progress took a bit of hit but we tried to continue the best we could. We intensified our work on this again over the last few months and got the patchsets focusing on images and volumes in good shape together with Glance and Cinder while also adding full Tempest scenario tests covering the full usage across Glance, Cinder and Nova.
We are now at a point where this is working for Glance and Cinder with favorable reviews from their side. The Nova team only requested to have the encryption format changed in the 2024 PTG, which we did as mentioned above but did not amend their stance on their involvement at any point. As we did agree in 2019 to leave Nova out of the implementation (as requested) and focus on Glance and Cinder, the patchset we are proposing to Nova is the bare minimum to get what we did for Glance and Cinder working in Nova but it does not touch anything else in Nova beyond that, especially ephemeral storage. And this is by design, because we were specifically asked not to and kept to the agreement.
so this si the disconnect when moving forward with the integration into nova you need to actually reach out again to agree that the design is still correct. all feature need to be re-appoved for each release by each team affected in a given release. we di not ask you to never talk to us again we asked you to come back to use when the depencies in cinder/glance were resolved and it was time to work on the nova part. in 2024 in dalmatian Dan asked for change to the local ephermal encuypion spec specifically related to ensuring that when we take a snapshot of a guest with an encrypted root disk that we support specifying how the encyption is done for the snapshot image. https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem... this is a nova api change that default to using the same barbiacn key as the root image but can generate a new key or use an existing secret key provide by the user. this change did not merge because of the CVEs related to backing files and data file in qcow2 and vmdk formats that were foudn during that cycle. but form a nove presepctive i do nto see why this woudl be any differnt for cinder boot volumes. now i have not had time to dug into your patch in detail but looking at the comment form lee https://review.opendev.org/c/openstack/nova/+/926326/3/nova/compute/api.py#7... the code xispclity skip the check in the boot form volume case so that resolve the main issue i had with that. these design issue evolve over time so what may have been ok in 2024 may not be ok in 2025 of ohter features have landed in between. dan also responed so if dan has context and has time review given the size it might eb oke to proceed but this should have been approved as a specless bluering in the nova irc meeting proir to milestoen 2 on july 3rd.
I hope this can clear up some of the misunderstanding around the topic and can help us identify what exactly is missing for Nova and how to proceed for the first iteration of this feature. Based on the discussions so far, I think ideally we would have image encryption compatibility for Cinder volumes only (in Nova) and have the base ready for an extension towards ephemeral storage encryption in the future. We will look at the technical points you raised in more detail and see if we can address them appropriately, either by adjusting a patchset or documenting this in dedicated spec(s).
Please let us know if you think we can discuss this more efficiently by joining the Nova IRC meetings again, creating a dedicated PTG slot with Nova and/or reviving the popup team slot.
Kind regards,
Markus
[1] https://meetings.opendev.org/meetings/image_encryption/
[2.1] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-a8642c... [2.2] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-e597b0... [2.3] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption...
[3.1] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption... [3.2] https://meetings.opendev.org/meetings/image_encryption/2023/image_encryption...
[4] https://etherpad.opendev.org/p/dalmatian-ptg-cinder#L382
[5] https://review.opendev.org/q/topic:%22image-encryption%22+status:abandoned
[6] https://review.opendev.org/q/topic:%22LUKS-image-encryption%22
Dear Sean,
so this si the disconnect when moving forward with the integration into nova you need to actually reach out again to agree that the design is still correct. all feature need to be re-appoved for each release by each team affected in a given release. we di not ask you to never talk to us again we asked you to come back to use when the depencies in cinder/glance were resolved and it was time to work on the nova part.
Thanks for clarifying your point of view on this! Let me share our perspective so that we can shed some light on why we didn't approach the Nova team with a dedicated blueprint/spec for the current patchset that we propose and figure out what exactly is missing. Outside of unittests, our patchset only changes about 20 lines in Nova as it is very tiny and addresses only two key points: 1. Correctly detect the newly established "os_encrypt_*" image properties in addition to the already existing "cinder_encryption_*" ones for the sole purpose of blocking those properties from getting inherited or ephemeral storage from getting created based on those images (which would fail since we have no compatible ephemeral storage encryption yet). 2. Switch to os-brick for converting Barbican keys to LUKS passphrases when attaching encrypted volumes. This is to support proper decryption of volume data which was created from encrypted images in Cinder with the new mechanism. It also fully retains the old key conversion behavior (hexlify), so unless it encounters the corresponding new "os_encrypt_*" volume metadata, the behavior in Nova is the exact same as before. This is all. There is no new feature for Nova in the patchset. It only ensures it correctly prepares keys for encrypted Cinder volumes which make use of a new passphrase conversion stemming from a new form of encrypted image. All the image handling happens outside of Nova, in either Cinder or Glance. The current proposed Nova patchset in addition to the overarching Glance and Cinder ones is the first step of getting standardized image encryption adopted across services, focusing only on images and volumes (i.e. Glance and Cinder) for now and not yet addressing ephemeral storage (as agreed with Nova). As such, this patchset is completely unrelated to any kind of ephemeral storage encryption or image handling in Nova except for the fact that the properties established with the new image encryption standardization in Glance may (and imo should) also be reused by such implementation in the future, which would consist of dedicated blueprints/specs and patchsets, some of which are already WIP as you already linked earlier. Based on this fact, we didn't consider a dedicated Nova spec or blueprint to be necessary for this minor adjustment to Nova for supporting changes in Cinder and Glance. It was no ill intent from our side to skip any of the required steps for contributions. Sorry if this was a misjudgment on our side. We'd appreciate your guidance if any form of documentation is still required, such as a specless blueprint. For the remainder of this message, let me address each technical point you raised in your last two mails in chronological order individually to make sure that we do not miss anything crucial.
Since you are using some of the same image properties that were created for local storage encryption, we need to ensure that they do not conflict with the usage described in
https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem... I think it is actually the other way around: for the standardized image encryption, we defined those new properties in our specs early on [1] [2] and then were adopted on your side over time [3]. Which is good, because fragmentation would hurt the whole interoperability that we had in mind when we initially proposed this standardization. Our Glance spec [2] is already merged, which describes the properties and their purpose in detail. So as long as we don't deviate from that in Nova in the future, I don't think there is any conflict. [1] https://review.opendev.org/c/openstack/nova-specs/+/608696 [2] https://review.opendev.org/c/openstack/glance-specs/+/915726 [3] https://review.opendev.org/c/openstack/nova-specs/+/907654/comment/a5209786_...
In any case, the patch is incorrect as it is using new image properties without extending https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L15... Nova does not support arbitrary image properties. We have a defined set of standard image properties in code, and any other image property is considered invalid, so you should extend that with the new os_* ones.
Any image with the new encryption parameters is not intended to be directly used in Nova (in the scope of our current patchset for now, that is). We explicitly bail out with an error if we encounter such an image while ephemeral storage is being created. Our current image encryption patchset is only meant for Cinder and Glance for use between images and volumes. Any integration with ephemeral storage encryption would be future work, based on the Glance spec.
I'm not seeing any consideration for scheduling based on this capability (there is no compute service bump or traits), so we can't select a host that supports this when creating a VM or moving it.
Our current implementation only concerns (attaching) Cinder volumes which are LUKS encrypted. The only change we introduce is to add an additional method to convert Barbican keys to LUKS passphrases before passed to Libvirt, in case they are direct passphrases and don't need binascii.hexlify conversion. All this happens in os-brick. There is no functional change in Nova or any new requirement to compute hosts. The encryption is the exact same as always has been, since LUKS-encrypted Cinder volumes already existed for a long time. It's only the key handling that has slight changes (calling or not calling `binascii.hexlify()` on it). All the actual image handling happens only in Cinder, way before it gets to Nova.
However, you have not raised the min version of os-brick, so that needs to be addressed.
The necessary changes in os-brick are already in since 6.9.0 [4] and currently Nova depends on 6.10.0. [4] https://github.com/openstack/os-brick/commit/2b42257eff03615816dc66d3171b87c...
Have you considered how this will work in a partly upgraded cloud where some nodes use the new approach and others use the old approach? 2025.2 is not a slurp, but we still need to support 2025.1 compute nodes, so it should not be possible to snapshot a VM on a 2025.2 host, then move it to a 2025.1 host and fail to restore from the snapshot via a rebuild.
I'm not sure if I'm understanding this completely the right way, so correct me if I'm wrong, but snapshots of VMs with attached volumes only lead to volume snapshots in Cinder directly for said volumes, right? (see also my last answer way down for more details) Since our changes are not related to any kind of ephemeral storage and image processing in Nova, I don't think we have any divergence with partly upgraded clouds and snapshots?
By the way, in addition to the work Melanie started for local image encryption, there was also
https://review.opendev.org/c/openstack/nova-specs/+/883516/3/specs/2023.2/ap...
to support NFS volumes with LUKS encryption. All three of these efforts need to work together in the long term although the other two are currently dormant but intended to be resumed in a future release.
We would be very glad to, as this was our intention from the very start. Let me quote from our Glance spec [5]: "That’s why we propose the introduction of a streamlined encrypted image format along with well-defined metadata specifications which will be supported across OpenStack services for the existing encryption implementations and increase interoperability as well as usability for users." [5] https://specs.openstack.org/openstack/glance-specs/specs/2025.1/approved/gla...
in 2024 in dalmatian Dan asked for change to the local ephermal encuypion spec specifically related to ensuring that when we take a snapshot of a guest with an encrypted root disk that we support specifying how the encyption is done for the snapshot image.
https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem...
this is a nova api change that default to using the same barbiacn key as the root image but can generate a new key or use an existing secret key provide by the user. this change did not merge [...] but form a nove presepctive i do nto see why this woudl be any differnt for cinder boot volumes.
I think the key difference here with regards to our proposed patchset is that any image (and key) handling concerning volumes happens outside of Nova and Nova is not involved in the image encryption but only the volume decryption (for attachment). Creating a snapshot (createImage action) of a VM that has volumes attached creates Cinder volume snapshots of affected volumes (in Cinder) and only actually produces image data for the ephemeral storage parts (if any). If you inspect the resulting image in Glance, you can see volume snapshot ID references to Cinder in the image metadata but the actual image payload blob will either contain only ephemeral storage dumps (of an ephemeral storage root disk) or nothing (in case the root disk was booted from volume). As such, Nova is not involved in the creation or consumption of encrypted images and their keys in case of volumes originating from such images, as all of this happens within Cinder. And for sake of completeness: in case of the createBackup Nova API action, volume-backed VMs are not supported anyway and any attached volumes on ephemeral-storage-based ones are ignored. I hope this addresses all of your technical concerns. Let me know if I missed anything! Kind regards, Markus Sean Mooney schrieb:
On 14/08/2025 14:43, Markus Hentsch wrote:
Dear Sean,
Procedurally, the problem is you do not have an approved spec or blueprint for this feature in Nova, and it has been several releases since we last discussed this feature. [...] One of the things that is not supported in your series is direct booting of an encrypted image. [...] Even though this is a small patch it is quite a lot to ask for Nova to review, especially when it was not discussed at the PTG with the Nova or Cinder teams.
With all due respect, I think this is misrepresenting the history of this whole contribution quite a bit. As we initially started with this initiative all the way back in 2019 with a lot of back and forth over time, the history and agreements may not be fully obvious to all parties involved, so let me recap the key points.
While talking to the TC about the challenges of starting a contribution in such cross-project manner in the beginning, we were advised to initiate a popup team for the image encryption. We organized a regular popup team IRC meeting from 2019 all the way into 2025 [1], asking all relevant parties (Nova, Cinder, Glance, Barbican) to participate.
We initially also did a Nova spec for this but very early on (2019) the Nova team said that they wanted to take care of LUKS support for ephemeral storage encryption on their side first before considering/adopting any local image encryption [2.1] [2.2] [2.3]. At this time we agreed with Nova to leave ephemeral storage out of it and focus on working with Glance, Cinder and Barbican to get the base image encryption format and interoperability volumes in Cinder landed.
Between 2019 and 2023 we did lots of iterations on the basic integration, wrote specs for Glance as well as Cinder and waited for the implementation of the Secret Consumers API in Barbican [3.1] [3.2] for the final workflow because it was crucial that when using Barbican secrets (image keys) cross-service between Glance and Cinder that usage is properly tracked and keys aren't deleted while still in-use by either.
In April 2024 we had a cross project session with Nova and Glance at the PTG [4]! There was a big discussion around the encryption format initiated by Dan Smith (Nova). He proposed to move away from GPG and use LUKS instead because this would streamline it with existing functionality and formats that are already available in Nova and Cinder. Due to this proposal from Nova, we agreed to discard our existing patchsets [5] and rewrite our image encryption feature with new patchsets [6] with LUKS as the encryption format, as suggested by Dan Smith (Nova). We also talked specifically about the cryptographic key differentiation (hexlify vs. non-hexlify) which materialized in the os-brick change that you mentioned.
From the beginning until the middle of 2025 we unfortunately had little resources to work on this from our side, so communication and progress took a bit of hit but we tried to continue the best we could. We intensified our work on this again over the last few months and got the patchsets focusing on images and volumes in good shape together with Glance and Cinder while also adding full Tempest scenario tests covering the full usage across Glance, Cinder and Nova.
We are now at a point where this is working for Glance and Cinder with favorable reviews from their side. The Nova team only requested to have the encryption format changed in the 2024 PTG, which we did as mentioned above but did not amend their stance on their involvement at any point. As we did agree in 2019 to leave Nova out of the implementation (as requested) and focus on Glance and Cinder, the patchset we are proposing to Nova is the bare minimum to get what we did for Glance and Cinder working in Nova but it does not touch anything else in Nova beyond that, especially ephemeral storage. And this is by design, because we were specifically asked not to and kept to the agreement.
so this si the disconnect when moving forward with the integration into nova you need to actually reach out again to agree that the design is still correct. all feature need to be re-appoved for each release by each team affected in a given release. we di not ask you to never talk to us again we asked you to come back to use when the depencies in cinder/glance were resolved and it was time to work on the nova part.
in 2024 in dalmatian Dan asked for change to the local ephermal encuypion spec specifically related to ensuring that when we take a snapshot of a guest with an encrypted root disk that we support specifying how the encyption is done for the snapshot image. https://specs.openstack.org/openstack/nova-specs/specs/2024.2/approved/ephem...
this is a nova api change that default to using the same barbiacn key as the root image but can generate a new key or use an existing secret key provide by the user. this change did not merge because of the CVEs related to backing files and data file in qcow2 and vmdk formats that were foudn during that cycle. but form a nove presepctive i do nto see why this woudl be any differnt for cinder boot volumes.
now i have not had time to dug into your patch in detail but looking at the comment form lee https://review.opendev.org/c/openstack/nova/+/926326/3/nova/compute/api.py#7... the code xispclity skip the check in the boot form volume case so that resolve the main issue i had with that.
these design issue evolve over time so what may have been ok in 2024 may not be ok in 2025 of ohter features have landed in between.
dan also responed so if dan has context and has time review given the size it might eb oke to proceed but this should have been approved as a specless bluering in the nova irc
meeting proir to milestoen 2 on july 3rd.
I hope this can clear up some of the misunderstanding around the topic and can help us identify what exactly is missing for Nova and how to proceed for the first iteration of this feature. Based on the discussions so far, I think ideally we would have image encryption compatibility for Cinder volumes only (in Nova) and have the base ready for an extension towards ephemeral storage encryption in the future. We will look at the technical points you raised in more detail and see if we can address them appropriately, either by adjusting a patchset or documenting this in dedicated spec(s).
Please let us know if you think we can discuss this more efficiently by joining the Nova IRC meetings again, creating a dedicated PTG slot with Nova and/or reviving the popup team slot.
Kind regards,
Markus
[1] https://meetings.opendev.org/meetings/image_encryption/
[2.1] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-a8642c... [2.2] https://review.opendev.org/c/openstack/nova-specs/+/608696/11#message-e597b0... [2.3] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption...
[3.1] https://meetings.opendev.org/meetings/image_encryption/2019/image_encryption... [3.2] https://meetings.opendev.org/meetings/image_encryption/2023/image_encryption...
[4] https://etherpad.opendev.org/p/dalmatian-ptg-cinder#L382
[5] https://review.opendev.org/q/topic:%22image-encryption%22+status:abandoned
[6] https://review.opendev.org/q/topic:%22LUKS-image-encryption%22
-- Markus Hentsch DevOps Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 | 01099 Dresden +49 351 479 367 00 markus.hentsch@cloudandheat.com | www.cloudandheat.com Green, Open, Efficient. Ihr Cloud-Service- und Cloud-Technologie-Provider aus Dresden. https://www.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann
participants (4)
-
Dan Smith
-
Josephine Seifert
-
Markus Hentsch
-
Sean Mooney