Wait, I just started realizing the bugfix .. So basically all previously uploaded images not in raw format are useless after upgrading so the safe version of cinder/nova? That is a very nasty regression for any public cloud I would say, as users used to upload images in arbitrary formats and rely on future conversions. And also - in case you have Ceph for cinder volumes (that requires raw), and local storage with qcow for nova - that means you now need to maintain 2 set of images, in raw and qcow respectfully? What would really make life slightly easier - a way to mandate usage of interoperable import, in case it still has the option to convert image to required format on upload.... But I guess it was also prohibited? On Fri, Jul 19, 2024, 23:09 Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:

Hello. Thank you very much, Do you mean it is a feature?. But I think it will be hard, because raw images are too big to use. It wll make complex things in operation. Qcow2 is too famous. Nguyen Huu Khoi

On Sat, Jul 20, 2024 at 3:59 AM Clark Boylan <cboylan@sapwetik.org> wrote:

...

On Fri, Jul 19, 2024, at 1:53 PM, Nguyễn Hữu Khôi wrote:

...

Hello, i encounter this error when apply patch about bug/2059809:

raise exception.ImageConversionNotAllowed( 2024-07-20 03:35:28.067 736 ERROR oslo_messaging.rpc.server cinder.exception.ImageConversionNotAllowed: Image Conversion disallowed for image 7bacbdfb-2ea3-4ed8-90ad-3ec0e274a1d9: Image conversion is disabled. The volume type you have requested requires that the image it is being created from be in 'raw' format, but the image you are using has the disk_format property 'qcow2'. You must use an image with the disk_format property 'raw' to create a volume of this type.

I am using Zed version. Could I have some suggestions?

The issue is that converting arbitrary qcow2 images to other formats is not necessarily secure. The bug fix you refer to addressed the security concerns with this process in OpenStack by preventing the conversion in the first place. This means you will need to upload images in the format that they will be used in. In this case the format is a raw image. If you upload a raw image version of the image you wish to use I suspect it will work.

Note if you convert the image locally you are potentially exposing yourself to the same risk. You should ensure you trust the image source before doing a conversion locally as well.

...

Thank you.

Nguyen Huu Khoi

On Fri, Jul 19, 2024, at 2:23 PM, Dmitriy Rabotyagov wrote:

Wait, I just started realizing the bugfix ..

So basically all previously uploaded images not in raw format are useless after upgrading so the safe version of cinder/nova? That is a very nasty regression for any public cloud I would say, as users used to upload images in arbitrary formats and rely on future conversions.

See below. I think maybe the conversion is still allowed in some circumstances (the config doesn't prevent it and the image checks don't find problems?).

And also - in case you have Ceph for cinder volumes (that requires raw), and local storage with qcow for nova - that means you now need to maintain 2 set of images, in raw and qcow respectfully?

I think nova can use raw as well as qcow2 so you could standardize on just raw images if necessary. But again see below. This may be less of an issue with the specific security fixes and more to do with configuration options (that may have been opted into in response to the security concern?)

What would really make life slightly easier - a way to mandate usage of interoperable import, in case it still has the option to convert image to required format on upload.... But I guess it was also prohibited?

On Fri, Jul 19, 2024, 23:09 Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:

...

Hello. Thank you very much, Do you mean it is a feature?. But I think it will be hard, because raw images are too big to use. It wll make complex things in operation. Qcow2 is too famous.

Actually looking more closely at the specific error you're getting it originates from here: https://opendev.org/openstack/cinder/src/branch/unmaintained/zed/cinder/imag... and that config option seems to allow conversions by default: https://opendev.org/openstack/cinder/src/branch/unmaintained/zed/cinder/imag.... So I'm less positive that this is directly related to 2059809. Are you sure this was happening due to the security bug fixes? Is it possible that instead you flipped the option to True and it was False before? Note that I believe cinder must convert image to raw and can't operate on them as qcow2 files so you don't really save much space in the end I don't think.

...

Nguyen Huu Khoi

On Sat, Jul 20, 2024 at 3:59 AM Clark Boylan <cboylan@sapwetik.org> wrote:

...

On Fri, Jul 19, 2024, at 1:53 PM, Nguyễn Hữu Khôi wrote:

...

Hello, i encounter this error when apply patch about bug/2059809:

raise exception.ImageConversionNotAllowed( 2024-07-20 03:35:28.067 736 ERROR oslo_messaging.rpc.server cinder.exception.ImageConversionNotAllowed: Image Conversion disallowed for image 7bacbdfb-2ea3-4ed8-90ad-3ec0e274a1d9: Image conversion is disabled. The volume type you have requested requires that the image it is being created from be in 'raw' format, but the image you are using has the disk_format property 'qcow2'. You must use an image with the disk_format property 'raw' to create a volume of this type.

I am using Zed version. Could I have some suggestions?

The issue is that converting arbitrary qcow2 images to other formats is not necessarily secure. The bug fix you refer to addressed the security concerns with this process in OpenStack by preventing the conversion in the first place. This means you will need to upload images in the format that they will be used in. In this case the format is a raw image. If you upload a raw image version of the image you wish to use I suspect it will work.

Note if you convert the image locally you are potentially exposing yourself to the same risk. You should ensure you trust the image source before doing a conversion locally as well.

...

Thank you.

Nguyen Huu Khoi

On Fri, 2024-07-19 at 21:44 +0000, Jeremy Stanley wrote:

On 2024-07-19 23:23:38 +0200 (+0200), Dmitriy Rabotyagov wrote:

...

Wait, I just started realizing the bugfix ..

So basically all previously uploaded images not in raw format are useless after upgrading so the safe version of cinder/nova? [...]

I'm no expert in Glance matters, but my understanding is that with OSSA-2024-001 fixes in place you can't lie about the kind of image you're uploading. If you upload QCOW2 you must upload it as QCOW2 type and not try to pretend that it's a raw image, or upload an ISO as ISO type and not try to claim that it's raw. If you want to upload a different image type than what the cloud wants you to upload, then Glance should still be able to perform image conversion for you as well.

Previously, you could pretend a QCOW2 was raw, and then rely on QEmu's image auto-detection to figure out what it really was, but that's dangerous because QEmu is not intended to be used with untrusted images and some specially-crafted images can be used by an attacker to get copies of arbitrary files off the server where its tools (even the qemu-img executable) are run. Glance now has an image inspection feature which doesn't rely on QEmu itself to determine whether an image is safe enough to hand off to QEmu tools or might be used to root your cloud.

yes the way we mitigated this CVE was by adding logic that allow use to validate the source fromat matches what it was specified as in glance. so image conversion is still a thing that you can do nova for example still allow you to configre the comptue hosts to use raw or qcow images and you can then boot a vm form any format qemu can convert form provided you uploaded it to glance and declard the format correctly we have ci converge ensuring you can boot form a amazon ami image the cirros uec images with nova configured to use qcow for the guest vm root disk. all of that still works but if you upload a qcow as a raw image to glance then nova and cinder will say "no you said this was raw this is unsafe" and refues to proceed. we are currently missing that for iso format but its on our todo list to build out more tempest coverage for image formats that are reasonable to use. its still possible to have the old conversion behavior provided you dont lie about the format when uploading it to glance. we also do other check to introspect the format for unsafe features. for qcow backing files are an example of a generally unsafe feature. qcow image used with openstack should be single file flat images with no refernce to backing files or other external files.

On Sat, 2024-07-20 at 19:21 +0200, Dmitriy Rabotyagov wrote:

Aha, ok, thanks so much for clarifying, that makes way more sense now!

I was actually wondering if running such conversions inside some kind of namespaces (sandboxing) was raised as an option to isolate conversions from the rest of data and why it was rejected (if it was raised)?

it wasn't but it would not be backporatable if we were to do that. we cannot introduced any new depencices on podman or systemd ectra. as such adding sandboxing would be a new feature which could only be done on master. it is something we can consider but we would have to ensure that any solution we did use would not break any deployemnt tool that deploy openstack inside our outside of a container so its less simple then you might think to do that. kolla deploys opentack in docker. redhat's new installer deploy opesntack on a mixture of openshfit and podman on rhel tripleo used podman canonical sunbeam uses a mix of containers (lxd?) and snaps? the point im trying to make is its very hard to create a sand box that will work in all deployment tools. we cant assume we can use podman, docker, systemd nspwan or flatpaks https://github.com/containers/bubblewrap as using any of the above may break one of all of the deployment tools. it may also cause issues with thing like apparmor and selinux. so adding sandboxing would be a very non trivial effort. that does not mean we cant explore that in the future but it would not be somethign we could do as part of the cve fix.

On Sat, Jul 20, 2024, 13:50 <smooney@redhat.com> wrote:

...

On Fri, 2024-07-19 at 21:44 +0000, Jeremy Stanley wrote:

...

On 2024-07-19 23:23:38 +0200 (+0200), Dmitriy Rabotyagov wrote:

...

Wait, I just started realizing the bugfix ..

So basically all previously uploaded images not in raw format are useless after upgrading so the safe version of cinder/nova? [...]

I'm no expert in Glance matters, but my understanding is that with OSSA-2024-001 fixes in place you can't lie about the kind of image you're uploading. If you upload QCOW2 you must upload it as QCOW2 type and not try to pretend that it's a raw image, or upload an ISO as ISO type and not try to claim that it's raw. If you want to upload a different image type than what the cloud wants you to upload, then Glance should still be able to perform image conversion for you as well.

Previously, you could pretend a QCOW2 was raw, and then rely on QEmu's image auto-detection to figure out what it really was, but that's dangerous because QEmu is not intended to be used with untrusted images and some specially-crafted images can be used by an attacker to get copies of arbitrary files off the server where its tools (even the qemu-img executable) are run. Glance now has an image inspection feature which doesn't rely on QEmu itself to determine whether an image is safe enough to hand off to QEmu tools or might be used to root your cloud.

yes the way we mitigated this CVE was by adding logic that allow use to validate the source fromat matches what it was specified as in glance.

so image conversion is still a thing that you can do nova for example still allow you to configre the comptue hosts to use raw or qcow images and you can then boot a vm form any format qemu can convert form provided you uploaded it to glance and declard the format correctly

we have ci converge ensuring you can boot form a amazon ami image the cirros uec images with nova configured to use qcow for the guest vm root disk.

all of that still works but if you upload a qcow as a raw image to glance then nova and cinder will say "no you said this was raw this is unsafe" and refues to proceed.

we are currently missing that for iso format but its on our todo list to build out more tempest coverage for image formats that are reasonable to use.

its still possible to have the old conversion behavior provided you dont lie about the format when uploading it to glance. we also do other check to introspect the format for unsafe features. for qcow backing files are an example of a generally unsafe feature.

qcow image used with openstack should be single file flat images with no refernce to backing files or other external files.

On Mon, 2024-07-22 at 13:46 +0200, Dmitriy Rabotyagov wrote:

Oh, yes, for sure it won't be applicable as a CVE fix, but could be some day2 discussion for the path forward to eliminate all possible future vectors, which could possibly simplify logic.

Why I was thinking about linux namespaces specifically, is because all of these deployment toolings (maybe except sunbeam?) are used to (and still do) deal with OVS Neutron driver, which rely on network namespaces a lot. So, theoretically, they could be used to isolate qemu-img from both networking and generic filesystem access for purposes of image conversion.

so redhat is deploying cinder volume on openshift and it does not support user namespaces and user namespacs are experimental in k8s we may or many not have premeisoin to create network namesapces or chroots when hosted on k8s/openshift for cinder volume but it might be possible on the compute node side at least for out new insttall nova is deployed a podman container and we may be able to create nested namesapces btu i dont think openstack code in generall should do that. neutron is an exception ranther than a template to follow in my view but im not ruling out some addtiona form of lockdown via namespace or similar. i would prefer to take the approce fo havign a pure python implementation of the image inspectorts with support for all the formats we support and reject all that are not recognised. that would be simpler for us to lockdown then sandboxing qemu-img. we wont be reimpelmting convert and resize logic but we can remove the use of qemu-img for basic image inspection which will significantly reduce the attach surface. that is the current direction we are moving in.

But yeah. It really can be non-trivial, slow and not worth it after all. But mainly I was just wondering about thoughts on that topic overall and if this was ever raised at all.

пн, 22 июл. 2024 г. в 13:20, <smooney@redhat.com>:

...

On Sat, 2024-07-20 at 19:21 +0200, Dmitriy Rabotyagov wrote:

...

Aha, ok, thanks so much for clarifying, that makes way more sense now!

I was actually wondering if running such conversions inside some kind of namespaces (sandboxing) was raised as an option to isolate conversions from the rest of data and why it was rejected (if it was raised)?

it wasn't but it would not be backporatable if we were to do that. we cannot introduced any new depencices on podman or systemd ectra.

as such adding sandboxing would be a new feature which could only be done on master. it is something we can consider but we would have to ensure that any solution we did use would not break any deployemnt tool that deploy openstack inside our outside of a container so its less simple then you might think to do that.

kolla deploys opentack in docker. redhat's new installer deploy opesntack on a mixture of openshfit and podman on rhel

tripleo used podman

canonical sunbeam uses a mix of containers (lxd?) and snaps?

the point im trying to make is its very hard to create a sand box that will work in all deployment tools.

we cant assume we can use podman, docker, systemd nspwan or flatpaks https://github.com/containers/bubblewrap as using any of the above may break one of all of the deployment tools.

it may also cause issues with thing like apparmor and selinux.

so adding sandboxing would be a very non trivial effort.

that does not mean we cant explore that in the future but it would not be somethign we could do as part of the cve fix.

...

On Sat, Jul 20, 2024, 13:50 <smooney@redhat.com> wrote:

...

On Fri, 2024-07-19 at 21:44 +0000, Jeremy Stanley wrote:

...

On 2024-07-19 23:23:38 +0200 (+0200), Dmitriy Rabotyagov wrote:

...

Wait, I just started realizing the bugfix ..

So basically all previously uploaded images not in raw format are useless after upgrading so the safe version of cinder/nova? [...]

I'm no expert in Glance matters, but my understanding is that with OSSA-2024-001 fixes in place you can't lie about the kind of image you're uploading. If you upload QCOW2 you must upload it as QCOW2 type and not try to pretend that it's a raw image, or upload an ISO as ISO type and not try to claim that it's raw. If you want to upload a different image type than what the cloud wants you to upload, then Glance should still be able to perform image conversion for you as well.

Previously, you could pretend a QCOW2 was raw, and then rely on QEmu's image auto-detection to figure out what it really was, but that's dangerous because QEmu is not intended to be used with untrusted images and some specially-crafted images can be used by an attacker to get copies of arbitrary files off the server where its tools (even the qemu-img executable) are run. Glance now has an image inspection feature which doesn't rely on QEmu itself to determine whether an image is safe enough to hand off to QEmu tools or might be used to root your cloud.

yes the way we mitigated this CVE was by adding logic that allow use to validate the source fromat matches what it was specified as in glance.

so image conversion is still a thing that you can do nova for example still allow you to configre the comptue hosts to use raw or qcow images and you can then boot a vm form any format qemu can convert form provided you uploaded it to glance and declard the format correctly

we have ci converge ensuring you can boot form a amazon ami image the cirros uec images with nova configured to use qcow for the guest vm root disk.

all of that still works but if you upload a qcow as a raw image to glance then nova and cinder will say "no you said this was raw this is unsafe" and refues to proceed.

we are currently missing that for iso format but its on our todo list to build out more tempest coverage for image formats that are reasonable to use.

its still possible to have the old conversion behavior provided you dont lie about the format when uploading it to glance. we also do other check to introspect the format for unsafe features. for qcow backing files are an example of a generally unsafe feature.

qcow image used with openstack should be single file flat images with no refernce to backing files or other external files.

Hello. I get it but It happened only when I upgrade from Yoga to Zed. I don't touch image_conversion_disable. Btw, thank you very much. I checked it and change it to False. On Sat, Jul 20, 2024, 5:19 AM Jeremy Stanley <fungi@yuggoth.org> wrote:

On 2024-07-20 03:53:21 +0700 (+0700), Nguyễn Hữu Khôi wrote:

...

Hello, i encounter this error when apply patch about bug/2059809:

raise exception.ImageConversionNotAllowed( 2024-07-20 03:35:28.067 736 ERROR oslo_messaging.rpc.server cinder.exception.ImageConversionNotAllowed: Image Conversion disallowed for image 7bacbdfb-2ea3-4ed8-90ad-3ec0e274a1d9: Image conversion is disabled. The volume type you have requested requires that the image it is being created from be in 'raw' format, but the image you are using has the disk_format property 'qcow2'. You must use an image with the disk_format property 'raw' to create a volume of this type.

I am using Zed version. Could I have some suggestions?

This looks like you've set the image_conversion_disable option in your Cinder configuration. That was added two years ago with https://review.opendev.org/839793 in order to provide a way for cloud operators to prevent on-the-fly image conversions from consuming additional system resources.

The Cinder fix for bug 2059809 (OSSA-2024-001, CVE-2024-32498) does not, that I can see, block image conversion operations. -- Jeremy Stanley