Is there a safer way to handle untrusted disk images than qemu-img?
For a while now I've been wondering if there's a safer way to handle untrusted disk images in various clouds. They're a frequent cause of security vulnerability announcements, largely because qemu is a fairly complicated code base where the block device handling is entangled into the larger product, qemu also implements a variety of features clouds don't even want, and poorly intentioned people can spend as much time as they want crafting custom images to find interesting edge conditions in the implementation. I am not convinced that the current strategy of closing gaps as they're found is really paying off. Recent examples of such `qemu-img` related vulnerabilities include: * CVE-2026-24708 (OSSA-2026-002) * CVE-2024-44082 (OSSA-2024-003) * CVE-2024-4467 (OSSA-2024-002) * CVE-2024-32498 (OSSA-2024-001) * CVE-2022-47951 (OSSA-2023-002) I toyed with the idea of a "stripped down" `qemu-img` which could simply not have the features which cause these vulnerabilities, but the complexity of the qemu codebase makes this approach painful, while also not presenting any of the other opportunities of a greenfields approach like memory safety and a tight sandbox. So what might a safer approach look like? My answer is to try and offload the processing of untrusted data to the virtual machine implementation in silicon -- it has strong separation guarantees and it can be orchestrated so that there is simply no access to anything not required for the strict processing of the image. So for example no access to a file system, a network, and so forth. Specifically, what if processing this untrusted and potentially malicious data was done as a KVM guest with a custom virtual machine manager (VMM)? So I wrote it, and by "I wrote it" I mean Claude Code mostly wrote it. Instar is a from scratch rethink of how to handle untrusted image data as well as an opportunity for me to become more familiar with code generation LLMs. All data processing happens in a custom KVM guest which does not run an operating system or have any access to system resources apart from those provided to it by the custom VMM which orchestrates it. The best way to think of this guest is as an embedded system running on a virtual CPU. This is similar in approach to how AWS Nitro Enclaves work in terms of prior art, whilst also being the opposite in intent: Nitro protects sensitive data *from* the cloud; instar protects the cloud *from* malicious data. To be clear, `instar info` doesn't detect or block malicious images -- it processes them safely and reports what it finds. The security comes from containment, not detection. It is then up to the caller to decide how to handle what is reported. Instar currently requires Linux with KVM support on x86-64. Early benchmarks suggest performance overhead from the KVM sandbox is modest -- in the noise for metadata operations like `info`. Images are provided to this guest as block devices using the virtio-block protocol, but it should be noted that while the guest and the VMM are using that protocol, the Linux kernel is not involved. Instar is implemented in rust because bare metal golang turned out to be a bit hard, and because the existing rust VMM crates are actually quite good and used by projects like Firecracker already. You can read more about the technology choices and concepts behind instar in painful detail at https://shakenfist.com/components/instar/technology-primer/ The primary use case I'm targeting is validating untrusted images at upload time (e.g., in Glance), though the same approach could apply anywhere qemu-img is invoked on user-supplied data. The current goal of Instar is to be command line compatible with qemu-img (especially `qemu-img info` and `qemu-img convert`), but I also wonder about opportunities to get out of the game of parsing command output. Whilst Instar can certainly emit JSON formatted output, there are other options as well like running it as a persistent daemon which services requests it receives over a unix domain socket. I've played that game with other code recently and find protobufs to be a reasonable protocol for such things. I don't know how OpenStack feels about protobufs however. By "command line compatible", I mean that my current goal is to provide a version of `instar info` and `instar convert` that is byte for byte identical to the output of the same `qemu-img info` or `qemu-img convert` command. This has some interesting edge cases in as much as `qemu-img` has changed its output format at least once since 2020. Additionally, I suspect I've found at least one `qemu-img` bug in this process. Instar resolves these issues by having a test suite of sample images, recording the `qemu-img` output for those images, and then generating a database of the different output formats. When you run `instar info` on your machine, instar attempts to detect the version of `qemu-img` you have installed, and will match its output format. The interesting little edge cases I've found along the way are documented at https://shakenfist.com/components/instar/quirks/, and the output formats the `qemu-img info` produces are documented at https://shakenfist.com/components/instar/output-formats/. At the moment I'd describe Instar as a "vigorous prototype". It implements `qemu-img info` and most of `qemu-img convert`, but does not yet know how to resize, rebase, snapshot, and so forth. I'd be interested in whether other people think this is an interesting idea or not, although honestly it was just kind of fun to build over the summer holidays. I'd particularly welcome feedback on whether this approach would be useful for your deployment, examples of images that produce unexpected results, and thoughts on the daemon / protobuf interface idea. Instar is licensed under Apache 2.0. The code and extensive notes are at https://github.com/shakenfist/instar/. Bug reports, examples of images which didn't work as expected, and pull requests should all go there please. Michael
Hey, Respect for not fearing to question the status quo and thinking outside of the box. I am on an SME on that particular topic, so please take my comments with a grain of salt. On Sat, 9 May 2026 at 23:00, Michael Still <mikal@stillhq.com> wrote:
So what might a safer approach look like? My answer is to try and offload the processing of untrusted data to the virtual machine implementation in silicon -- it has strong separation guarantees and it can be orchestrated so that there is simply no access to anything not required for the strict processing of the image. So for example no access to a file system, a network, and so forth. Specifically, what if processing this untrusted and potentially malicious data was done as a KVM guest with a custom virtual machine manager (VMM)?
So I wrote it, and by "I wrote it" I mean Claude Code mostly wrote it. Instar is a from scratch rethink of how to handle
For me a statement like this is unfortunately not a quality stamp. While playing around with AI agents I ensured myself that AI does not take care about future maintenance of the code. It proposes the solution to your question (depending on how you ask it) using the statistics - it is not a secret that current AI is nothing else rather than just a statistical probability machine. I have no problem with asking the AI for support but a statement "Claude mostly wrote it" is not a great signal. I recommend you not to use such phrasing if you want to reach acceptance.
untrusted image data as well as an opportunity for me to become more familiar with code generation LLMs. All data processing happens in a custom KVM guest which does not run an operating system or have any access to system resources apart from those provided to it by the custom VMM which orchestrates it. The best way to think of this guest is as an embedded system running on a virtual CPU. This is similar in approach to how AWS Nitro Enclaves work in terms of prior art, whilst also being the opposite in intent: Nitro protects sensitive data *from* the cloud; instar protects the cloud *from* malicious data. To be clear, `instar info` doesn't detect or block malicious images -- it processes them safely and reports what it finds. The security comes from containment, not detection. It is then up to the caller to decide how to handle what is reported.
I think the proper strategy would be to do both: protect images from the cloud (confidential computing) and cloud for the image. Nitro (all of them) are giving very useful hints about what could and should be done and we definitely should learn from competitors.
At the moment I'd describe Instar as a "vigorous prototype". It implements `qemu-img info` and most of `qemu-img convert`, but does not yet know how to resize, rebase, snapshot, and so forth. I'd be interested in whether other people think this is an interesting idea or not, although honestly it was just kind of fun to build over the summer holidays. I'd particularly welcome
I like the statement: "was just kind of fun over ...". Many in the community are desperately frustrated about more and more maintenance work with less and less people. We should ensure that at least this work is fun, otherwise people just burn out or give up.
feedback on whether this approach would be useful for your deployment, examples of images that produce unexpected results, and thoughts on the daemon / protobuf interface idea.
I definitely support the gRPC introduction. I am already using it in keystone-rs for adding raft support. Also I plan to expose the Keystone API also over gRPC so that control plane communication may benefit from it (not sure whether gRPC for OpenStack public API is a great thing though). Actually the idea of using gRPC in OpenStack is not new and some people were already playing around with this idea nearly 10 years ago. I also agree with your arguments why Rust is a great fit here. Great work, Artem
On Thu, May 14, 2026 at 8:24 PM Artem Goncharov <artem.goncharov@gmail.com> wrote: Thanks for taking the time to reply. I appreciate that. On Sat, 9 May 2026 at 23:00, Michael Still <mikal@stillhq.com> wrote:
So I wrote it, and by "I wrote it" I mean Claude Code mostly wrote it. Instar is a from scratch rethink of how to handle
For me a statement like this is unfortunately not a quality stamp. While playing around with AI agents I ensured myself that AI does not take care about future maintenance of the code. It proposes the solution to your question (depending on how you ask it) using the statistics - it is not a secret that current AI is nothing else rather than just a statistical probability machine. I have no problem with asking the AI for support but a statement "Claude mostly wrote it" is not a great signal. I recommend you not to use such phrasing if you want to reach acceptance.
Honestly? Meh. If the OpenStack community isn't willing to consider possible new things based on their merits alone then I am not too fussed by OpenStack missing out on new things. Instar is not proposed as a potential OpenStack project, so I see flexibility to do pretty much whatever I want as the author. I think I was also pretty clear that instar is a concept _prototype_. On the other hand, having done this for about six months, I think the "it's just statistical" view is naive. There is much more to it than that: planning frameworks, clear definitions of correctness, good precommits and CI, etc. This is one of the reasons I have started publishing the planning templates and implementation documents I use, as I think those are just as important in terms of review as including a summary prompt in the commit message. Sadly, instar doesn't have a complete history of those because it was before I realised those things, but given one of my goals was to learn about the tools that seems like a success to me regardless. Isn't another way to rephrase your stance simply as encouraging authors using ML coding tools not to disclose that because then their code isn't taken seriously? Instar runs thousands of comparison tests between its output and qemu-img's. There are 121 images in the test suite, including some crafted to specifically trigger previous vulnerabilities. So, as best as I can tell it _works_. It might not be maintainable, but it might also be. I haven't found any evidence yet that it is not. Then again, should maintainability be a primary goal for a _prototype_? Nova has had lots of histocial design missteps we've had to clean up -- surely it's unreasonable to expect machine-assisted code to be perfect at first attempt?
untrusted image data as well as an opportunity for me to become more
familiar with code generation LLMs. All data processing happens in a custom KVM guest which does not run an operating system or have any access to system resources apart from those provided to it by the custom VMM which orchestrates it. The best way to think of this guest is as an embedded system running on a virtual CPU. This is similar in approach to how AWS Nitro Enclaves work in terms of prior art, whilst also being the opposite in intent: Nitro protects sensitive data *from* the cloud; instar protects the cloud *from* malicious data. To be clear, `instar info` doesn't detect or block malicious images -- it processes them safely and reports what it finds. The security comes from containment, not detection. It is then up to the caller to decide how to handle what is reported.
I think the proper strategy would be to do both: protect images from the cloud (confidential computing) and cloud for the image. Nitro (all of them) are giving very useful hints about what could and should be done and we definitely should learn from competitors.
To my current understanding a key underpinning to nitro is hardware acceleration of KVM. Unless deployers want to go all-in on DPUs, much of that remains out of reach for many. Michael
On 15/05/2026 01:12, Michael Still wrote:
On Thu, May 14, 2026 at 8:24 PM Artem Goncharov <artem.goncharov@gmail.com> wrote:
Thanks for taking the time to reply. I appreciate that.
On Sat, 9 May 2026 at 23:00, Michael Still <mikal@stillhq.com> wrote:
So I wrote it, and by "I wrote it" I mean Claude Code mostly wrote it. Instar is a from scratch rethink of how to handle
For me a statement like this is unfortunately not a quality stamp. While playing around with AI agents I ensured myself that AI does not take care about future maintenance of the code. It proposes the solution to your question (depending on how you ask it) using the statistics - it is not a secret that current AI is nothing else rather than just a statistical probability machine. I have no problem with asking the AI for support but a statement "Claude mostly wrote it" is not a great signal. I recommend you not to use such phrasing if you want to reach acceptance.
Honestly? Meh. If the OpenStack community isn't willing to consider possible new things based on their merits alone then I am not too fussed by OpenStack missing out on new things. Instar is not proposed as a potential OpenStack project, so I see flexibility to do pretty much whatever I want as the author. I think I was also pretty clear that instar is a concept _prototype_. so nova already has configoption for xoriso command for which cli to build the config drive.
i have not had time to look at instar but if your inten is to provd a compatiabel cli with similar output we coudl likely just make that cofnigureabl in the relevent proejct can allow opertors to opt in the other approach we had been taking was to build out the image inspector in oslo. with the intent of eventually removing our dependency on qemu-image for doign the safety/validation entirely and only using it if needed for format conversion. so i would need to load context on exactly what you have built but i would not discard it because an experience engineer built it using the assistance of an llm. i have built useful tools, features and bugs-fixes with ai and when used with care it can produce high quality output.
On the other hand, having done this for about six months, I think the "it's just statistical" view is naive. There is much more to it than that: planning frameworks, clear definitions of correctness, good precommits and CI, etc. This is one of the reasons I have started publishing the planning templates and implementation documents I use, as I think those are just as important in terms of review as including a summary prompt in the commit message. Sadly, instar doesn't have a complete history of those because it was before I realised those things, but given one of my goals was to learn about the tools that seems like a success to me regardless.
Isn't another way to rephrase your stance simply as encouraging authors using ML coding tools not to disclose that because then their code isn't taken seriously?
that would be very counter productive and not a stance we shoudl take as a comunity IMO. llm or not is not the gate of quality human but the maintneres and the authors are :) devaluing the work of contibutors just ebcause the use a tool to help them woudl be counter productive so disculosure is good but shoudl not be a signal that it is of good or bad quality.
Instar runs thousands of comparison tests between its output and qemu-img's. There are 121 images in the test suite, including some crafted to specifically trigger previous vulnerabilities. So, as best as I can tell it _works_. It might not be maintainable, but it might also be. I haven't found any evidence yet that it is not. Then again, should maintainability be a primary goal for a _prototype_? Nova has had lots of histocial design missteps we've had to clean up -- surely it's unreasonable to expect machine-assisted code to be perfect at first attempt?
if its optioanl or configurable we can evaulate that with our ci. qemu-img has had many bugs and secuirty issues so human written code is also not perfect.
untrusted image data as well as an opportunity for me to become more familiar with code generation LLMs. All data processing happens in a custom KVM guest which does not run an operating system or have any access to system resources apart from those provided to it by the custom VMM which orchestrates it. The best way to think of this guest is as an embedded system running on a virtual CPU. This is similar in approach to how AWS Nitro Enclaves work in terms of prior art, whilst also being the opposite in intent: Nitro protects sensitive data *from* the cloud; instar protects the cloud *from* malicious data. To be clear, `instar info` doesn't detect or block malicious images -- it processes them safely and reports what it finds. The security comes from containment, not detection. It is then up to the caller to decide how to handle what is reported.
I think the proper strategy would be to do both: protect images from the cloud (confidential computing) and cloud for the image. Nitro (all of them) are giving very useful hints about what could and should be done and we definitely should learn from competitors.
To my current understanding a key underpinning to nitro is hardware acceleration of KVM. Unless deployers want to go all-in on DPUs, much of that remains out of reach for many.
Michael
Actually it feels to me it is way too much testosterone in the air that people overreact on every single word in every single conversation. I just said I think you should not advertise "mostly created by AI" as a form of a quality stamp. Period. Calm down, please On Fri, 15 May 2026 at 14:24, Sean Mooney <smooney@redhat.com> wrote:
On 15/05/2026 01:12, Michael Still wrote:
On Thu, May 14, 2026 at 8:24 PM Artem Goncharov <artem.goncharov@gmail.com> wrote:
Thanks for taking the time to reply. I appreciate that.
On Sat, 9 May 2026 at 23:00, Michael Still <mikal@stillhq.com> wrote:
So I wrote it, and by "I wrote it" I mean Claude Code mostly wrote it. Instar is a from scratch rethink of how to handle
For me a statement like this is unfortunately not a quality stamp. While playing around with AI agents I ensured myself that AI does not take care about future maintenance of the code. It proposes the solution to your question (depending on how you ask it) using the statistics - it is not a secret that current AI is nothing else rather than just a statistical probability machine. I have no problem with asking the AI for support but a statement "Claude mostly wrote it" is not a great signal. I recommend you not to use such phrasing if you want to reach acceptance.
Honestly? Meh. If the OpenStack community isn't willing to consider possible new things based on their merits alone then I am not too fussed by OpenStack missing out on new things. Instar is not proposed as a potential OpenStack project, so I see flexibility to do pretty much whatever I want as the author. I think I was also pretty clear that instar is a concept _prototype_. so nova already has configoption for xoriso command for which cli to build the config drive.
i have not had time to look at instar but if your inten is to provd a compatiabel cli with similar output we coudl likely just make that cofnigureabl in the relevent proejct can allow opertors to opt in
the other approach we had been taking was to build out the image inspector in oslo. with the intent of eventually removing our dependency on qemu-image for doign the safety/validation entirely and only using it if needed for format conversion.
so i would need to load context on exactly what you have built but i would not discard it because an experience engineer built it using the assistance of an llm. i have built useful tools, features and bugs-fixes with ai and when used with care it can produce high quality output.
On the other hand, having done this for about six months, I think the "it's just statistical" view is naive. There is much more to it than that: planning frameworks, clear definitions of correctness, good precommits and CI, etc. This is one of the reasons I have started publishing the planning templates and implementation documents I use, as I think those are just as important in terms of review as including a summary prompt in the commit message. Sadly, instar doesn't have a complete history of those because it was before I realised those things, but given one of my goals was to learn about the tools that seems like a success to me regardless.
Isn't another way to rephrase your stance simply as encouraging authors using ML coding tools not to disclose that because then their code isn't taken seriously?
that would be very counter productive and not a stance we shoudl take as a comunity IMO.
llm or not is not the gate of quality human but the maintneres and the authors are :) devaluing the work of contibutors just ebcause the use a tool to help them woudl be counter productive so disculosure is good but shoudl not be a signal that it is of good or bad quality.
Instar runs thousands of comparison tests between its output and qemu-img's. There are 121 images in the test suite, including some crafted to specifically trigger previous vulnerabilities. So, as best as I can tell it _works_. It might not be maintainable, but it might also be. I haven't found any evidence yet that it is not. Then again, should maintainability be a primary goal for a _prototype_? Nova has had lots of histocial design missteps we've had to clean up -- surely it's unreasonable to expect machine-assisted code to be perfect at first attempt?
if its optioanl or configurable we can evaulate that with our ci. qemu-img has had many bugs and secuirty issues so human written code is also not perfect.
untrusted image data as well as an opportunity for me to become more familiar with code generation LLMs. All data processing happens in a custom KVM guest which does not run an operating system or have any access to system resources apart from those provided to it by the custom VMM which orchestrates it. The best way to think of this guest is as an embedded system running on a virtual CPU. This is similar in approach to how AWS Nitro Enclaves work in terms of prior art, whilst also being the opposite in intent: Nitro protects sensitive data *from* the cloud; instar protects the cloud *from* malicious data. To be clear, `instar info` doesn't detect or block malicious images -- it processes them safely and reports what it finds. The security comes from containment, not detection. It is then up to the caller to decide how to handle what is reported.
I think the proper strategy would be to do both: protect images from the cloud (confidential computing) and cloud for the image. Nitro (all of them) are giving very useful hints about what could and should be done and we definitely should learn from competitors.
To my current understanding a key underpinning to nitro is hardware acceleration of KVM. Unless deployers want to go all-in on DPUs, much of that remains out of reach for many.
Michael
On Fri, May 15, 2026 at 10:24 PM Sean Mooney <smooney@redhat.com> wrote:
On 15/05/2026 01:12, Michael Still wrote:
On Thu, May 14, 2026 at 8:24 PM Artem Goncharov <artem.goncharov@gmail.com> wrote:
so nova already has configoption for xoriso command for which cli to build the config drive.
This is an aside, but my recollection is that we used an external binary for config drive creation because at the time all of the libraries we could find to do it in process were GPL'ed and that was a licensing line OpenStack wasn't willing to cross. I suspect no one has ever looked to see if there is an ISO9660 creation library which has an acceptable license now?
i have not had time to look at instar but if your inten is to provd a compatiabel cli with similar output we coudl likely just make that cofnigureabl in the relevent proejct can allow opertors to opt in
Yes, although that could also happen at the distro level. Its all irrelevant if the prototype doesn't "pay out", but my presumption was a drop in replacement that required no / minimal code change to OpenStack was more desirable that something more invasive.
the other approach we had been taking was to build out the image inspector in oslo. with the intent of eventually removing our dependency on qemu-image for doign the safety/validation entirely and only using it if needed for format conversion.
I am aware of the Oslo work. It worries me that its an arms race to be honest. Instar has a portion of its test suite devoted to comparisons with oslo.utils.format_inspector.
so i would need to load context on exactly what you have built but i would not discard it because an experience engineer built it using the assistance of an llm. i have built useful tools, features and bugs-fixes with ai and when used with care it can produce high quality output.
Thank you. The goal is to produce a binary that you can't tell isn't qemu-img, in that for "any" given input you give it you get the same output as you would with qemu-img. Now, that's neither a good idea (some of the qemu-img defaults are in fact unsafe), nor actually 100% feasible (instar wants you to allowlist the filesystem paths backing files can come from), but it is the high level goal. https://shakenfist.com/components/instar/configuration/ is probably the fastest introduction to what it can do now, and https://shakenfist.com/components/instar/quirks/ is an attempt to document all the places where the qemu-img behaviour was deviated from _so_ _far_. The other most notable difference is that because instar runs as a KVM guest, the user running instar _must_ have permissions to read and write to /dev/kvm. Instar _does_ _not_ use qemu or libvirt, so those don't need to be accessible to that user. This also means for now platforms other that x86_64 are _not_ supported because instar only knows how to bootstrap a KVM guest of that architecture.
On the other hand, having done this for about six months, I think the "it's just statistical" view is naive. There is much more to it than that: planning frameworks, clear definitions of correctness, good precommits and CI, etc. This is one of the reasons I have started publishing the planning templates and implementation documents I use, as I think those are just as important in terms of review as including a summary prompt in the commit message. Sadly, instar doesn't have a complete history of those because it was before I realised those things, but given one of my goals was to learn about the tools that seems like a success to me regardless.
Isn't another way to rephrase your stance simply as encouraging authors using ML coding tools not to disclose that because then their code isn't taken seriously? that would be very counter productive and not a stance we shoudl take as a comunity IMO.
llm or not is not the gate of quality human but the maintneres and the authors are :) devaluing the work of contibutors just ebcause the use a tool to help them woudl be counter productive so disculosure is good but shoudl not be a signal that it is of good or bad quality.
I agree, but that is not my interpretation of the current behaviour of all OpenStack community members.
Instar runs thousands of comparison tests between its output and
qemu-img's. There are 121 images in the test suite, including some crafted to specifically trigger previous vulnerabilities. So, as best as I can tell it _works_. It might not be maintainable, but it might also be. I haven't found any evidence yet that it is not. Then again, should maintainability be a primary goal for a _prototype_? Nova has had lots of histocial design missteps we've had to clean up -- surely it's unreasonable to expect machine-assisted code to be perfect at first attempt?
if its optioanl or configurable we can evaulate that with our ci. qemu-img has had many bugs and secuirty issues so human written code is also not perfect.
I don't think instar is ready to be dropped into CI to replace qemu-img -- for example `resize` is not currently supported. That's the goal, but we are some way away from that right now. That said, it does have github CI. That CI is somewhat hampered by my hesitation to publicly share the test data set, but in the end I decided that sharing a collection of images specifically crafted to trigger historic qemu-img bugs was probably not a good idea. Michael
So what might a safer approach look like? My answer is to try and offload the processing of untrusted data to the virtual machine implementation in silicon -- it has strong separation guarantees and it can be orchestrated so that there is simply no access to anything not required for the strict processing of the image. So for example no access to a file system, a network, and so forth. Specifically, what if processing this untrusted and potentially malicious data was done as a KVM guest with a custom virtual machine manager (VMM)?
Isn't this basically libguestfs? I mean, perhaps we can't use libguestfs to actually do all the things we want, but it's basically the same model I think. Last I knew, we tried to avoid doing that as much as possible. Partially for overhead, partially for complexity, etc.
The primary use case I'm targeting is validating untrusted images at upload time (e.g., in Glance), though the same approach could apply anywhere qemu-img is invoked on user-supplied data.
Glance already does this with format inspector, and in a way that doesn't store (or duplicate) all the data - it inspects it mid-stream, and for both upload and import in the same way. It also has tighter restrictions on things we want to disallow than even qemu-img does. It also does it for basically all the formats we support (like vhdx and vmdk) and even supports strict checking of things qemu-img doesn't support (like MBR/GPT). Nova also uses format inspector to double-check what it got from glance, and in some cases what is on-disk before it does stuff with it (like upload). Cinder does as well and I think ironic too?
The current goal of Instar is to be command line compatible with qemu-img (especially `qemu-img info` and `qemu-img convert`), but I also wonder about opportunities to get out of the game of parsing command output. Whilst Instar can certainly emit JSON formatted output, there are other options as well like running it as a persistent daemon which services requests it receives over a unix domain socket. I've played that game with other code recently and find protobufs to be a reasonable protocol for such things. I don't know how OpenStack feels about protobufs however.
I haven't quite grokked if you're really planning a full replacement of qemu-img or a wrapping of it. However, qcow2 has evolved multiple times over the years in ways that we'd need to support for things like format conversions and so a full re-implementation would need to handle all of that, and with the same concerned-about-integrity level of QA that feels like would make that quite a job. The VMDK format is actually a group of formats all loosely related but quite different. Every time we've accidentally or intentionally broken some seemingly obscure VMDK thing, we hear about it from operators so it definitely has to work. And of course, we need LUKS support for the existing cinder and upcoming glance and nova support for that format.
At the moment I'd describe Instar as a "vigorous prototype". It implements `qemu-img info` and most of `qemu-img convert`, but does not yet know how to resize, rebase, snapshot, and so forth. I'd be interested in whether other people think this is an interesting idea or not, although honestly it was just kind of fun to build over the summer holidays. I'd particularly welcome feedback on whether this approach would be useful for your deployment, examples of images that produce unexpected results, and thoughts on the daemon / protobuf interface idea.
If your goal is to be fully command-line compatible and all we need to do is allow telling nova and glance what the binary is then that seems fine to me. Personally, I think we're in a much better spot than we once were in terms of inspecting malicious images. The recent nova redux was a place where we had missed applying the same inspection to one spot, but glance totally caught it before any data could be exfiltrated. If we want to further tighten down our actual use of qemu-img I think we have a lot of room to be able to just quarantine its execution away from all the dangerous bits. --Dan
On Sat, May 16, 2026 at 3:09 AM Dan Smith <dms@danplanet.com> wrote:
So what might a safer approach look like? My answer is to try and offload the processing of untrusted data to the virtual machine implementation in silicon -- it has strong separation guarantees and it can be orchestrated so that there is simply no access to anything not required for the strict processing of the image. So for example no access to a file system, a network, and so forth. Specifically, what if processing this untrusted and potentially malicious data was done as a KVM guest with a custom virtual machine manager (VMM)?
Isn't this basically libguestfs? I mean, perhaps we can't use libguestfs to actually do all the things we want, but it's basically the same model I think. Last I knew, we tried to avoid doing that as much as possible. Partially for overhead, partially for complexity, etc.
I did ponder libguestfs along the way. The big difference, at least to my understanding, is that libguestfs runs a linux kernel in its KVM guest with an associated small operating system, and then uses standard command line tools to service requests. Instar is more like a unikernel than libguestfs in the sense that instar _does_ not run a kernel as we think of it, and has no userspace. Instar is more like having the qemu-img operations performed on a service processor that just happens to be an x86 virtual core if that makes sense. I think that makes the security posture (and complexity) of the two quite different.
The primary use case I'm targeting is validating untrusted images at upload time (e.g., in Glance), though the same approach could apply anywhere qemu-img is invoked on user-supplied data.
Glance already does this with format inspector, and in a way that doesn't store (or duplicate) all the data - it inspects it mid-stream, and for both upload and import in the same way. It also has tighter restrictions on things we want to disallow than even qemu-img does. It also does it for basically all the formats we support (like vhdx and vmdk) and even supports strict checking of things qemu-img doesn't support (like MBR/GPT). Nova also uses format inspector to double-check what it got from glance, and in some cases what is on-disk before it does stuff with it (like upload). Cinder does as well and I think ironic too?
If you discount moving pages around in memory, which I think all inspection tools would have to do, instar does not copy the input data prior to inspection.
The current goal of Instar is to be command line compatible with qemu-img (especially `qemu-img info` and `qemu-img convert`), but I also wonder about opportunities to get out of the game of parsing command output. Whilst Instar can certainly emit JSON formatted output, there are other options as well like running it as a persistent daemon which services requests it receives over a unix domain socket. I've played that game with other code recently and find protobufs to be a reasonable protocol for such things. I don't know how OpenStack feels about protobufs however.
I haven't quite grokked if you're really planning a full replacement of qemu-img or a wrapping of it. However, qcow2 has evolved multiple times over the years in ways that we'd need to support for things like format conversions and so a full re-implementation would need to handle all of that, and with the same concerned-about-integrity level of QA that feels like would make that quite a job. The VMDK format is actually a group of formats all loosely related but quite different. Every time we've accidentally or intentionally broken some seemingly obscure VMDK thing, we hear about it from operators so it definitely has to work. And of course, we need LUKS support for the existing cinder and upcoming glance and nova support for that format.
https://shakenfist.com/components/instar/format-coverage/ already addresses the format coverage question so I'll let it stand on its own two feet. Instar is a replacement, not a wrapper, noting that its also a prototype and therefore incomplete. That said, for the subcommands that are "done", I am unaware of an image or set of command line options that produces output that is different from qemu-img, and an example of such would be treated as a bug. You're right that qemu-img has changed its behaviour over time. To address this at build time, instar builds all qemu-img versions going back to 6.0 and builds a mapping of the various historical output formats. Then at run time instar attempts to detect the version of qemu-img installed on your machine and gives you that output format.
At the moment I'd describe Instar as a "vigorous prototype". It implements `qemu-img info` and most of `qemu-img convert`, but does not yet know how to resize, rebase, snapshot, and so forth. I'd be interested in whether other people think this is an interesting idea or not, although honestly it was just kind of fun to build over the summer holidays. I'd particularly welcome feedback on whether this approach would be useful for your deployment, examples of images that produce unexpected results, and thoughts on the daemon / protobuf interface idea.
If your goal is to be fully command-line compatible and all we need to do is allow telling nova and glance what the binary is then that seems fine to me. Personally, I think we're in a much better spot than we once were in terms of inspecting malicious images. The recent nova redux was a place where we had missed applying the same inspection to one spot, but glance totally caught it before any data could be exfiltrated. If we want to further tighten down our actual use of qemu-img I think we have a lot of room to be able to just quarantine its execution away from all the dangerous bits.
That is my goal for now. Perhaps I'll encounter some way that's impossible. Again, I don't really mind if you guys use it or not -- I just think its interesting. Michael
I did ponder libguestfs along the way. The big difference, at least to my understanding, is that libguestfs runs a linux kernel in its KVM guest with an associated small operating system, and then uses standard command line tools to service requests. Instar is more like a unikernel than libguestfs in the sense that instar _does_ not run a kernel as we think of it, and has no userspace. Instar is more like having the qemu-img operations performed on a service processor that just happens to be an x86 virtual core if that makes sense. I think that makes the security posture (and complexity) of the two quite different.
Okay I guess I'm not sure that's really a mark in favor of "more secure" to me. It certainly /could/ be so, and a massively smaller stack /could/ mean a smaller surface area. If it were something highly cultivated and scrutinized perhaps, but just running something on "bare metal" that was largely output that we didn't review/understand at every level doesn't really ring the bell for me. Definitely interesting and cool though :)
If you discount moving pages around in memory, which I think all inspection tools would have to do, instar does not copy the input data prior to inspection.
I'm not talking about copying pages in memory: what I meant is that glance does not ever need to store the image on disk while it's being uploaded in order to inspect it. If you back glance with swift and upload a qcow2, we never store it on disk first, inspect, and then upload to swift (aside from the import workflow that does that incidentally). We look at the stream _as_ it's being proxied to swift (or rbd, or whatever else) and we can abort the upload after you've send a few KiB, we know it's qcow2 and we know it's bad. --Dan
On 18/05/2026 14:45, Dan Smith wrote:
I did ponder libguestfs along the way. The big difference, at least to my understanding, is that libguestfs runs a linux kernel in its KVM guest with an associated small operating system, and then uses standard command line tools to service requests. Instar is more like a unikernel than libguestfs in the sense that instar _does_ not run a kernel as we think of it, and has no userspace. Instar is more like having the qemu-img operations performed on a service processor that just happens to be an x86 virtual core if that makes sense. I think that makes the security posture (and complexity) of the two quite different. Okay I guess I'm not sure that's really a mark in favor of "more secure" to me. It certainly /could/ be so, and a massively smaller stack /could/ mean a smaller surface area. If it were something highly cultivated and scrutinized perhaps, but just running something on "bare metal" that was largely output that we didn't review/understand at every level doesn't really ring the bell for me. Definitely interesting and cool though :) for what it worth i think the image inspector work in oslo effectivly mitigates much of the securtiy concern of invoking qemu-image or other tools to do things like resize/format conversion as we have effecctivly promoted the image form untrusted to trusted. now its also true that unless we reject images with featuer we do not have knowldage of there is a bit of an arms race between preventing a unsafe feature from being used and updating it but we can ligitiamtly do that
i.e. look at teh image and say your enableing a feature we dont know about and reject that means that any iamge we are passing to qemu-image is know safe and at that point qemu-iamge is nolonger in the critial path to protect the cloud form its users. the oslo.utils approch si never executing anythign its just parsing the bytes of the straem so its much much safer then qemu-image in that regard and its inhernetly portable without a depency on kvm or similar so my current expectation fo how we are eveoling this is not to fined a better way to handel untrusted disk tehn qemu-img its ot make sure we never pass an untrusted disk to qemu-image. dan has been leading much of the work on that and the Glance as the defender work but that context is relevnet to evaulating qemu-image alternitives. the qemu-image maintainer have told us to not rely on it for untursted images so dan and other have been tryign to make sure we have validated the image before we ever do that so its nolonger untrusted. it may still be user supplied but thos are two diffent thigns.
If you discount moving pages around in memory, which I think all inspection tools would have to do, instar does not copy the input data prior to inspection. I'm not talking about copying pages in memory: what I meant is that glance does not ever need to store the image on disk while it's being uploaded in order to inspect it. If you back glance with swift and upload a qcow2, we never store it on disk first, inspect, and then upload to swift (aside from the import workflow that does that incidentally). We look at the stream _as_ it's being proxied to swift (or rbd, or whatever else) and we can abort the upload after you've send a few KiB, we know it's qcow2 and we know it's bad.
yep that a very nice capablity and sideefect of the way the image inspector is stream based. for some format we unfortuently do need to process the stream of the entire disk to determin if its safe but for many formats we only need the first few bytes and we can bail early if we ever detect its invailed without processign the rest.
--Dan
for what it worth i think the image inspector work in oslo effectivly mitigates much of the securtiy concern of invoking qemu-image or other tools to do things like resize/format conversion as we have effecctivly promoted the image form untrusted to trusted.
Agree. Even with it, I'd certainly be happy with a more-secure qemu-img alternative for our usage (or, really, I'd just prefer qemu-img took this more seriously).
now its also true that unless we reject images with featuer we do not have knowldage of
To be clear, we do that today: https://github.com/openstack/oslo.utils/blob/master/oslo_utils/imageutils/fo... Any feature we don't have mapped makes us reject the image. This was the direct recommendation from the qemu team.
there is a bit of an arms race between preventing a unsafe feature from being used and updating it but we can ligitiamtly do that
For an inspector, it's easy to default to "fail on unknown features". If a new one pops up, we can evaluate and bless/ban it and move on. For a reimplementation of qemu-img, it's truly an arms race to keep up with not just identifying but _implementing_ all those features.
yep that a very nice capablity and sideefect of the way the image inspector is stream based. for some format we unfortuently do need to process the stream of the entire disk to determin if its safe but for many formats we only need the first few bytes and we can bail early if we ever detect its invailed without processign the rest.
We have no formats that we need to process the entire stream (currently). Even LUKS is stream-inspectable with decryption to inspect the first block of the encapsulated disk. For glance, the upload process can _never_ store the image to inspect it first (there's literally no place to do it). --Dan
participants (4)
-
Artem Goncharov
-
Dan Smith
-
Michael Still
-
Sean Mooney