Hey y'all, We have an internal use case which requires a VM with a TPM, to be used to store a private key. Libvirt has two ways to present a TPM to a VM: passthrough or emulated. Per kashyap and the #qemu IRC channel, libvirt stores the TPM's state on disk, unencrypted. Our risk profile includes "someone walks away with a disk", so this won't work for our use case. The QEMU devs have asked for RFEs to implement vTPMs where the state never touches the disk, so I have hopes that this will be done eventually. However, I suspect that this will still take some time, especially as nobody has volunteered to actually do the work yet. So, I'd like to propose we implement TPM passthrough in Nova. My team is happy to do the work, but I'd love some guidance as to the best way to implement this so we can get a spec done (I assume it's "just another resource class"?). If Nova doesn't want this feature in, and would rather just wait for the features in QEMU, we'll carry it downstream, I guess. :) Thoughts? // jim
On Thu, Jun 20, 2019 at 10:40 AM Jim Rollenhagen <jim@jimrollenhagen.com> wrote:
Hey y'all,
We have an internal use case which requires a VM with a TPM, to be used to store a private key. Libvirt has two ways to present a TPM to a VM: passthrough or emulated. Per kashyap and the #qemu IRC channel, libvirt stores the TPM's state on disk, unencrypted. Our risk profile includes "someone walks away with a disk", so this won't work for our use case.
The QEMU devs have asked for RFEs to implement vTPMs where the state never touches the disk, so I have hopes that this will be done eventually.
However, I suspect that this will still take some time, especially as nobody has volunteered to actually do the work yet. So, I'd like to propose we implement TPM passthrough in Nova. My team is happy to do the work, but I'd love some guidance as to the best way to implement this so we can get a spec done (I assume it's "just another resource class"?).
https://wiki.qemu.org/Features/TPM Would it be using this? I'm just trying to gauge out what TPM passthrough involves out of personal curiosity.
If Nova doesn't want this feature in, and would rather just wait for the features in QEMU, we'll carry it downstream, I guess. :)
Thoughts?
// jim
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com
// jim On Thu, Jun 20, 2019 at 10:44 AM Mohammed Naser <mnaser@vexxhost.com> wrote:
On Thu, Jun 20, 2019 at 10:40 AM Jim Rollenhagen <jim@jimrollenhagen.com> wrote:
Hey y'all,
We have an internal use case which requires a VM with a TPM, to be used
store a private key. Libvirt has two ways to present a TPM to a VM:
to passthrough
or emulated. Per kashyap and the #qemu IRC channel, libvirt stores the TPM's state on disk, unencrypted. Our risk profile includes "someone walks away with a disk", so this won't work for our use case.
The QEMU devs have asked for RFEs to implement vTPMs where the state never touches the disk, so I have hopes that this will be done eventually.
However, I suspect that this will still take some time, especially as nobody has volunteered to actually do the work yet. So, I'd like to propose we implement TPM passthrough in Nova. My team is happy to do the work, but I'd love some guidance as to the best way to implement this so we can get a spec done (I assume it's "just another resource class"?).
https://wiki.qemu.org/Features/TPM
Would it be using this? I'm just trying to gauge out what TPM passthrough involves out of personal curiosity.
Yes, though I think those notes are from before it was implemented. Here's the libvirt XML to make it work: https://libvirt.org/formatdomain.html#elementsTpm I assume we'd just translate a TPM resource class in the flavor to this XML, but I'm hoping a nova developer can confirm this. :) // jim
Jim-
So, I'd like to propose we implement TPM passthrough in Nova. My team is happy to do the work, but I'd love some guidance as to the best way to implement this so we can get a spec done (I assume it's "just another resource class"?).
And by "just another resource class" you mean: - Add TPM to os-resource-classes (exact name subject to bikeshedding). - Virt driver's update_provider_tree() looks at the guts of the host to figure out how many TPM devices exist and, if nonzero, tacks an inventory of that many TPM onto the root provider (max_unit 1 presumably; all others default). - Flavor desiring this thingy is authored with extra spec resources:TPM=1. - Scheduler lands instance on host with TPM inventory, and allocates one. (This is free, no additional code changes necessary.) - Virt driver's spawn() looks at the allocation, sees TPM:1, and augments the guest's domain XML to attach the thingy. Is it any more complicated than that? I'm fine with this. efried .
On Thu, Jun 20, 2019 at 11:20 AM Eric Fried <openstack@fried.cc> wrote:
Jim-
So, I'd like to propose we implement TPM passthrough in Nova. My team is happy to do the work, but I'd love some guidance as to the best way to implement this so we can get a spec done (I assume it's "just another resource class"?).
And by "just another resource class" you mean:
- Add TPM to os-resource-classes (exact name subject to bikeshedding). - Virt driver's update_provider_tree() looks at the guts of the host to figure out how many TPM devices exist and, if nonzero, tacks an inventory of that many TPM onto the root provider (max_unit 1 presumably; all others default). - Flavor desiring this thingy is authored with extra spec resources:TPM=1. - Scheduler lands instance on host with TPM inventory, and allocates one. (This is free, no additional code changes necessary.) - Virt driver's spawn() looks at the allocation, sees TPM:1, and augments the guest's domain XML to attach the thingy.
Is it any more complicated than that?
That makes sense to me. I don't know these bits well enough to comment if there's anything else to do. Maybe choosing the correct /dev/tpmN may get weird?
I'm fine with this.
Cool, will attempt to get a spec going, unless violent opposition shows up in this thread in the meantime. Thanks! // jim
That makes sense to me. I don't know these bits well enough to comment if there's anything else to do. Maybe choosing the correct /dev/tpmN may get weird?
If a) they're all the same; and b) you can tell which ones are in use, then I don't see an issue.
Cool, will attempt to get a spec going, unless violent opposition shows up in this thread in the meantime.
I want to say you could probably get away with a specless blueprint, since we've been able to describe the problem and solution in like two dozen lines. Perhaps you want to start there and see if anyone complains. efried .
On 6/20/2019 12:32 PM, Eric Fried wrote:
I want to say you could probably get away with a specless blueprint, since we've been able to describe the problem and solution in like two dozen lines. Perhaps you want to start there and see if anyone complains.
As Resident Complainer I feel compelled to say there should at least be a small spec (as I said in IRC). The question in this thread about how to keep track of and know which devices are allocated is a good one that is going to require some thought. Also the thing not mentioned here (and usually not thought about early on) is move operations and how those will be handled - what will and will not be supported when you have a TPM passthrough device on a guest and will there be any issues, e.g. resizing to/from a flavor with one of these. If using a resource class for tracking inventory then scheduling should be straight-forward, i.e. migration shouldn't pick a host that can't support the flavor. -- Thanks, Matt
On 6/20/2019 1:05 PM, Matt Riedemann wrote:
As Resident Complainer I feel compelled to say there should at least be a small spec (as I said in IRC). The question in this thread about how to keep track of and know which devices are allocated is a good one that is going to require some thought.
It's probably also worth thinking about what happens when evacuating a sever from a host with one of these and/or deleting a server when the compute service on which it was running is down. Upon restart of the compute service we'll need to detect the guest is gone and cleanup the previously allocated devices somehow (I think this has been a problem for PCI device allocations in the past as well). -- Thanks, Matt
Folks- I've filed a blueprint [1] and started on a spec [2]. The latter needs love from someone who understands the low-level aspects better -- I left a bunch of todos. It would be nice to have this ready for next Tuesday [3] since we're getting pretty close to spec freeze. I also started some (very rough) code [4] to show how this could work. efried [1] https://blueprints.launchpad.net/nova/+spec/physical-tpm-passthrough [2] https://review.opendev.org/667926 [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007381.html [4] https://review.opendev.org/667928
On Thu, Jun 20, 2019 at 10:34:24AM -0400, Jim Rollenhagen wrote:
Hey y'all,
We have an internal use case which requires a VM with a TPM, to be used to store a private key. Libvirt has two ways to present a TPM to a VM: passthrough or emulated. Per kashyap and the #qemu IRC channel, libvirt stores the TPM's state on disk, unencrypted. Our risk profile includes "someone walks away with a disk", so this won't work for our use case.
The QEMU devs have asked for RFEs to implement vTPMs where the state never touches the disk, so I have hopes that this will be done eventually.
I haven't gotten around to file the two things requested for this feature for TPM passthrough. (At least for libvirt, anyone with an upstream Bugzilla account can file the RFE in its tracker[1].) (*) File a libvirt RFE to make it not store encryption keys on disk (currently it stores the TPM state under: /var/lib/libvirt/swtpm/). The libvirt/QEMU folks are still debating the design. It is likely to be similar to how libvirt handles the keys for LUKS encryption. (*) Daniel Berrangé suggested that we might also need an RFE for "swtpm" project[2] so as to allow libvirt / QEMU pass keys around without them being on disk. Currently `swtpm` has: --key file=<keyfile>[,format=<hex|binary>][,mode=aes-cbc|aes-256-cbc],[remove[=true|false]] E.g. to have a parameter that allows file descriptor passing: "--key fd=NN" - - - And for Nova, I agree, a "lightweight" spec would be useful to flesh out details. Based on the IRC chat on #qemu (OFTC), there's a subtle detail that Nova needs to pay attention to: as the key file would still pass through libvirt, the "mgmt app" (e.g. Nova) can control how long it is around for. Daniel suggested "mgmt app" can purge the keys from libvirt once the guest is running, and can tell libvirt to keep it only in memory. [1] https://libvirt.org/bugs.html [2] https://github.com/stefanberger/swtpm/wiki [...] -- /kashyap
participants (5)
-
Eric Fried
-
Jim Rollenhagen
-
Kashyap Chamarthy
-
Matt Riedemann
-
Mohammed Naser