<html><body>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="33"><img width="1" height="1" src="cid:1__=0ABBF1F9DFD8CBE68f9e8a93df938@us.ibm.com" border="0" alt=""></td><td width="331" valign="middle">
<ul style="padding-left: 0pt"><font size="3" face="serif">injection thread on openstack-dev<br>
Jarrod Johnson <br>
to:<br>
Jarrod B Johnson<br>
05/12/2013 10:17 AM<br>
Hide Details <br>
From: Jarrod Johnson <jarrod.b.johnson@gmail.com><br>
To: Jarrod B Johnson/Raleigh/IBM@IBMUS<br>
Default custom expiration date: 05/12/2014</font></ul>
</td></tr>
</table>
<br>
<font size="3" face="serif">Upon further consideration, I realized I wasn't thinking enough about the status quo for virtualization and also conflating a few things. As such, I'd like to correct, refine, and clarify my thoughts. </font><br>
<br>
<font size="3" face="serif">I am going to presume that virtualization guests will continue to use a block-oriented strategy for boot volumes. This is by far the most ubiquitous strategy. In my experience container solutions are not done this way and I have also helped at least one enterprise implement an nfsroot option for tenants of their private cloud for their virtualization platform. I appreciate the philosophy that fs on block on fs on block that is the norm is an awkward reality, but I'm going to ignore that viewpoint for simplicity.</font><br>
<br>
<font size="3" face="serif">I'm also going to ignore scripted install case as that flow is pretty much straightforward and the same regardless of a lot of circumstances. I also posit that for Xen and ESXi, scripted install makes the most sense. I will confess to inadequate knowledge of the BSDs and Solaris.</font><br>
<br>
<font size="3" face="serif">On the baremetal side, I'm going to assume some sort of BMC. Conceptually, the relationship of baremetal to its BMC and a managed switch (or SDN enabled switch) can provide something analogous to a guest to its hypervisor for purposes of this discussion.</font><br>
<br>
<font size="3" face="serif">So images that apply to baremetal and virtual machines. I will consider the flow in three steps:</font><br>
<br>
<font size="3" face="serif">Getting a new copy of the image into the right place. For Virtualization and SAN boot, this is usually done outside the instance. Whether its CoW or full copy doesn't matter for this discussion. For direct attached storage, it requires that the environment be booted into a deployment environment If the payload is block oriented, that deployment platform need know nothing about the contents of the payload. However, for consistency with the next phase, I suggest this environment should be paired with the class of OS being deployed. For cases requiring a deployment environment, file-oriented payload makes the most sense, but for virtualization and SAN boot it almost never makes sense. It is less awkward for direct attached storage to do block imaging than it is for virtualized and SAN instances to do file-oriented, so block oriented I suppose wins if you pick just one. </font><br>
<br>
<font size="3" face="serif">Fixing up the instance to be bootable. This would include driver injection. For UEFI, it can also mean setting boot variables (some do not opt to create bootx64.efi, and even if they do it might not always be easy for the platform to find the correct block device, which is a long story). Here, you must be booting an environment that understands the deployed target enough to perform those activities, which strongly suggests WinPE for Windows and Linux for linux distros. Note this can be of import for situations that do not require the in-instance deployment (boot form SAN and Virtual guests) and as such the environment should be capable of being invoked for driver injection on an already cloned set of contents. For example, if someone crafted an image based on vmxnet3 and vmscsi, this environment would be helpful to modify the instance to run with virtio storage instead under KVM.</font><br>
<br>
<font size="3" face="serif">Finally, we come to the more well recognized step: establishing instance unique configuration and credentials. This is the one step that requires the most cooperation from the image contents. If there is disparity, there is a high chance an image author fails to test for one or the other scheme, so ideally this looks as similar as possilbe. I that the extensible bulk of it is best done with some sort of SSL protected network scheme (probably https). It is the bootstrap of that process that warrants more consideration. </font><br>
<br>
<font size="3" face="serif">I consider a reasonable bootstrap payload to be:</font><br>
<font size="3" face="serif">current Date and time (certificates are in play after all)</font><br>
<font size="3" face="serif">Network configuration</font><br>
<font size="3" face="serif">Instance identifier information (whatever you'd want to put in a CSR)</font><br>
<font size="3" face="serif">The singular CA certificate the subsequent setup process should trust (don't trust public CAs when you don't need to).</font><br>
<font size="3" face="serif">URL(s) for the instance to check into.</font><br>
<font size="3" face="serif">Either:</font><br>
<font size="3" face="serif">A certificate and private key</font><br>
<font size="3" face="serif">Or (preferably in my mind)</font><br>
<font size="3" face="serif">System can generate private key and CSR and push it up the same secure mechanism the payload come down. Infrastructure private CA auto-signs CSRs if they match what was injected.</font><br>
<br>
<font size="3" face="serif">Everything else could be subsequently handled by less restricted facilities, meaning this only has to handle a few kilobytes of well defined data.</font><br>
<br>
<font size="3" face="serif">The mechanisms I can think of include:</font><br>
<font size="3" face="serif">Boot config/ramfs injection:</font><br>
<font size="3" face="serif">This really only applies to the relation between the infrastructure and the deployment/injection environment. Not directly to the image. In environments without either case being applicable, this isn't critical. This would be a step on the road to file injection for baremetal.</font><br>
<br>
<br>
<font size="3" face="serif">File injection:</font><br>
<font size="3" face="serif">For environments that incur the driver injection phase, file injection is trivial, but otherwise is highly awkward. Requires private key be injected.</font><br>
<br>
<font size="3" face="serif">Partition injection:</font><br>
<font size="3" face="serif">Trivial for scenarios that use either in-instance deployment or injection, awkward otherwise. Requires private key be injected.</font><br>
<br>
<font size="3" face="serif">Attaching a config drive:</font><br>
<font size="3" face="serif">Trivial for virtualization possible on select server hardware. As it stands, this is not a ubiquitous feature of server hardware, and also it is implemented in proprietary ways. Requires private key be injected.</font><br>
<br>
<font size="3" face="serif">Bridge-Filtered Protocol:</font><br>
<font size="3" face="serif">Possible today, but requires very peculiar use of protocols that I think would be unpopular. Also would not be durable in the face of some virtualization networking strategies. Can facilitate either private key strategy.</font><br>
<br>
<font size="3" face="serif">Having a special purpose dedicated nic with a recognizable fingerprint (e.g. a certain mac):</font><br>
<font size="3" face="serif">Easy in virtualization, possible on select server hardware if the vendors bother to do it. Not possible on existing firmware in capable hardware and impossible on most server hardware designs. Can facilitate either private key strategy.</font><br>
<br>
<font size="3" face="serif">Having a special behavior to the networking that assures safety of a layer 3 protocol:</font><br>
<font size="3" face="serif">I think this strategy is highly risky. Possible, sure, but highly error prone. Likely unable to do the key publish strategy securely.</font><br>
<br>
<font size="3" face="serif">Serial port:</font><br>
<font size="3" face="serif">I personally think this would be the winner. In KVM the hypervisor can host the security bootstrap agent or have it hosted elsewhere. In server space, SOL over IPMI 2.0 is bog standard and the bootstrap agent can reside wherever. I think for this to not look like a horrible beast, it requires instrumenting the SOL console handler, which could come cheap if I provide the native Python IPMI support I'm considering doing. VMware environments also provide serial port redirection which would require instrumenting it off-hypervisor.</font></body></html>