Open Stack

On Tue, 21 Jun 2011, Thorsten von Eicken wrote:

> Comments inline below.
>
> On 6/20/2011 1:40 PM, Scott Moser wrote:
> > On Fri, 17 Jun 2011, Thorsten von Eicken wrote:
> >> We're very much looking forward to the new "portable configuration
> >> drive" functionality and would like to provide feedback. If this is not
> >> the best forum, please point me to it.
> >>
> >> The blueprint
> >> is: https://blueprints.launchpad.net/nova/+spec/configuration-drive
> >> We reviewed the initial work
> >> in: https://github.com/ChristopherMacGown/nova/commit/47c041a642ff32085b3140475d7a2a62dcb62c1a
> >>
> >> Feedback:
> > This spec is very similar to implementing what is "OVF Transport".  So
> > similar in fact that I think it should/could be made compatible.
> > Read more at
> > http://www.dmtf.org/standards/published_documents/DSP0243_1.1.0.pdf .
> Sounds like pretty much the same thing. I wouldn't make it identical in
> the naming of the file, though. Also, the ability to change the contents
> of the configuration drive needs to be defined carefully, for example,
> when stopping and restarting the VM after configuration changes.

If you don't like the name, thats fine, thats why I suggested it be made
configurable.  Additionally, I want the user to bypass openstack
completely, and give what looks like garbage to it, but the VM will
understand.

This is something Amazon did right in their user data.  You can give
whatever you want in it, and it can be obtained from inside the instance
bit-for-bit.  The only real restriction is a 16k limit (which I would hope
is not duplicated here).

I'm hoping that this "config drive" can be used in the same way.  I'm not
asking you to implement OVF ISO transport, I'm asking you to let the
entity that launches the instance decide what exactly the contents are.

> >> 1. It is not always obvious how to enumerate attached devices and the
> >> document doesn't explicitly cover how we'd determine which attached
> >> device represents the configuration volume. Specifically consider
> >> Windows as well.
> > I am not very knowledgeable about windows.  However, I'd think as long as
> > the volume doesn't end up being the "boot volume" ("C:") or whatever, then
> > you do not need to worry.  The data on the disk should be formatted in
> > such a way that it is identifiable.  That can be done in a number of ways:
> >  * filesystem label (iso9660 and even i think FAT have a label)
> >  * contents of the filesystem
> >
> > The guest OS can be expected to go searching around for its configuration
> > data.
> We do worry, because we do have to implement it, and we've been burned
> in the past with this stuff! I appreciate your "number of ways" but we'd
> much rather have one well=documented way that we can rely on. Our point
> here is that this should be thought out and specified in a non-ambiguous
> way. That's all.

My suggestion really is to keep openstack providing dynamically
provisioned hardware (and disk contents).  As you've been burned in the
past, so have others.  Very simply, you can *not* come up with a solution
that will will be easy for all current and future operating systems.

By letting the entity launching the instance indicate what the contents of
the disk are, your only restriction on future guests is that they can read
the disk.

> >> 2. Will the configuration drive be available at boot time on a new
> >> instance? Or does it appear "later" (that would be bad)?
> > It needs to be available immediately on first boot. It is very difficult
> > to figure out that the guest is done with the volume, so it should remain
> > attached until the user did some explicit action to detach it.  I really
> > like the suggestion in the etherpad that it is a "normal volume", and then
> > it can be detached via api call.
> I'll look into the etherpad, but detaching via an API call rom the
> instance is not something generally attractive because it requires
> having the account's credentials on the instance, which most security
> conscious users should avoid as much as possible.

It does not need to be made from inside the instance.  The entity that
launched the instance can make a subsequent API call to detach the volume
after determining that it has been consumed.

It *can* be be done from inside the instance, and the need for temporary
credentials that can be utilized from inside the instance (and even
possibly passed in via "configuration data") is something that openstack
is going to have to address at some point in the future.  Amazon does this
fairly well with Identity Access Management.

I'm fairly sure there was a spec on this discussed at the summit, but some
quick googling did not find it.

> >> 3. How does one send the configuration drive content?  What is the API
> >> call where we provide the configuration information and what is the
> >> expected format?
> > In another message on this thread, Christopher said :
> >>> a small ~64MB ISO formatted volume that would be attached and mounted
> >>> read-only by the guest instance on boot
> > I think we can greatly increase the functionality of this feature by
> > removing restrictions.   The essence of this feature is that you're
> > allowing the user to provide you with some data that you'll then put on a
> > volume, attach the volume to the guest, and boot the guest.
> >
> > That can all be done today within AWS with a utility instance, by doing:
> >  * launch utility instance
> >  * attach volume to utility instance
> >  * populate volume via 'mkisofs' or anything else inside the instance
> >  * snapshot volume
> >  * [ delete original volume ]
> >  * launch instance with --block-device-mapping="....snap-abcdefg"
> >
> > So, to me, the only value of this spec is being able to do that list of
> > things above without the utility instance.
> While I agree with you and think that what you propose is nice, there is
> a different aspect that you're perhaps missing, which is simplicity of
> implementation. There's a huge implementation difference if the host
> gets handed a small amount of data (say <1MB), drops it into a tempfs or
> loopback-fs, and makes that available as drive to the guest, vs.
> creating a mountable volume in the cloud that could be attached to any
> VM and then mounting that across the network at boot time. Maybe I'm
> misunderstanding something.

volumes created at boot-time from snapshots is a requirement for
"boot-from-volume" (http://wiki.openstack.org/BootFromVolume), so
utilizing that function make sense to me.

I agree that there is a difference in difficulty of implementation between
a "quick fix" and a more generic fix.  I believe what I'm suggesting is a
generic fix that will address longer term needs of openstack, and do so in
a fairly clean way.

> >> 4. It looks like the configuration content is placed into a config.json
> >> file, is this correct? Is that a blob passed through the API? Does it
> >> have to be json or is that just a convention? What is the max size?
> > Please, a blob. Build in as few expectations on that data as possible.
> Agreed.
> >
> >> 7. We need to be able to change the configuration content when
> >> stopping&starting instances, will this be possible?
> > If you use "normal volumes" for this, then with function in AWS, you could
> > just stop the instance, attach it to another instance, modify, start....
> >
> > More generically, if these are generic volumes, you could provide a
> > 'volumeId' input to 'PopulateVolume' (or any of the other suggested
> > functions above , and if the volume was not attached the hypervisor would
> > make that volume have the specified data.
> In principle yes, but the cost of doing all this is very high when all
> we need is to pass 1-2KB of data into each and every guest...

This last suggestion was the most far out.  I think the rest of them are
extremely generic.