[openstack-dev] [nova] Device tagging: rebuild config drive upon instance reboot to refresh metadata on it

Artom Lifshitz alifshit at redhat.com
Mon Feb 20 15:46:09 UTC 2017

I don't think we're trying to re-invent configuration management in
Nova. We have this problem where we want to communicate to the guest,
from the host, a bunch of dynamic metadata that can change throughout
the guest's lifetime. We currently have two possible avenues for this
already in place, and both have problems:

1. The metadata service isn't universally deployed by operators for
security and other reasons.
2. The config drive was never designed for dynamic metadata.

So far in this thread we've mostly been discussing ways to shoehorn a
solution into the config drive avenue, but that's going to be ugly no
matter what because it was never designed for what we're trying to do
in the first place.

Some folks are saying that we admit that the config drive is only for
static information and metadata that is known at boot time, and work
on a third way to communicate dynamic metadata to the guest. I can get
behind that 100%. I like the virtio-vsock option, but that's only
supported by libvirt IIUC. We've got device tagging support in hyper-v
as well, and xenapi hopefully on the way soon [1], so we need
something a bit more universal. How about fixing up the metadata
service to be more deployable, both in terms of security, and IPv6

[1] https://review.openstack.org/#/c/333781/

On Mon, Feb 20, 2017 at 10:35 AM, Clint Byrum <clint at fewbar.com> wrote:
> Excerpts from Jay Pipes's message of 2017-02-20 10:00:06 -0500:
>> On 02/17/2017 02:28 PM, Artom Lifshitz wrote:
>> > Early on in the inception of device role tagging, it was decided that
>> > it's acceptable that the device metadata on the config drive lags
>> > behind the metadata API, as long as it eventually catches up, for
>> > example when the instance is rebooted and we get a chance to
>> > regenerate the config drive.
>> >
>> > So far this hasn't really been a problem because devices could only be
>> > tagged at instance boot time, and the tags never changed. So the
>> > config drive was pretty always much up to date.
>> >
>> > In Pike the tagged device attachment series of patches [1] will
>> > hopefully merge, and we'll be in a situation where device tags can
>> > change during instance uptime, which makes it that much more important
>> > to regenerate the config drive whenever we get a chance.
>> >
>> > However, when the config drive is first generated, some of the
>> > information stored in there is only available at instance boot time
>> > and is not persisted anywhere, as far as I can tell. Specifically, the
>> > injected_files and admin_pass parameters [2] are passed from the API
>> > and are not stored anywhere.
>> >
>> > This creates a problem when we want to regenerated the config drive,
>> > because the information that we're supposed to put in it is no longer
>> > available to us.
>> >
>> > We could start persisting this information in instance_extra, for
>> > example, and pulling it up when the config drive is regenerated. We
>> > could even conceivably hack something to read the metadata files from
>> > the "old" config drive before refreshing them with new information.
>> > However, is that really worth it? I feel like saying "the config drive
>> > is static, deal with it - if you want to up to date metadata, use the
>> > API" is an equally, if not more, valid option.
>> Yeah, config drive should, IMHO, be static, readonly. If you want to
>> change device tags or other configuration data after boot, use a
>> configuration management system or something like etcd watches. I don't
>> think Nova should be responsible for this.
> I tend to agree with you, and I personally wouldn't write apps that need
> this. However, in the interest of understanding the desire to change this,
> I think the scenario is this:
> 1) Servers are booted with {n_tagged_devices} and come up, actions happen
> using automated thing that reads device tags and reacts accordingly.
> 2) A new device is added to the general configuration.
> 3) New servers configure themselves with the new devices automatically. But
> existing servers do not have those device tags in their config drive. In
> order to configure these, one would now have to write a fair amount of
> orchestration to duplicate what already exists for new servers.
> While I'm a big fan of the cattle approach (just delete those old
> servers!) I don't think OpenStack is constrained enough to say that
> this is always going to be efficient. And writing two paths for server
> configuration feels like repeating yourself.
> I don't have a perfect answer to this, but I don't think "just don't
> do that" is sufficient as a response. We allowed the tags in config
> drive. We have to deal with the unintended consequences of that decision.
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Artom Lifshitz

More information about the OpenStack-dev mailing list