[openstack-dev] [trove][sahara][infra][Octavia][manila] discussion of image building in Trove

Gregory Haynes greg at greghaynes.net
Wed May 4 18:36:04 UTC 2016

On Wed, May 4, 2016, at 09:57 AM, Ethan Gafford wrote:
> Sahara has support for several image generation-related cases:
> 1) Packing an image pre-cluster spawn in Nova.
> 2) Building clusters from a "clean" OS image post-Nova spawn, by
>    downloading and installing Hadoop/Spark/Storm/WhatHaveYou.
> 3) Validating images after Nova spawn (to ensure that they're up to
>    spec for the plugin in question.)
> Because of this, we are pulling image generation logic into the
> plugins themselves, from a monolithic sahara-image-elements
> repository. This will aid us in our eventual hope to pull plugins
> out of the Sahara tree entirely; more immediately, it will allow us
> to define logic for all of these cases in one place, written
> eventually in Python. In our Sahara session at summit, which was
> also attended by several members of the Ironic team (thanks all), we
> discussed our current plan to use libguestfs rather than DIB as our
> toolchain; our intent to define a linear, idempotent set of steps to
> pack images for any plugin lends itself much more neatly to
> libguestfs' API than to DIB's.
Ok, there is clearly some kind of communication breakdown between DIB
and some of its users that we *really* need to solve. AFAIK the primary
set of DIB developers had no idea this was going on. I am going to push
up a patch to our docs to make it a lot more clear where to find us
(#tripleo or the ML). I'm not really sure what else we could do to make
it easier for users to find us / raise issues with us so we can explore
ways to solve them, but any suggestions would be a huge help.
The idea that a linear set of idempotent steps is mutually exclusive
with DIB's API is really interesting. IIUC this is something we
grappled with in TripleO when creating the tool and simply never spent
time looking in to solving for the existing distro elements, although
that is an element issue not an API issue. DIB's API is just a linear
set of commands you opt in to (by way of elements), if you make those
commands idempotent then you have what you want, the problem is that
the upstream element's themselves are not written in that way so you
couldn't depend on them.  I think there are some pretty obvious ways to
solve this, potentially by making an element which provides the same
API as libguestfs virt-customize (this would be very simple to do, I
just haven't heard of anyone wanting it) or by hacking on some of the
in-tree elements. Could you go in to some more detail on how you use
this feature?
> Beyond that, the Sahara team has certainly seen profound difficulty in
> the field when customers attempt to generate their own images, and
> even for Sahara cores, building images is occasionally quite
> harrowing. These issues are seldom based on real issues in the scripts
> themselves, but are frequently the result of bleed between the host
> and guest; when these issues occur for a customer, they become
> extremely difficult to diagnose remotely. Still, it's entirely
> possible that DIB has answers to these problems, and it'd be a
> universal good to identify real flaws in DIB, or just to educate the
> uninitiated into how DIB can be made to work more cleanly if the
> features are already there (which they may well be; far be it from me
> to claim exhaustive mastery of DIB.) The technical reasons we like
> libguestfs over DIB are:
It would be *extremely* helpful to know about these issues when they
come up. There's definitely some things we can do to prevent this but
again, there hasn't been a lot of feedback that this is an issue people
are running in to. There's a few different ways to prevent these types
of problems ranging from doing something like virt-dib to having dib use
another chroot during the image building process but I would really like
to hear about some specific issues to get a better idea of what the root
cause is. Hopefully we could even get these filed as bugs.
> 1) Libguestfs manipulates images in a clean helper VM created by
>    libguestfs in a predictable way. As such, no mounts are made on the
>    host, no scripts can affect the host system, and no variables on
>    the host system are likely to affect the image packing process. See
>    http://libguestfs.org/guestfs-security.1.html for information on
>    libguestfs security.
Yep, isolation is definitely something DIB currently gives up in order
to provide speed/lower resource usage. That being said, there are lots
of things that could be done to either mitigate these issues or remove
them altogether (by compromising on speed/resource requirements). My
biggest concern is that we are creating two sets of code to perform the
same task due to a false dichotomy - the obvious case being that at a
minimum we could still be using something compatible with DIB while
utilizing a vm/libguestfs. This would then have the benefits of having a
common toolset among the community without the raised issues.
> 2) In-place image manipulation means that relatively minor changes to
>    images (package installs, conf declaration, etc.) can occur
>    swiftly, and subsequent changes can occur without uncompressing and
>    recompressing an entire image.
This isn't an issue - its pretty trivial to make a dib element which
modifies an existing image, and making it in-place versus outputting a
new image is even more trivial. The ubuntu distro element effectively
works this way, even - it is based off a cloud-image and then performs
modifications to it.
> 3) DIB scripts' configuration settings, passed in freeform environment
>    variables, can be difficult to understand and document for new
>    users. Libguestfs' API demands more formal parameter passing,
>    reducing the likelihood of accidental misconfiguration.
Back when there was a lot more development effort going in to DIB this
was pretty high on the list of things to solve. There is even a spec
somewhere to this effect. I simply haven't heard this issue raised much
recently otherwise I would be pushing more for it to be solved since its
really a matter of engineering effort at this point. Getting feedback
like this is *extremely* helpful though, and again, any specific use
cases / instances where this has caused issues would be even better.
> I'm heartened that we're having this discussion: user-facing image
> packing tools are a critical part of any application-tier provisioning
> service, and cleaning our upstream image gen processes will be even
> more crucial as these projects mature and their support matrices grow.
> Cheers,
> -Egafford
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160504/71f51943/attachment.html>

More information about the OpenStack-dev mailing list