Open Stack

Hi

diskimage-builder has been working on a feature/v2 branch
incorporating some largely internal changes to the way it finds and
calls elements, enhancements to partitioning and removal of some
long-deprecated elements.  We have just tagged 2.0.0rc1 and are
requesting testing by interested parties.

You can view the branch at

 https://git.openstack.org/cgit/openstack/diskimage-builder/log/?h=feature/v2

For any issues it is probably best to file a bug.  Developers are
standing by in #openstack-dib.

=== User facing changes

If you use dib exclusively via the command-line disk-image-create
installed from a package or via pypi you are unlikely to notice any
difference (if you run it directly from a git-tree checkout, you're in
development territory so read on).

v2 includes a new framework for partitioning contributed by Andreas
Florath.  This should allow for creating multiple partitions, images
with encryption, LVM support and flexibility for multiple-devices, all
of which are currently not supported.  Please check the v2
documentation, specs and reach out if these features interest you
(some parts still in review).

Element override is now supported.  If you have an element of the same
name earlier in the ELEMENTS_PATH, it will override later instances
(previously, the behaviour was undefined).

A number of long-deprecated elements have been removed in v2, which
are to the best of our knowledge unused.

 - partitioning-sfdisk
 - deploy-ironic-element
 - ironc-discovered-ramdisk
 - serial-console-element
 - map-services

=== Developer changes

Developers, or those that are doing more "interesting" things in
deployment or CI, might like to read below to follow the history of
the branch and hopefully do some testing with the feature/v2 branch
ASAP so we can address any concerns.

dib started out mostly as a shell script collection that used setup.py
to install itself.  Over time it has made more and more use of Python.
The current instructions say to checkout the source tree and run
"./bin/disk-image-create"; this causes a number of issues for the
internal Python components.  We carry hacks to try and figure out of
we're being called uninstalled from a source tree, or installed in pip
develop mode, or installed on the system.

For purposes of both users and development we want dib to be as
"pythonic" as possible and behave like all other projects.  Two major
visible changes are:

 - command-line scripts are entry points (i.e. need to be installed)
 - elements have moved under diskimage_create module

The result of the first is that "./bin/disk-image-create" from the
source tree is no longer there.  Like all other projects, you should
install dib into a virtualenv (if you're developing, use pip -e) and
"disk-image-create" will "just work".  FYI, on the back-end, this
entry-point essentially exec()s the old shell-script driver; but we
reserve the right to move more of the logic into python over time.
Also, instead of having to os.system(/path/disk-image-create), we can
allow direct calls like a real library.

A side-effect of this is that we have removed and deprecated the
dib-utils package.  This was intended to be a more generic repository
of tools that might be useful outside dib, but that did not eventuate
and it has been folded back into dib for simplicity.

The second change, moving the inbuilt elements under the
diskimage_create module, is a simplification so we always have a
canonical path to our elements.  Currently the elements are installed
via "data_files" in setup.cfg, which means setup.py puts the elements
in /usr/share... when installing, or leaves them in the tree when
installed via pip -e.  Again we have path guessing hacks trying to
guess what situation we're in.  Elements have moved from elements/ to
diskimage_builder/elements in the source tree (we did try to minimise
this with symlinks, but setuptools doesn't like it, it complicates
things when git merging across the branches and is legacy cruft to go
wrong in the future.  It was better to make a clean break).

Since we always know where elements are relative to the imported
diskimage_builder module we can drop all the path guessing complexity.
This has other good flow-on effects such as testr being able to find
unit-tests for elements in the normal fashion and having imports work
as usual.

We are aware there are a number of tools that like to take dib
elements and do things with them.  This is fine, I guess, although of
course if we don't know about how you use elements we're liable to
break them.  Thus your use may be confused that the elements have
moved.

Reading some of the dib source you may find there is a canonical way
to find out the included dib elements path -- ask dib itself,
something like:

 DIB_ELEMENTS=$(python -c '
 import diskimage_builder.paths;
 diskimage_builder.paths.show_path("elements")')

Note you probably do not want this.  As mentioned, another feature of
v2 is override elements -- an element that appears first in the
element path-list will override any built-in one (just like $PATH).
There is a function, diskimage_builder.get_elements(), which will
correctly process the element path, calculate overrides and return a
canonical list of elements, their dependencies and correct paths.

THAT said, you probably do not want this either!  There are a number
of elements that do things on behalf of other elements -- they look
for a file in the included elements, say, and use that as a manifest
for something.  Previously, these would just have to make up their own
element processing via inspection of the command-line arguments.  dib
now exports pre-computed variables that an element can walk for all
the current build elements -- a YAML list for easy python decoding and
a function that builds an array for Bash elements.

=== Merge plans

Barring any new developments, I would think that 1.28.0 would be the
last release before the v2 branch is merged back to master.  I imagine
this would be in about one to two weeks.

The future is obviously dependent on what contributions appear.  It
seems the main area of interest currently is in creation of
containers; if this interests you keep an eye on the review queue.

Thanks,

-i