Open Stack

On Fri, Jan 6, 2017, at 09:57 AM, Paul Belanger wrote:
> On Fri, Jan 06, 2017 at 09:48:31AM +0100, Andre Florath wrote:
> > Hello Paul,
> > 
> > thank you very much for your contribution - it is very appreciated.
> > 

Seconded - I'm very excited for some effort to be put in to improving
the use case of making containers with DIB. Thanks :).

> > You addressed a topic with your patch set that was IMHO not in a wide
> > focus: generating images for containers.  The ideas in the patches are
> > good and should be implemented.
> > 
> > Nevertheless I'm missing the concept behind your patches. What I saw
> > are a couple of (independent?) patches - and it looks that there is
> > one 'big goal' - but I did not really get it.  My proposal is (as it
> > is done for other bigger changes or introducing new concepts) that
> > you write a spec for this first [1].  That would help other people
> > (see e.g. Matthew) to use the same blueprint also for other
> > distributions.

I strongly agree with the point that this is something were going to end
up repeating across many distros so we should make sure there's some
common patterns for doing so. A spec seems fine to me, but ideally the
end result involves some developer documentation. A spec is probably a
good place to get started on getting some consensus which we can turn in
to the dev docs.

> Sure, I can write a spec if needed but the TL;DR is:
> 
> Use diskimage-builder to build debootstrap --variant=minbase chroot, and
> nothing
> else. So I can then use take the generated tarball and do something else
> with
> it.
> 
> > One possibility would be to classify different element sets and define
> > the dependency between them.  E.g. to have a element class 'container'
> > which can be referenced by other classes, but is not able to reference
> > these (e.g. VM or hardware specific things).
> > 

It sounds like we need to step back a bit get a clear idea of how were
going to manage the full use case matrix of distro * (minimal / full) *
(container / vm / baremetal), which is something that would be nice to
get consensus on in a spec. This is something that keeps tripping up
both users and devs and I think adding containers to the matrix is sort
of a tipping point in terms of complexity so again, some docs after
figuring out our plan would be *awesome*.

Currently we have distro-minimal elements which are minimal
vm/baremetal, and distro elements which actually are full vm/baremetal
elements. I assume by adding an element class you mean add a set of
distro-container elements? If so, I worry that we might be falling in to
a common dib antipattern of making distro-specific elements. I have a
alternate proposal:

Lets make two elements: kernel, and minimal-userspace which,
respectively, install the kernel package and a minimal set of userspace
packages for dib to function (e.g. dependencies for dib-run-parts,
package-installs). The kernel package should be doable as basically a
package-installs and a pkg-map. The minimal-userspace element gets
tricky because it needs to install deps which are required for things
like package-installs to function (which is why the various distro
elements do this independently).  Even so, I think it would be nice to
take care of installing these from within the chroot rather than from
outside (see https://review.openstack.org/#/c/392253/ for a good reason
why). If we do this then the minimal-userspace element can have some
common logic to enter the chroot as part of root.d and then install the
needed deps.

The end result of this would be we have distro-minimal which depends on
kernel, minimal-userspace, and yum/debootstrap to build a vm/baremetal
capable image. We could also create a distro-container element which
only depends on minimal-userspace and yum/debootstrap and creates a
minimal container. The point being - the top level -container or
-minimal elements are basically convenience elements for exporting a few
vars and pulling in the proper elements at this point and the
elements/code are broken down by the functionality they provide rather
than use case.

> > There are additional two major points:
> > 
> > * IMHO you addressed only some elements that needs adaptions to be
> >   able to used in containers.  One element I stumbled over yesterday
> >   is the base element: it is always included until you explicitly
> >   exclude it.  This base element depends on a complete init-system -
> >   which is for a container unneeded overhead. [2]

I think you're on the right track with removing base - we had consensus
a while back that it should go away but we never got around to it. The
big issue is going to be preserving backwards compat or making it part
of a major version bump and not too painful to upgrade to. I think we
can have this convo on the patchset, though.

> 
> Correct, for this I simply pass the -n flag to disk-image-create. This
> removes
> the need for include the base element. If we want to make a future
> optimization
> to remove or keep, I am okay with that. But the main goal for me is to
> include
> the new ubuntu-rootfs element with minimal disruption as possible.
> > 
> > * Your patches add a lot complexity and code duplication.
> >   This is not the way it should be (see [3], p 110, p 345).
> The main reason this was done, is yes there is some code duplication, but
> the
> because, this is done in the root.d phase.  Moving this logic into
> another
> phase, then requires the need to install python into chroot, and then
> dpkg,
> dib-python, package-install, etc. This basically contaminants the
> pristine
> debootstrap environment, something I am trying hard not to do. I figure,
> 2 lines
> to delete stale data is fine.  However, if there is an objection, we can
> remove
> it.  Keep in mind, by deleting the cache we get the tarball size to 42Mb
> (down
> from 79Mb).

I think my above proposal about a common userspace and kernel element
would solve most of the duplication issues. I am unclear about the not
wanting python and some other elements as part of the build. Currently
python as part of the target image is something we require for a large
portion of the dib functionality. It is possible to not have it as part
of the target image, but there's a significant cost to doing so. As
such, I'd like to know what the motivation is for this? Is it purely a
size reason, and if so, what sizes are we talking about for including
python?

> 
> >   One reason is, that you do everything twice: once for Debian and
> >   once for Ubuntu - and both in a (slightly) different way.
> Yes, sadly the debian elements came along after the ubuntu-minimal
> elements,
> with different people writing the code. For the most part, I've been
> trying to
> condense the code path between the 2, but we are slowly getting there.
> 
> As you can see, the debian-rootfs element does now work correctly[6]
> based on
> previous patches in the stack.
> 
> However, I don't believe this is the stack to make things better between
> the 2
> flavors. We can use the existing ubuntu-minimal and debian-minimal
> elements and
> iterate atop of that.  One next steps is to address how we handle the
> sources.list file, between ubuntu and debian we do things differently.
> 
> [6] https://review.openstack.org/#/c/414765/
> 
> >   Please: factor out common code.
> >   Please: improve code as you touch it.
> > 
> > And three minor:
> > 
> > * Release notes are missing (reno is your friend)
> > 
> Sure, I can add release notes.
> 
> > * Please do not introduce code which 'later on' can / should / will be
> >   cleaned up.  Do it correct right from the beginning. [4]
> > 
> I can rebase code if needed.
> 
> > * It looks that this is a bigger patch set - so maybe we should
> >   include it in v2?
> > 
> I'm not sure we need to wait for v2 (but I am biased).  I've recently
> revamped
> our testing infra for diskimage-builder. We now build images, along with
> launching them with nodepool and SSHing into them.
> 
> Side note, when is v2 landing?  I know there has been issues with
> tripleo.
> 

We basically have the one patch (new block-device.d system) which needs
to land before we can cut an RC. As a result I would prefer to not add
things to v2 (so we can get on with the process of getting it released).
This patch is blocking on a final +A and then someone doing a merge of
master in to the v2 branch before the +A. I plan on doing this once I'm
back from holiday (1/11) if someone hasn't done so by then. Once this
merges the rough plan is to cut an RC, mail out to the list asking folks
for feedback, and cycle on that until we feel comfortable releasing.

> > Kind regards
> > 
> > Andre
> > 
> > 
> > [1] https://review.openstack.org/#/c/414728/
> > [2] https://review.openstack.org/#/c/417310/
> > [3] "Refactoring - Improving the Design of Existing Code", Martin
> >     Fowler, Addison Wesley, Boston, 2011
> > [4] https://review.openstack.org/#/c/414728/8/elements/debootstrap-minimal/root.d/99-clean-up-cache
> > [5] https://review.openstack.org/#/c/413221/
> > 

Thanks,
Greg