Open Stack

On 04/04/2013 10:13 AM, Daniel P. Berrange wrote:
> On Thu, Apr 04, 2013 at 12:27:29PM +1300, Robert Collins wrote:
>> On 4 April 2013 11:10, Ian McLeod <imcleod at redhat.com> wrote:
>>> We've put together a simple proof of concept of driving native operating
>>> system installers using Nova:
>>>
>>> https://github.com/redhat-openstack/image-building-poc
>>>
>>> The python CLI tool in this repo, when combined with the example install
>>> scripts, can build both glance-backed and cinder-backed JEOS images for
>>> the following operating systems entirely inside of Nova:
>>>
>>> Fedora 17, 18
>>> Ubuntu 10.04, 12.04, 12.10
>>> RHEL 5.9, 6.4
>>>
>>> These builds can be done using network install sources and, in some
>>> limited cases, DVD/ISO install sources.  Full details can be found in
>>> the README.md file on github.
>>>
>>> Some background discussion about this approach from earlier this year
>>> can be found here:
>>>
>>> https://wiki.openstack.org/wiki/NovaImageBuilding
>>>
>>> The code is a bit rough around the edges but we believe the approach has
>>> promise and can form the basis for a proper image building service
>>> native to OpenStack.
>>
>> It's cool you've done this - I think such a general solution is
>> necessary, particularly for folk running systems like Windows.
>>
>>> I'd be grateful if anyone who's interested in the problem of image
>>> building could have a look at this, give it a try and share
>>> thoughts/feedback.
>>
>> We've been working on a much lighter weight solution - using as you
>> note chroots - because we need something that can build images *fast*.
>> TripleO wants to deploy Openstack as part of the CI pipeline, so even
>> short (multiple second) delays add up rapidly as you starting
>> generating images inline in the landing process.
>> https://github.com/stackforge/diskimage-builder/ is the project we put
>> together.
>>
>> The focus we have is building images which can be used to deploy
>> openstack, so we have a hard requirement to not depend on openstack
>> itself - but having an API that can drive building of images (by
>> spinning up a machine on the back end) would be useful, to make custom
>> images easier for folk on limited hardware etc.
>>
>> Perhaps we can collaborate on having a common API for creating images,
>> with multiple backends - one oz for super generality, and one
>> diskimage-builder for fast creation?
> 
> There is much more than just speed to consider here - there are a great
> many downsides to the chroot based approach IMHO, to the extent that I
> don't think it is worthwhile going down that route.

Interesting...

> First of all a chroot based approach is not taking advantage of the
> OS distro's installer, so you have to writen a load of custom install
> code for each OS to be supported. The many existing/previous chroot
> based installers show this OS specific code is very fragile and will
> frequently break when a new release comes out & is alot of work to
> maintain.

So - actually avoiding the OS installer is one of the reasons we're
doing chroot/base-image based approach. You aren't gaining anything in
fixing fragility by using kickstart/preseed + installer in terms of OS
upgrades. The kickstart/preseed file will still need to be updated.

On the other hand, starting from an upstream base cloud image, mounting
it locally, modifying it by installing things in to it, and then
re-packing it into a new image is simple and straightforward and doesn't
require image creators to learn kickstart/preseed.

Installer-based approaches are also quite hard to debug, since kickstart
and preseed both are running inside of a framework context. On the other
hand, booting a base cloud image and then running specialization
commands is super simple to grok.

Finally - the fact that kickstart and preseed are both completely
unrelated to one another makes it harder to grok the installer approach
from a cross-distro perspective. I mean, deb vs. rpm is always going to
get us into a problem space - but the difference between "apt-get
install mysql-server" and "yum install mysql-server" is not nearly as
hard to reason about as the difference in writing the kickstart and
preseed stanzas to do the same thing.

However - if you like doing that, then by all means - I don't think
there is anything about having a base-image based approach that should
preclude someone else from doing an installer based approach!

> You're limited by the types of OS that can bootstrapped via chroot,
> to mostly just Linux distros. No Windows, BSD, Solaris etc. Even with
> Linux OS, you can run into compatibility issues if the kernel running
> the chroot is not the same as the kernel that the OS usually uses.
> For example, at certain times glibc in Fedora is set to require a new
> minimum kernel version. So if your OpenStack is say RHEL-6 you may
> find it impossible to do a chroot install of, say, Fedora 22.
> 
> Unless you boot up a guest in which run the chroot install, you are
> exposing yourself to serious security risks. If you are booting up a
> guest to run the chroot install, then you might as well just run the
> real OS installer in a guest and forget about chroot installs.

Booting a guest to deal with the parts of the install that actually need
the guest kernel/libc to be running is super quick and painless.

> Installation time is going to be dominated by the time to install the
> packages actual packages, regardless of what technique is used to do
> the installation. Running the native guest installer requires booting
> a VM which has a little overhead, but as mentioned above and chroot
> tool should be run inside a throw-away guest too, otherwise you have
> unacceptable security risks to your infrastructure. So I don't really
> think there is going to be much of a compelling speed advantage to
> chroot based installs - a few % difference at best.

The installer has to install the base OS via package installation. The
image-based installer only has to install the specific packages that
diverge from the base OS image. Booting and running in a local guest
image isn't going to figure in to the time discussion.

> Finally there are already many chroot based installers for OS that
> exist. Given that the amount of custom code required for each OS
> to be supported, I'm not convinced it is a good use of time to be
> re-inventing this in OpenStack. For the few places which really do
> want todo chroot based installs, just use an existing tool.

Well, the work is already done. The existing tools either didn't do what
we needed them to do or were super complex (think XML config files)

diskimage-builder uses collections of run-parts directories, which means
that if you know how to write a shell script to install software onto a
machine, you know how to write an element to produce a disk image. The
tool is quick and easy (just a couple of minutes to run) and has a very
low learning curve.

> OpenStack's value add comes in providing an easy way to automate the
> creating of images, by taking a distro installer from glance and
> running it in nova.

I explicitly do not want to run an distro installer. But - I do support
your desire to do that.

Monty