Open Stack

Hi all,

I've recently started looking at the various methods for deploying and
developing tripleo.  What I would like to bring up is the current
combination of the tooling for managing the VM instances and the
actual deployment method to launch the undercloud/overcloud
installation.  While running through the various methods and reading
up on the documentation, I'm concerned that they are not currently
flexible enough for a developer (or operator for that matter) to be
able to setup the various environment configurations for testing
deployments and doing development.  Additionally I ran into issues
just trying get them working at all so this probably doesn't help when
trying to attract new contributors as well.  The focus of this email
and of my experience seems to relate with workflow-simplification
spec[0].  I would like to share my experiences with the various
tooling available and raise some ideas.

Example Situation:

For example, I have a laptop with 16G of RAM and an SSD and I'd like
to get started with tripleo.  How can I deploy tripleo?

Tools:

instack:

I started with the tripleo docs[1] that reference using the instack
tools for virtual environment creation while deploying tripleo.   The
docs say you need at least 12G of RAM[2].  The docs lie (step 7[3]).
So after basically shutting everything down and letting it deploy with
all my RAM, the deployment fails because the undercloud runs out of
RAM and OOM killer kills off heat.  This was not because I had reduced
the amount of ram for the undercloud node or anything.  It was because
by default, 6GB of RAM with no swap is configured for the undercloud
(not sure if this is a bug?).  So I added a swap file to the
undercloud and continued. My next adventure was having the overcloud
deployment fail because lack of memory as puppet fails trying to spawn
a process and gets denied.  The instack method does not configure swap
for the VMs that are deployed and the deployment did not work with 5GB
RAM for each node.  So for a full 16GB I was unable to follow the
documentation and use instack to successfully deploy.  At this point I
switched over to trying to use tripleo-quickstart.  Eventually I was
able to figure out a configuration with instack to get it to deploy
when I figured out how to enable swap for the overcloud deployment.


tripleo-quickstart:

The next thing I attempted to use was the tripleo-quickstart[4].
Following the directions I attempted to deploy against my localhost.
I turns out that doesn't work as expected since ansible likes to do
magic when dealing with localhost[5].  Ultimately I was unable to get
it working against my laptop locally because I ran into some libvirt
issues.  But I was able to get it to work when I pointed it at a
separate machine.  It should be noted that tripleo-quickstart creates
an undercloud with swap which was nice because then it actually works,
but is an inconsistent experience depending on which tool you used for
your deployment.

Thoughts:

What these two methods showed me is that the deployment of tripleo is
not exactly a foolproof thing and that there are a lot of assumptions
that are being handled by the both of these tools.  My initial goal to
start this conversation around tooling and workflows was to bring the
idea of separation of the (virtual) environment configuration from the
actual deployment of tripleo as well as identifying places for
improvement as a way to speed up development and deployment testing.
I believe there are a few reasons why this can be beneficial.

The first reason is that as a developer, I would like to simplify the
development environment creation process and be able to draw the line
between environment and actual deployment tool.  By developing and
documenting a working development/deployment workflow, we can simplify
the onboarding experience as well as possibly accelerating the
existing development processes by reducing the time spent messing with
creating environments.  Does tripleo need to manage creation of VMs to
deploy on? The answer is probably no.  As the end user will want to
deploy tripleo on his or her gear, the focus for tripleo probably
should be on improving that process.  Now this doesn't mean that we
can't write stuff to do this, as it's important for development and
testing.  I'm not sure this is a core part of what should be
'tripleo'.

Another reason why I think this is important is as we talk about
creating different scenarios for CI[6] to improve testing, it would
also be useful for a developer or qa engineer to be able to test
different environmental configurations that would be more realistic of
actual deployment scenarios without having to hunt down multiple
machines or configure actual hardware networking.  For example,
creating environments using multiple networks, changing NICs,
providing different sized nodes based on roles, etc can all be done
virtually.  While tripleo-quickstart has some of these options, it is
mixed in with the tripleo deployment process and does not seem to
align with being able to deploy tripleo in more real world networking
or environmental scenarios.

Since there are a bunch of assumptions baked into the existing
development scripts, I would say the current approach is more 'it
works in devstack' than 'it works for the end user'.  This is not to
say the currently tools don't have their uses as they currently work
for the existing CI setup and for many developers today.  I think we
can do better if we draw clearer lines between what is tripleo and
what is something that is environmental and a supporting tool.

Ideas:

As part of bringing something to get the conversation started and to
better understand how things work, I spent about two days coming up
with a PoC[7] for a workflow that splits the environment creation,
configuration, and management out from the actual deployment of the
undercloud/overcloud.  Having previously used other tools for managing
environments for deploying openstack, I thought I'd try deploying
tripleo using something I was familiar with, fuel-devops[8].  The
whole point of fuel-devops is to be able to create and manage entire
virtual environments (and their networking configuration) on a given
host.  With this, I was able to create my environment setup in a yaml
file[9] which would then be able to be reproduced.  So with this tool,
I'm able to create a number nodes of a given memory, disk, network
configuration as part of an 'environment'.  This environment is
completely separated from another environment which means given a
large enough virtual host, I could have multiple tripleo deployments
occurring simultaneously on their own networks.  This is a nice
feature, but just an added bonus to the tool (along with snapshotting
and a few other nifty things).  The bigger feature here is that this
is more representative of what someone using tripleo is going to
experience. They are going to have their environment already
configured and would like to deploy tripleo on it.  Once the
environment was created, I started to understand what it would be like
for an end user to take an undercloud image and deploy it.
Fortunately because we're still dealing with VMs, you can just point
the undercloud node at the undercloud image itself[10] for testing
purposes.

Once the environment exists, it starts exposing what exactly it means
to deploy a tripleo undercloud/overcloud.  The majority of the effort
I had to expend for this PoC was actually related to the construction
of the instackenv.json to load the overcloud nodes into ironic.  As
mentioned in the workflow-simplification spec[0], this is a known
limitation and there are possible solutions and I think this is
important of the end user experience when trying to work with tripleo.
It should be noted that I managed to get the undercloud and
controller/compute deployed (but eating into VM swap space) in 12GB on
my laptop.  This was something I was unable to do with either instack
or tripleo-quickstart.

There are some short coming with this particular tool choice. My
understanding is that fuel-devops is still limited to managing a
single host.  So you don't use it against remote nodes, but it is good
if you have decently sized physical machine or want to work locally.
I ran into issues with network configurations and pxe booting, but I
have a feeling that's more of a bug in libvirt and my lack of time to
devote to undercloud setup.  So it's not perfect, but it does show off
the basics of the concept.  Overall I think clearly defining the
tripleo installation process from the environment configuration is an
important step for end user usability and even developer workflows.

Thanks,
-Alex

[0] https://review.openstack.org/238192
[1] http://docs.openstack.org/developer/tripleo-docs/environments/environments.html#virtual-environment
[2] http://docs.openstack.org/developer/tripleo-docs/environments/environments.html#minimum-system-requirements
[3] http://docs.openstack.org/developer/tripleo-docs/environments/environments.html#preparing-the-virtual-environment-automated
[4] https://github.com/openstack/tripleo-quickstart
[5] http://eavesdrop.openstack.org/irclogs/%23rdo/%23rdo.2016-09-06.log.html#t2016-09-06T19:33:44
[6] http://lists.openstack.org/pipermail/openstack-dev/2016-August/101786.html
[7] https://github.com/mwhahaha/fuel-devops-tripleo
[8] https://github.com/openstack/fuel-devops
[9] https://github.com/mwhahaha/fuel-devops-tripleo/blob/master/ooo-template.yaml
[10] https://github.com/mwhahaha/fuel-devops-tripleo/blob/master/ooo-template.yaml#L78