Open Stack

On 09/19/2016 01:21 PM, Steven Hardy wrote:
> Hi Alex,
> 
> Firstly, thanks for this detailed feedback - it's very helpful to have
> someone with a fresh perspective look at the day-1 experience for TripleO,
> and while some of what follows are "know issues", it's great to get some
> perspective on them, as well as ideas re how we might improve things.
> 
> On Thu, Sep 15, 2016 at 09:09:24AM -0600, Alex Schultz wrote:
>> Hi all,
>>
>> I've recently started looking at the various methods for deploying and
>> developing tripleo.  What I would like to bring up is the current
>> combination of the tooling for managing the VM instances and the
>> actual deployment method to launch the undercloud/overcloud
>> installation.  While running through the various methods and reading
>> up on the documentation, I'm concerned that they are not currently
>> flexible enough for a developer (or operator for that matter) to be
>> able to setup the various environment configurations for testing
>> deployments and doing development.  Additionally I ran into issues
>> just trying get them working at all so this probably doesn't help when
>> trying to attract new contributors as well.  The focus of this email
>> and of my experience seems to relate with workflow-simplification
>> spec[0].  I would like to share my experiences with the various
>> tooling available and raise some ideas.
>>
>> Example Situation:
>>
>> For example, I have a laptop with 16G of RAM and an SSD and I'd like
>> to get started with tripleo.  How can I deploy tripleo?
> 
> So, this is probably problem #1, because while I have managed to deploy a
> minimal TripleO environment on a laptop with 16G of RAM, I think it's
> pretty widely known that it's not really enough (certainly with our default
> configuration, which has unfortunately grown over time as more and more
> things got integrated).
> 
> I see two options here:
> 
> 1. Document the reality (which is really you need a physical machine with
> at least 32G RAM unless you're prepared to deal with swapping).
> 
> 2. Look at providing a "TripleO lite" install option, which disables some
> services (both on the undercloud and default overcloud install).
> 
> Either of these are defintely possible, but (2) seems like the best
> long-term solution (although it probably means another CI job).
> 
>> Tools:
>>
>> instack:
>>
>> I started with the tripleo docs[1] that reference using the instack
>> tools for virtual environment creation while deploying tripleo.   The
>> docs say you need at least 12G of RAM[2].  The docs lie (step 7[3]).
>> So after basically shutting everything down and letting it deploy with
>> all my RAM, the deployment fails because the undercloud runs out of
>> RAM and OOM killer kills off heat.  This was not because I had reduced
>> the amount of ram for the undercloud node or anything.  It was because
>> by default, 6GB of RAM with no swap is configured for the undercloud
>> (not sure if this is a bug?).  So I added a swap file to the
>> undercloud and continued. My next adventure was having the overcloud
>> deployment fail because lack of memory as puppet fails trying to spawn
>> a process and gets denied.  The instack method does not configure swap
>> for the VMs that are deployed and the deployment did not work with 5GB
>> RAM for each node.  So for a full 16GB I was unable to follow the
>> documentation and use instack to successfully deploy.  At this point I
>> switched over to trying to use tripleo-quickstart.  Eventually I was
>> able to figure out a configuration with instack to get it to deploy
>> when I figured out how to enable swap for the overcloud deployment.
> 
> Yeah, so this definitely exposes that we need to update the docs, and also
> provide an easy install-time option to enable swap on all-the-things for
> memory contrained environments.
> 
>> tripleo-quickstart:
>>
>> The next thing I attempted to use was the tripleo-quickstart[4].
>> Following the directions I attempted to deploy against my localhost.
>> I turns out that doesn't work as expected since ansible likes to do
>> magic when dealing with localhost[5].  Ultimately I was unable to get
>> it working against my laptop locally because I ran into some libvirt
>> issues.  But I was able to get it to work when I pointed it at a
>> separate machine.  It should be noted that tripleo-quickstart creates
>> an undercloud with swap which was nice because then it actually works,
>> but is an inconsistent experience depending on which tool you used for
>> your deployment.
> 
> Yeah, so while a lot of folks have good luck with tripleo-quickstart, it
> has the disadvantage of not currently being the tool used in upstream
> TripleO CI (which folks have looked at fixing, but it's not yet happened).
> 
> The original plan was for tripleo-quickstart to completely replace the
> instack-virt-setup workflow:
> 
> https://blueprints.launchpad.net/tripleo/+spec/tripleo-quickstart
> 
> But for a variety of reasons, we never quite got to that - we may need a
> summit discussion on the path forward here.
> 
> For me (as an upstream developer) it really boils down to the CI usage
> issue - at all times I want to use the tool which gets me closest to what
> runs in upstream CI (which although we actually use instack-virt-setup, we
> otherwise follow the tripleo-docs procedure pretty closely, using a helper
> script called tripleo.sh, which you can run locally):
> 
> http://paste.fedoraproject.org/431073/30480114
> 
This kind of feels like quickstart FUD to me.

CI does not run tripleo.sh. So the paste above is not actually
reproducing any tripleo-ci job. tripleo-ci runs a wrapper around
tripleo.sh[1] with specific ENV variables set in a different script[2],
and then some required steps to actually make that work in yet another
script[3].

Further, if you replaced the instack-virt-setup line in the above script
with a run of tripleo-quickstart, the other steps could still be run on
top of a quickstart undercloud.

[1]
https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/deploy.sh
[2]
https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L107-L200
[3]
https://github.com/openstack-infra/tripleo-ci/blob/master/toci_instack.sh#L180-L182

>> Thoughts:
>>
>> What these two methods showed me is that the deployment of tripleo is
>> not exactly a foolproof thing and that there are a lot of assumptions
>> that are being handled by the both of these tools.  My initial goal to
>> start this conversation around tooling and workflows was to bring the
>> idea of separation of the (virtual) environment configuration from the
>> actual deployment of tripleo as well as identifying places for
>> improvement as a way to speed up development and deployment testing.
>> I believe there are a few reasons why this can be beneficial.
> 
> Yep, I think this goal is uncontentious, and it's pretty much the original
> aim of tripleo-quickstart.
> 
>> The first reason is that as a developer, I would like to simplify the
>> development environment creation process and be able to draw the line
>> between environment and actual deployment tool.  By developing and
>> documenting a working development/deployment workflow, we can simplify
>> the onboarding experience as well as possibly accelerating the
>> existing development processes by reducing the time spent messing with
>> creating environments.  Does tripleo need to manage creation of VMs to
>> deploy on? The answer is probably no.  As the end user will want to
>> deploy tripleo on his or her gear, the focus for tripleo probably
>> should be on improving that process.  Now this doesn't mean that we
>> can't write stuff to do this, as it's important for development and
>> testing.  I'm not sure this is a core part of what should be
>> 'tripleo'.
> 
> Yeah, agreed - the automation around setting up the VMs is really just a
> convenience, and it's not really a core part of TripleO - any tool could be
> used provided the VMs end up configured in the way we require.
> 
>> Another reason why I think this is important is as we talk about
>> creating different scenarios for CI[6] to improve testing, it would
>> also be useful for a developer or qa engineer to be able to test
>> different environmental configurations that would be more realistic of
>> actual deployment scenarios without having to hunt down multiple
>> machines or configure actual hardware networking.  For example,
>> creating environments using multiple networks, changing NICs,
>> providing different sized nodes based on roles, etc can all be done
>> virtually.  While tripleo-quickstart has some of these options, it is
>> mixed in with the tripleo deployment process and does not seem to
>> align with being able to deploy tripleo in more real world networking
>> or environmental scenarios.
> 
> Yeah, so I think this is one reason why the tripleo-quickstart discussion
> has sometimes proven tricky - the original spec was about replacing only
> the virt-setup pieces, but there was subsequently some scope-creep.  I
> think this is being adressed, but it'd be good to have folks working on
> that chime in here.
> 

Indeed, we have moved all of the ansible code for doing full deployments
outside of tripleo-quickstart. There is still a bunch of CI helper
scripts for RDO CI in tree, and a playbook that exercises the full
deployment code, but I would like to move all of that out of the
quickstart tree as well.

We have also added the ability to consume the images produced by
tripleo-ci rather than using images produced by RDO. This required
adding the ability to use an overcloud-full image as an undercloud
image, since tripleo-ci no longer produces one. I think this will
actually allow us to stop producing an undercloud image downstream of
TripleO as well, which means that those images could be produced using
only methods from tripleo-docs.

I also recently started to look at what it would be like to add a
"virt-setup" option to tripleo.sh that runs quickstart[4]. It is
actually pretty simple for doing just the basic instack-virt-setup part.

However, what developers actually want is to reproduce exactly what runs
in CI. With our current CI architecture it is a bit harder to do that
cleanly in an external tool, but I actually was able to get a minimal
POC of it working[5]. It requires quite a bit of hacky stuff to make
tripleo-ci deploy.sh work[6], but I think it shines a light on what we
need to improve to make tripleo-ci externally consumable. It does seem
like we could get there in under 5 patches though, which is less than I
thought when starting on the POC.

The one piece not addressed in either POC patch is building changes
under test using DLRN. There is a function in tripleo.sh that does this
for us in CI, and it can be leveraged for the developer use case as
well. There is also an ansible role in RDO[7] that could be used for
that purpose. The ansible role has quite a few more features than
tripleo.sh, including multi-gerrit support and allowing to build
packaging changes along with the code changes. It also has a hook in the
tripleo-quickstart code to inject the DLRN repo created into the
overcloud image before we ever boot anything, so from a quickstart
perspective it is quite a bit nicer.

[4] https://review.openstack.org/371587
[5] https://review.openstack.org/374116
[6]
https://review.openstack.org/#/c/374116/1/roles/libvirt/setup/undercloud/templates/complete_deploy.sh.j2
[7] https://github.com/redhat-openstack/ansible-role-tripleo-gate

>> Since there are a bunch of assumptions baked into the existing
>> development scripts, I would say the current approach is more 'it
>> works in devstack' than 'it works for the end user'.  This is not to
>> say the currently tools don't have their uses as they currently work
>> for the existing CI setup and for many developers today.  I think we
>> can do better if we draw clearer lines between what is tripleo and
>> what is something that is environmental and a supporting tool.
> 
> I'm not sure I agree here - after you have your virt stuff setup, the
> TripleO pieces which do the deployment are identical to those used by the
> end user (unlike Devstack where it's likely the entire environment has been
> configured using a different tool to production deployments).
> 
>> Ideas:
>>
>> As part of bringing something to get the conversation started and to
>> better understand how things work, I spent about two days coming up
>> with a PoC[7] for a workflow that splits the environment creation,
>> configuration, and management out from the actual deployment of the
>> undercloud/overcloud.  Having previously used other tools for managing
>> environments for deploying openstack, I thought I'd try deploying
>> tripleo using something I was familiar with, fuel-devops[8].  The
>> whole point of fuel-devops is to be able to create and manage entire
>> virtual environments (and their networking configuration) on a given
>> host.  With this, I was able to create my environment setup in a yaml
>> file[9] which would then be able to be reproduced.  So with this tool,
>> I'm able to create a number nodes of a given memory, disk, network
>> configuration as part of an 'environment'.  This environment is
>> completely separated from another environment which means given a
>> large enough virtual host, I could have multiple tripleo deployments
>> occurring simultaneously on their own networks.  This is a nice
>> feature, but just an added bonus to the tool (along with snapshotting
>> and a few other nifty things).  The bigger feature here is that this
>> is more representative of what someone using tripleo is going to
>> experience. They are going to have their environment already
>> configured and would like to deploy tripleo on it.  Once the
>> environment was created, I started to understand what it would be like
>> for an end user to take an undercloud image and deploy it.
>> Fortunately because we're still dealing with VMs, you can just point
>> the undercloud node at the undercloud image itself[10] for testing
>> purposes.

I think this could be knowing one tool better than another.
tripleo-quickstart can be run in such a way as it only does the
environment setup part. And it takes a yaml config file that describes
that environment. Further, it deploys the VMs using qemu:///session
inside a non-root user, so they would be totally segregated from VMs
using qemu:///system or another non-root user. There may be some small
amount of effort to actually use quickstart to deploy 2 environments on
the same host. I have not tried that, as on a 32G virthost getting a
single realistic deployment (ie HA w/ ceph) is challenging enough.
However, I do run a couple of utility VMs (irc bouncer etc.) on the same
virthost that I do quickstart deploys on, and quickstart does not ever
interfere.

I actually thought it would be neat to add an opposite feature, where a
user could pair two smaller virthosts and deploy VMs across them. I
havent spent any time looking at how difficult that would be to
implement though.

I do think snapshotting would be a really great feature for
tripleo-quickstart, and could be something worth looking at for Ocata.
> 
> This looks very interesting, thanks for sharing! :)
> 
> That said, my main concern if we go this way is we'd end up with three ways
> to do a virt setup vs the current two ;)
> 
> Definitely worthy of broader discussion though, particularly in the context
> of this vs the ansible based tripleo-quickstart.
> 
>> Once the environment exists, it starts exposing what exactly it means
>> to deploy a tripleo undercloud/overcloud.  The majority of the effort
>> I had to expend for this PoC was actually related to the construction
>> of the instackenv.json to load the overcloud nodes into ironic.  As
>> mentioned in the workflow-simplification spec[0], this is a known
>> limitation and there are possible solutions and I think this is
>> important of the end user experience when trying to work with tripleo.
>> It should be noted that I managed to get the undercloud and
>> controller/compute deployed (but eating into VM swap space) in 12GB on
>> my laptop.  This was something I was unable to do with either instack
>> or tripleo-quickstart.

Again, I think this could be knowing one tool better than another.
tripleo-quickstart can tune CPU and memory for each VM[8]. Though it
looks like we could document that better. What is in the defaults and
config files is what we have found to work well on 32G virthosts.

I will say I would be pretty surprised if the "cloud in 12GB" could do
any post-deploy validation without falling over.

[8]
https://github.com/openstack/tripleo-quickstart/blob/master/roles/common/defaults/main.yml#L15-L56

> 
> So, I'm a little unclear here, presumably the actual RAM usage was the
> same, so is this just because you were able to easily configure swap on all
> the VMs?
> 
>> There are some short coming with this particular tool choice. My
>> understanding is that fuel-devops is still limited to managing a
>> single host.  So you don't use it against remote nodes, but it is good
>> if you have decently sized physical machine or want to work locally.
>> I ran into issues with network configurations and pxe booting, but I
>> have a feeling that's more of a bug in libvirt and my lack of time to
>> devote to undercloud setup.  So it's not perfect, but it does show off
>> the basics of the concept.  Overall I think clearly defining the
>> tripleo installation process from the environment configuration is an
>> important step for end user usability and even developer workflows.
> 
> I think the multi-node use-case is mostly handled via OVB[1] now which is
> where you basically use an OpenStack cloud to host the VMs used for a
> TripleO deployment (yes, that is OpenStack on OpenStack on OpenStack).
> 
> We're using that in CI and it works pretty well, so I think the main gap is
> a super-easy day-1 workflow that allows users/developers to get up and
> running easily on a single node (as mentioned above tho, quickstart was
> aimed at closing this gapm and has been working well for a lot of folks).
> 
> Thanks for the feedback - defintely more here we can discuss and hopefully
> refine into actionable bugs/specs/patches! :)
> 
> Steve
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>