[openstack-dev] [TripleO][CI] Ability to reproduce failures

Steven Hardy shardy at redhat.com
Fri Apr 8 15:18:40 UTC 2016


Hi Gabriele,

On Thu, Apr 07, 2016 at 05:03:33PM -0400, Gabriele Cerami wrote:
> Hi,
> 
> I'm trying to find an entry point to join the effort in TripleO CI.
> I studied the infrastructure and the scripts, but there's still something I'm missing.
> The last step of studying the complex landscape of TripleO CI and the first to start contributing
> is being able to reproduce failures in an accessible environment, to start debugging issues.
> I have not found an easy and stable way to do this. Jobs are certainly gathering
> a lot of logs, but that's not enough.
> 
> At the moment, I started launching periodic jobs on my local test box using
> this script
> https://github.com/sshnaidm/various/blob/master/tripleo_repr.sh
> 
> It's quite handy, but I'm not sure it's able to produce perfectly compatible 
> environments with what's in CI.
> 
> Can anyone suggest a way to make jobs reproducible locally? I know it may be complicated
> to setup an environment through devtest, but may If we can start with just a list of steps, 
> then it would be easier to put them into a script, hten make it availabe in the log in place
> of the current reproduce.sh that is not very useful.

Note we're not using devtest at all anymore, the developer script many
folks use is tripleo.sh:

https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh

Your script is already calling this so you're pretty close to what others
are using I think.

There is an effort in https://github.com/openstack/tripleo-quickstart to
make this process more automated, but my setup process looks like this
http://paste.fedoraproject.org/351818/1283361/ (note the RAM allocation
there is more than is typically required)

In both cases you end up using a different process to setup the instack VM,
but otherwise gets you very close to what CI runs and generally reproduces
issues.

Note that once you have an environment, you can often just yum update the
undercloud (and possibly re-run openstack undercloud install) to reproduce
issues (sometimes you'll also have to rebuild the overcloud-full image) -
so when I seen an issue in CI typically my first step is incrementally
updating things on my local environment rather than completly building a
new one from scratch (which obviously takes much longer).

I agree we could make this easier, and it'd be good if those scripts other
than tripleo.sh were more easily reusable in developer environments.

Steve



More information about the OpenStack-dev mailing list