Open Stack

Hi Monty,

I spent a few moments to put together some more detailed notes on SmokeStack:

  http://wiki.openstack.org/smokestack

Screenshots and a link to the live server are on that wiki page.

At this point the ability to specify custom nova and glance branches is supported. Each job runs a set of smoke tests that cover both the OS and EC2 APIs.

--

We've been torpedoing quite a few branches with SmokeStack lately. Its proving very useful in doing reviews and helping make sure we get things right before they land in trunk.

Dan

-----Original Message-----
From: "Monty Taylor" <mordred at inaugust.com>
Sent: Tuesday, June 7, 2011 12:30pm
To: "openstack at lists.launchpad.net" <openstack at lists.launchpad.net>
Cc: "Dan Prince" <dan.prince at rackspace.com>, "Mihai Ibanescu" <misa at rpath.com>, "Peter J. Pouliot" <ppouliot at suse.com>
Subject: Overview of CI/Testing

Hey everybody,

Here's a quick write up on what we have at the moment and what the plan
is moving forward, as it stands today. In case you weren't aware, the
Jenkins instance sits at:

http://jenkins.openstack.org

If you think you want to help administer/hack on our Jenkins, that is
managed through the openstack-ci-admins team on Launchpad.

As most of you are aware, the main trunk branches for nova, swift,
glance and burrow are all managed by Jenkins and Tarmac. What this means
in short is that jenkins/tarmac finds approved merge proposals, runs the
in-tree unit tests on them, and if successful, pushes back to trunk.
Additionally, we have jenkins jobs which run pep8, pylint and code
coverage jobs and produce reports. For instance:

http://jenkins.openstack.org/job/nova-coverage/815/cobertura/?

Once the code has been merged, we have jenkins jobs which produce
tarballs, and debian source packages which are pushed up to launchpad
PPAs so that effectively every trunk commit winds up having an
associated package.

There is also a feature that it seems no one knew about which allows
developers to submit branch URLs to jenkins to have it run its tests on
that branch. For each project, this is ${project_name}-param, so for
instance:

http://jenkins.openstack.org/job/nova-param/

Will allow anyone in the ~nova team to submit a branch and have jenkins
pull it and run on it what it would do via tarmac - helpful when fixing
something tarmac complained about.

We are planning to get rid of tarmac and get that integrated in as a
Jenkins plugin, because otherwise supporting things like NTT's wish for
stricter coverage metrics as branch gatekeepers, or smoketesting is
preventing branches from getting in become really baroque to support.

** If you like hacking on Java - this should be a fun little project.
Send me a note and I can outline what I was going to do and if you want
it it's all yours. **

Moving forward, the big ticket item is testing not only in-tree unit
tests, but installing things and testing that.

Currently, we have jenkins spinning up a cloud server, copying the most
recently built debs from the last successful build to that server,
installing them, starting things up and then running the smoketests
against the installed code. Currently the install/setup is being done by
hand rather than via chef purely so that I could walk through what's
actually needed to get a single-node minimal install working and collect
information on bugs/workarounds that have to happen.

http://jenkins.openstack.org/job/nova-smoketests

The test failures here are due to config issues which should go away
once I migrate that job to using chef for the node code - so I expect
that to go green soon.

The next step then is to replace the use of shell commands over ssh with
the chef recipes. (now that I have a good handle on what's going
on/needed there, following the existing chef automation work is much
more fruitful)

After that, we'll move from launching cloud servers using the one-off
libcloud-based python script I wrote to using the jenkins jclouds
plugin. (if for no other reason that to make sure that all of the moving
parts of this of any complexity are sensibly checked in and have
lifecycles that people can hack on.)

** If you like hacking on Java - this is a project that's both helpful
for us as consumers in OpenStack - but also is something that
effectively will allow other people to use OpenStack to manage Jenkins
build slaves... SO - come and hack on/improviding the jenkins
jclouds-plugin. It's at https://github.com/jenkinsci/jclouds-plugin **

And then we need to apply this to swift/glance as well.


That's just testing API in a VM though, and doesn't get us to testing
actual bare-metal deployment or integration testing. At Rackspace, we
have some machines set aside at the moment, and have had others offer
chunks of machines to test various combinations of things. At its heart,
the abstract version of this looks fairly identical to the smoketests
job - pxe boot machines, shove version to be tested on them, run tests.
However, there are several moving bits on the best way to actually do
the how. At the moment, the fine folks at rPath have a Jenkins
installing and testing rPath OpenStack images, so Mihai and I are going
to look at getting that setup ported to our Jenkins. However, although
that will be an excellent test of code, as our main target platform is
Ubuntu, we're also looking at doing a straight-up cobbler install using
generated .debs. In any case, this is the bit which is still in the
planning and discussion phase, but so far all of the conversations I've
had with folks have been great - and I'd love to get more folks involved
in that (thus this email)

However- latent goal here is that whatever mechanism we're having
Jenkins use to deploy OpenStack onto real hardware should be consumable
and one that actual people might actually use - otherwise what the heck
are we testing?

Additionally, as you may have surmised, it is also a goal to run as much
of this as possible from the OpenStack Jenkins, because that way we can
as a project choose to incorporate as much of the feedback/results of
various forms of testing directly in to branch testing/approval as we
want. For some things (spinning up 20 node OpenStack clusters) doing it
on every merge proposal or giving all devs the ability to click a button
and have it run on their branch will likely be overkill - but if it
turns out not to be, it would be great to have the ability to do it.

End goal is to have:
  - publicly accessible and usable system for testing and build automation
  - resources that it uses to spin up clouds in order to test them are
themselves usable by people to spin up clouds
  - tooling around this is done in a manner that makes us of and
contributes back to existing projects (jenkins plugins, patches back to
cobbler/orchestra/whatever)

If you didn't read my _other_ long email from a few moments ago, actual
discussion of getting this done - and figuring out other people's
needs/tools and how to integrate them - is hopefully happening next week
right before the regular openstack-meeting. In the mean time, please
either flame on right here in list, or ping me back personally.

Thanks everyone!
Monty