Open Stack

On 06/07/2011 02:36 PM, Dan Prince wrote:
> Hi Monty,
> 
> I spent a few moments to put together some more detailed notes on
> SmokeStack:
> 
> http://wiki.openstack.org/smokestack
> 
> Screenshots and a link to the live server are on that wiki page.
> 
> At this point the ability to specify custom nova and glance branches
> is supported. Each job runs a set of smoke tests that cover both the
> OS and EC2 APIs.

Excellent. I'm looking forward to working with you to figure out how we
can integrate this in a way which is usable/consumable by the Jenkins
system.

> We've been torpedoing quite a few branches with SmokeStack lately.
> Its proving very useful in doing reviews and helping make sure we get
> things right before they land in trunk.

Yup. I couldn't agree more - the more pre-merge testing that can happen
the better!

Monty

> -----Original Message----- From: "Monty Taylor"
> <mordred at inaugust.com> Sent: Tuesday, June 7, 2011 12:30pm To:
> "openstack at lists.launchpad.net" <openstack at lists.launchpad.net> Cc:
> "Dan Prince" <dan.prince at rackspace.com>, "Mihai Ibanescu"
> <misa at rpath.com>, "Peter J. Pouliot" <ppouliot at suse.com> Subject:
> Overview of CI/Testing
> 
> Hey everybody,
> 
> Here's a quick write up on what we have at the moment and what the
> plan is moving forward, as it stands today. In case you weren't
> aware, the Jenkins instance sits at:
> 
> http://jenkins.openstack.org
> 
> If you think you want to help administer/hack on our Jenkins, that
> is managed through the openstack-ci-admins team on Launchpad..
> 
> As most of you are aware, the main trunk branches for nova, swift, 
> glance and burrow are all managed by Jenkins and Tarmac. What this
> means in short is that jenkins/tarmac finds approved merge proposals,
> runs the in-tree unit tests on them, and if successful, pushes back
> to trunk. Additionally, we have jenkins jobs which run pep8, pylint
> and code coverage jobs and produce reports. For instance:
> 
> http://jenkins.openstack.org/job/nova-coverage/815/cobertura/?
> 
> Once the code has been merged, we have jenkins jobs which produce 
> tarballs, and debian source packages which are pushed up to
> launchpad PPAs so that effectively every trunk commit winds up having
> an associated package.
> 
> There is also a feature that it seems no one knew about which allows 
> developers to submit branch URLs to jenkins to have it run its tests
> on that branch. For each project, this is ${project_name}-param, so
> for instance:
> 
> http://jenkins.openstack..org/job/nova-param/
> 
> Will allow anyone in the ~nova team to submit a branch and have
> jenkins pull it and run on it what it would do via tarmac - helpful
> when fixing something tarmac complained about.
> 
> We are planning to get rid of tarmac and get that integrated in as a 
> Jenkins plugin, because otherwise supporting things like NTT's wish
> for stricter coverage metrics as branch gatekeepers, or smoketesting
> is preventing branches from getting in become really baroque to
> support.
> 
> ** If you like hacking on Java - this should be a fun little
> project. Send me a note and I can outline what I was going to do and
> if you want it it's all yours. **
> 
> Moving forward, the big ticket item is testing not only in-tree unit 
> tests, but installing things and testing that.
> 
> Currently, we have jenkins spinning up a cloud server, copying the
> most recently built debs from the last successful build to that
> server, installing them, starting things up and then running the
> smoketests against the installed code. Currently the install/setup is
> being done by hand rather than via chef purely so that I could walk
> through what's actually needed to get a single-node minimal install
> working and collect information on bugs/workarounds that have to
> happen.
> 
> http://jenkins.openstack.org/job/nova-smoketests
> 
> The test failures here are due to config issues which should go away 
> once I migrate that job to using chef for the node code - so I
> expect that to go green soon.
> 
> The next step then is to replace the use of shell commands over ssh
> with the chef recipes. (now that I have a good handle on what's
> going on/needed there, following the existing chef automation work is
> much more fruitful)
> 
> After that, we'll move from launching cloud servers using the
> one-off libcloud-based python script I wrote to using the jenkins
> jclouds plugin. (if for no other reason that to make sure that all of
> the moving parts of this of any complexity are sensibly checked in
> and have lifecycles that people can hack on.)
> 
> ** If you like hacking on Java - this is a project that's both
> helpful for us as consumers in OpenStack - but also is something
> that effectively will allow other people to use OpenStack to manage
> Jenkins build slaves... SO - come and hack on/improviding the
> jenkins jclouds-plugin. It's at
> https://github.com/jenkinsci/jclouds-plugin **
> 
> And then we need to apply this to swift/glance as well.
> 
> 
> That's just testing API in a VM though, and doesn't get us to
> testing actual bare-metal deployment or integration testing. At
> Rackspace, we have some machines set aside at the moment, and have
> had others offer chunks of machines to test various combinations of
> things. At its heart, the abstract version of this looks fairly
> identical to the smoketests job - pxe boot machines, shove version to
> be tested on them, run tests. However, there are several moving bits
> on the best way to actually do the how. At the moment, the fine folks
> at rPath have a Jenkins installing and testing rPath OpenStack
> images, so Mihai and I are going to look at getting that setup ported
> to our Jenkins. However, although that will be an excellent test of
> code, as our main target platform is Ubuntu, we're also looking at
> doing a straight-up cobbler install using generated .debs. In any
> case, this is the bit which is still in the planning and discussion
> phase, but so far all of the conversations I've had with folks have
> been great - and I'd love to get more folks involved in that (thus
> this email)
> 
> However- latent goal here is that whatever mechanism we're having 
> Jenkins use to deploy OpenStack onto real hardware should be
> consumable and one that actual people might actually use - otherwise
> what the heck are we testing?
> 
> Additionally, as you may have surmised, it is also a goal to run as
> much of this as possible from the OpenStack Jenkins, because that way
> we can as a project choose to incorporate as much of the
> feedback/results of various forms of testing directly in to branch
> testing/approval as we want. For some things (spinning up 20 node
> OpenStack clusters) doing it on every merge proposal or giving all
> devs the ability to click a button and have it run on their branch
> will likely be overkill - but if it turns out not to be, it would be
> great to have the ability to do it.
> 
> End goal is to have: - publicly accessible and usable system for
> testing and build automation - resources that it uses to spin up
> clouds in order to test them are themselves usable by people to spin
> up clouds - tooling around this is done in a manner that makes us of
> and contributes back to existing projects (jenkins plugins, patches
> back to cobbler/orchestra/whatever)
> 
> If you didn't read my _other_ long email from a few moments ago,
> actual discussion of getting this done - and figuring out other
> people's needs/tools and how to integrate them - is hopefully
> happening next week right before the regular openstack-meeting. In
> the mean time, please either flame on right here in list, or ping me
> back personally.
> 
> Thanks everyone! Monty
> 
> 
>