Open Stack

Hey Everyone,

So we merged the super simple driver cert test script in to devstack a
while back.  For those that aren't familiar you can check it out here
[1].  First iteration of this is simply a do it yourself config and
run that goes through the same volume-tests that every cinder patch
runs through the gate.

There's definitely room for growth here but this seems like a
reasonable first step.  The remaining question here is how do we want
to use this?  I've made a coupe of suggestions that I'd like to review
and get some feed-back.  To be clear this can obviously evolve over
time, but I'd like to start somewhat simple, try it out and build off
of if depending on how things go.  So with that here's a couple of
options I've been considering:

1. File a bug in launchpad:
This bug would be for tracking purposes, it would be something like
"no cert results available for driver-X".  This would require that the
driver maintainer download/install devstack, configure their driver
and backend and then run the supplied script.

The next question is what to do with the results, there are some options here:
  a. Take the resultant tgz file and post it into the bug report as an
attachment.  Assuming everything passes the bug can then be marked as
closed/resolved.
  b. Create a repo (or even a directory in the Cinder tree) that
includes results files.  That way the bug is logged and a gerrit
commit referencing the bug id is submitted and reviewed very similar
to how we handle source changes.

Option 'a' is path of least resistance, however it becomes a very
manual process and it's somewhat ugly.  Option 'b' fits more with how
we operate anyway, and provides some automation, and it also leaves a
record of the cert process in the tree that makes visibility and
tracking much easier.



2.  Create a web/wiki page specifically for this information:
This would basicly be a matrix of the drivers, and the current status
of the cert results for the current iteration.  It would be something
like a row for every driver in the tree and a column for "last cycle"
and "current cycle".  We'd basically set it up so that the
"current-cycle" entries are all listed as "not submitted" after the
milestone is cut.  The current entries in that column would roll back
to the "last-cycle" column.  Then the driver maintainer could
run/update the matrix at any time during that cycle.

The only thing with this is again it's very manual in terms of
tracking, might be a bit confusing (this may make perfect sense to me,
but seems like jibberish to others :)), and we'd want to have a
repository to store the results files for people to reference.

I'm open to ideas/suggestions here keeping in mind that the initial
purpose is to provide publicly viewable information as to the
integration status of the drivers in the Cinder project.  This would
help people building OpenStack clouds to make sure that the backend
devices they may be choosing actually implement all of the base
features and that they actually work.  Vendors can of course choose
not to participate, that just tells consumers "beware, vendor-a
doesn't necessarily care all that much, or doesn't have time to test
this".

Anyway, hopefully this makes sense, if more clarification is needed I
can try and clean up my descriptions a bit.

Thanks,
John

[1]: https://github.com/openstack-dev/devstack/blob/master/driver_certs/cinder_driver_cert.sh