Open Stack

On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
> 
> So I'm one of the first people to utter "if it isn't tested, it's
> probably broken", however I also think we need to be realistic about the
> fact that if you did out the permutations of dependencies and config
> options, we'd have as many test matrix scenarios as grains of sand on
> the planet.
> 
> I do think in some ways this is unique to OpenStack, in that our
> automated testing is head and shoulders above any other Open Source
> project out there, and most proprietary software systems I've seen.
> 
> So this is about being pragmatic. In our dependency testing we are
> actually testing with most recent versions of everything. So I would
> think that even with libvirt, we should err in that direction.

I'm very much against that, because IME, time & time again across
all open source projects I've worked on, people silently introduce
use of features/apis that only exist in newer versions without anyone
ever noticing until it is too late.

> That being said, we also need to be a little bit careful about taking
> such a hard line about "supported vs. not" based on only what's in the
> gate. Because if we did the following things would be listed as
> unsupported (in increasing level of ridiculousness):
> 
>  * Live migration
>  * Using qpid or zmq
>  * Running on anything other than Ubuntu 12.04
>  * Running on multiple nodes
> 
> Supported to me means we think it should work, and if it doesn't, it's a
> high priority bug that will get fixed quickly. Testing is our sanity
> check. But it can't be considered that it will catch everything, at
> least not before the heat death of the universe.

I agree we should be pragmatic here to some extent. We shouldn't aim to
test every single intermediate version, or every possible permutation of
versions - just a representative sample. Testing both lowest and highest
versions is a reasonable sample set IMHO.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|