Open Stack

On Fri, Feb 21, 2014 at 10:46:22AM -0500, Sean Dague wrote:
> On 02/21/2014 09:45 AM, Daniel P. Berrange wrote:
> > On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
> >>
> >> So I'm one of the first people to utter "if it isn't tested, it's
> >> probably broken", however I also think we need to be realistic about the
> >> fact that if you did out the permutations of dependencies and config
> >> options, we'd have as many test matrix scenarios as grains of sand on
> >> the planet.
> >>
> >> I do think in some ways this is unique to OpenStack, in that our
> >> automated testing is head and shoulders above any other Open Source
> >> project out there, and most proprietary software systems I've seen.
> >>
> >> So this is about being pragmatic. In our dependency testing we are
> >> actually testing with most recent versions of everything. So I would
> >> think that even with libvirt, we should err in that direction.
> > 
> > I'm very much against that, because IME, time & time again across
> > all open source projects I've worked on, people silently introduce
> > use of features/apis that only exist in newer versions without anyone
> > ever noticing until it is too late.
> > 
> >> That being said, we also need to be a little bit careful about taking
> >> such a hard line about "supported vs. not" based on only what's in the
> >> gate. Because if we did the following things would be listed as
> >> unsupported (in increasing level of ridiculousness):
> >>
> >>  * Live migration
> >>  * Using qpid or zmq
> >>  * Running on anything other than Ubuntu 12.04
> >>  * Running on multiple nodes
> >>
> >> Supported to me means we think it should work, and if it doesn't, it's a
> >> high priority bug that will get fixed quickly. Testing is our sanity
> >> check. But it can't be considered that it will catch everything, at
> >> least not before the heat death of the universe.
> > 
> > I agree we should be pragmatic here to some extent. We shouldn't aim to
> > test every single intermediate version, or every possible permutation of
> > versions - just a representative sample. Testing both lowest and highest
> > versions is a reasonable sample set IMHO.
> 
> Testing lower bounds is interesting, because of the way pip works. That
> being said, if someone wants to take ownership of building that job to
> start as a periodic job, I'm happy to point in the right direction. Just
> right now, it's a lower priority item than things like Tempest self
> testing, Heat actually gating, Neutron running in parallel, Nova API
> coverage.

If it would be hard work to do it for python modules, we can at least
not remove the existing testing of an old libvirt version - simply add
an additional test with newer libvirt.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|