Open Stack

On Thu, Feb 20, 2014 at 10:31:06AM -0800, Joe Gordon wrote:
> Hi All,
> 
> I discussion recently came up inside of nova about what it means
> supported version for a dependency means.  in libvirt we gate on the
> minimal version that we support but for all python dependencies we
> gate on the highest version that passes our requirements. While we all
> agree that having two different ways of choosing which version to test
> (min and max) is bad, there are good arguments for doing both.
> 
> testing most recent version:
>     * We want to make sure we support the latest and greatest
>     * Bug fixes
>     * Quickly discover backwards incompatible changes so we can deal
> with them as they arise instead of in batch
> 
> Testing lowest version supported:
>     * Make sure we don't land any code that breaks compatibility with
> the lowest version we say we support

I'm pretty strongly of the opinion that unless you test the minimum
declared version, you shouldn't claim to support it. Experience across
many open source projects is that far too often people silently introduce
features that require new versions of external deps and only get found
by the poor downstream user who tries to actually use the min required
version.

> A few questions and ideas on how to move forward.
>     * How do other projects deal with this? This problem isn't unique
> in OpenStack.

The level of testing is fairly unique to openstack amongst
open source projects.

>     * What are the issues with making one gate job use the latest
> versions and one use the lowest supported versions?

Double the resources is the obvious one I guess :-)

>     * Only test some things on every commit or every day (periodic
> jobs)? But no one ever fixes those things when they break? who wants
> to own them? distros? deployers?

I think for testing done by openstack it should always be gating,
otherwise there's not much incentive to deal with the fallout.
If distros want to periodically test their own stack in a non-gating
manner let them make that choice themselves.

>     * Other solutions?
>     * Does it make sense to gate on the lowest version of libvirt but
> the highest version of python libs?

I think we should be consistent and at very least add testing of the
lowest python lib versions, so we can be confident that our declared
min versions are actually capable of working.

>     * Given our finite resources what gets us the furthest?

As you say above, testing the lowest vs highest is targetting two different
use cases. 

  - Testing the lowest version demonstrates that OpenStack has not
    broken its own code by introducing use of a new feature.

  - Testing the highest version demonstrates that OpenStack has not
    been broken by 3rd party code introducing a regression.

If I was to prioritize things given limited resources, I'd suggest that
we should be validating that OpenStack has not broken its own code as the
top priority. So testing lowest version would rank above testing highest
version. I do think that both a very important to openstack though.

So if we have the resources to cope, I do think it would be very valuable
if we were able to have 2 sets of jobs, one focused on the highest version
and one focused on the lowest version both gating.

Of course, there are also intermediate versions to worry about. Given
that our test suite is fully opensource, I think it is pretty reasonable
to say that distro vendors / other downstream consumers should take the
responsibility to testing any intermediate versions that are in their
own distro.

One might argue that distros should also have responsibility for testing
the highest version, but I think there is value in having openstack
keep that responsibility to avoid too much duplication of effort, and to
ensure that openstack releases keep a good reputation for quality operation
wrt latest versions of external deps.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|