Open Stack

On 02/20/2014 01:31 PM, Joe Gordon wrote:
> Hi All,
> 
> I discussion recently came up inside of nova about what it means
> supported version for a dependency means.  in libvirt we gate on the
> minimal version that we support but for all python dependencies we
> gate on the highest version that passes our requirements. While we all
> agree that having two different ways of choosing which version to test
> (min and max) is bad, there are good arguments for doing both.
> 
> testing most recent version:
>     * We want to make sure we support the latest and greatest
>     * Bug fixes
>     * Quickly discover backwards incompatible changes so we can deal
> with them as they arise instead of in batch
> 
> Testing lowest version supported:
>     * Make sure we don't land any code that breaks compatibility with
> the lowest version we say we support
> 
> 
> A few questions and ideas on how to move forward.
>     * How do other projects deal with this? This problem isn't unique
> in OpenStack.
>     * What are the issues with making one gate job use the latest
> versions and one use the lowest supported versions?
>     * Only test some things on every commit or every day (periodic
> jobs)? But no one ever fixes those things when they break? who wants
> to own them? distros? deployers?
>     * Other solutions?
>     * Does it make sense to gate on the lowest version of libvirt but
> the highest version of python libs?
>     * Given our finite resources what gets us the furthest?

So I'm one of the first people to utter "if it isn't tested, it's
probably broken", however I also think we need to be realistic about the
fact that if you did out the permutations of dependencies and config
options, we'd have as many test matrix scenarios as grains of sand on
the planet.

I do think in some ways this is unique to OpenStack, in that our
automated testing is head and shoulders above any other Open Source
project out there, and most proprietary software systems I've seen.

So this is about being pragmatic. In our dependency testing we are
actually testing with most recent versions of everything. So I would
think that even with libvirt, we should err in that direction.

That being said, we also need to be a little bit careful about taking
such a hard line about "supported vs. not" based on only what's in the
gate. Because if we did the following things would be listed as
unsupported (in increasing level of ridiculousness):

 * Live migration
 * Using qpid or zmq
 * Running on anything other than Ubuntu 12.04
 * Running on multiple nodes

Supported to me means we think it should work, and if it doesn't, it's a
high priority bug that will get fixed quickly. Testing is our sanity
check. But it can't be considered that it will catch everything, at
least not before the heat death of the universe.

	-Sean

-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140220/6325e6ad/attachment.pgp>