[openstack-dev] supported dependency versioning and testing

Sean Dague sean at dague.net
Fri Feb 21 15:46:22 UTC 2014


On 02/21/2014 09:45 AM, Daniel P. Berrange wrote:
> On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
>>
>> So I'm one of the first people to utter "if it isn't tested, it's
>> probably broken", however I also think we need to be realistic about the
>> fact that if you did out the permutations of dependencies and config
>> options, we'd have as many test matrix scenarios as grains of sand on
>> the planet.
>>
>> I do think in some ways this is unique to OpenStack, in that our
>> automated testing is head and shoulders above any other Open Source
>> project out there, and most proprietary software systems I've seen.
>>
>> So this is about being pragmatic. In our dependency testing we are
>> actually testing with most recent versions of everything. So I would
>> think that even with libvirt, we should err in that direction.
> 
> I'm very much against that, because IME, time & time again across
> all open source projects I've worked on, people silently introduce
> use of features/apis that only exist in newer versions without anyone
> ever noticing until it is too late.
> 
>> That being said, we also need to be a little bit careful about taking
>> such a hard line about "supported vs. not" based on only what's in the
>> gate. Because if we did the following things would be listed as
>> unsupported (in increasing level of ridiculousness):
>>
>>  * Live migration
>>  * Using qpid or zmq
>>  * Running on anything other than Ubuntu 12.04
>>  * Running on multiple nodes
>>
>> Supported to me means we think it should work, and if it doesn't, it's a
>> high priority bug that will get fixed quickly. Testing is our sanity
>> check. But it can't be considered that it will catch everything, at
>> least not before the heat death of the universe.
> 
> I agree we should be pragmatic here to some extent. We shouldn't aim to
> test every single intermediate version, or every possible permutation of
> versions - just a representative sample. Testing both lowest and highest
> versions is a reasonable sample set IMHO.

Testing lower bounds is interesting, because of the way pip works. That
being said, if someone wants to take ownership of building that job to
start as a periodic job, I'm happy to point in the right direction. Just
right now, it's a lower priority item than things like Tempest self
testing, Heat actually gating, Neutron running in parallel, Nova API
coverage.

	-Sean

-- 
Sean Dague
http://dague.net



More information about the OpenStack-dev mailing list