Open Stack

On 02/21/2014 11:02 AM, Daniel P. Berrange wrote:
> On Fri, Feb 21, 2014 at 10:46:22AM -0500, Sean Dague wrote:
>> On 02/21/2014 09:45 AM, Daniel P. Berrange wrote:
>>> On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
>>>>
>>>> So I'm one of the first people to utter "if it isn't tested, it's
>>>> probably broken", however I also think we need to be realistic about the
>>>> fact that if you did out the permutations of dependencies and config
>>>> options, we'd have as many test matrix scenarios as grains of sand on
>>>> the planet.
>>>>
>>>> I do think in some ways this is unique to OpenStack, in that our
>>>> automated testing is head and shoulders above any other Open Source
>>>> project out there, and most proprietary software systems I've seen.
>>>>
>>>> So this is about being pragmatic. In our dependency testing we are
>>>> actually testing with most recent versions of everything. So I would
>>>> think that even with libvirt, we should err in that direction.
>>>
>>> I'm very much against that, because IME, time & time again across
>>> all open source projects I've worked on, people silently introduce
>>> use of features/apis that only exist in newer versions without anyone
>>> ever noticing until it is too late.
>>>
>>>> That being said, we also need to be a little bit careful about taking
>>>> such a hard line about "supported vs. not" based on only what's in the
>>>> gate. Because if we did the following things would be listed as
>>>> unsupported (in increasing level of ridiculousness):
>>>>
>>>>  * Live migration
>>>>  * Using qpid or zmq
>>>>  * Running on anything other than Ubuntu 12.04
>>>>  * Running on multiple nodes
>>>>
>>>> Supported to me means we think it should work, and if it doesn't, it's a
>>>> high priority bug that will get fixed quickly. Testing is our sanity
>>>> check. But it can't be considered that it will catch everything, at
>>>> least not before the heat death of the universe.
>>>
>>> I agree we should be pragmatic here to some extent. We shouldn't aim to
>>> test every single intermediate version, or every possible permutation of
>>> versions - just a representative sample. Testing both lowest and highest
>>> versions is a reasonable sample set IMHO.
>>
>> Testing lower bounds is interesting, because of the way pip works. That
>> being said, if someone wants to take ownership of building that job to
>> start as a periodic job, I'm happy to point in the right direction. Just
>> right now, it's a lower priority item than things like Tempest self
>> testing, Heat actually gating, Neutron running in parallel, Nova API
>> coverage.
> 
> If it would be hard work to do it for python modules, we can at least
> not remove the existing testing of an old libvirt version - simply add
> an additional test with newer libvirt.

Simply adding a test with newer libvirt isn't all that simple at the end
of the day, as it requires building a new nodepool image. Because
getting new libvirt in the existing test environment means cloud
archive, and cloud archive means a ton of other new code as well. Plus
in Juno we're presumably going to jump to 14.04 as our test base, which
is going to be it's own big transition.

So, I'm not opposed, but I also think bifurcating libvirt testing is a
big enough change in the pipeline that it needs some pretty dedicated
folks looking at it, and the implications there in. This isn't just a
yaml change, set and forget it. And given where we are in the
development cycle, I'm not sure trying to keep the gate stable with a
new libvirt which we've known to be problematic, is the right time to do
this.

But, if someone is stepping up to work through it, can definitely mentor
them on the right places to be poking.

	-Sean

-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 547 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140221/92226d2a/attachment.pgp>