Open Stack

>> Based on these experiences, libvirt version differences seem to be as
>> substantial as major hypervisor differences.
> 
> I think that is a pretty dubious conclusion to draw from just a
> couple of bugs. The reason they really caused pain is that because
> the CI test system was based on old version for too long.

I think the conclusion being made is that "libvirt versions two years
apart are effectively like different major versions of a hypervisor." I
don't think that's wrong.

> That is rather misleading statement you're making there. Libvirt is
> in fact held to *higher* standards than xen/vmware/hypver because it
> is actually gating all commits. The 3rd party CI systems can be
> broken for days, weeks and we still happily accept code for those
> virt. drivers.

Right, and we've talked about raising that bar as well, by tracking
their status more closely, automatically -2'ing patches that touch the
subdirectory but don't get a passing vote from the associated CI system,
etc.

You're definitely right that libvirt is held to a higher bar in terms of
it being required to pass tests before we can even mechanically land a
patch. However, there is a lot of function in the driver that we don't
test right now because of the version we're tied to in the gate nodes.
It's actually *easier* for a 3rd party system like vmware to roll their
environment and enable tests of newer features, so I don't think that
this requirement would cause existing 3rd party CI systems any trouble.

> AFAIK there has never been any statement that every feature added
> to xen/vmware/hyperv must be tested by the 3rd party CI system.

On almost every spec that doesn't already call it out, a reviewer asks
"how are you going to test this beyond just unit tests?" I think the
assumption and feeling among most reviewers is that new features,
especially that depend on new things (be it storage drivers, hypervisor
versions, etc) are concerned about approving without testing.

> AFAIK the requirement for 3rd party CI is merely that it has to exist,
> running some arbitrary version of the hypervisor in question. We've
> not said that 3rd party CI has to be covering every version or every
> feature, as is trying to be pushed on libvirt here.

The requirement in the past has been that it has to exist. At the last
summit, we had a discussion about how to raise the bar on what we
currently have. We made a lot of progress getting those systems
established (only because we had a requirement, by the way) in the last
cycle. Going forward, we need to have new levels of expectations in
terms of coverage and reliability of those things, IMHO.

> As above, aside from the question of gating vs non-gating, the bar is
> already set at the same level of everyone. There has to be a CI system
> somewhere testing some arbitrary version of the software. Everyone meets
> that requirement.

Wording our current requirement as you have here makes it sound like an
"arbitrary" ticky mark, which saddens and kind of offends me. What we
currently have was a step in the right direction. It was a lot of work,
but it's by no means arbitrary nor sufficient, IMHO.

--Dan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 538 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140716/aa084ba1/attachment.pgp>