Open Stack

On Wed, 2014-07-16 at 16:15 +0200, Sean Dague wrote:
..
> Based on these experiences, libvirt version differences seem to be as
> substantial as major hypervisor differences. There is a proposal here -
> https://review.openstack.org/#/c/103923/ to hold newer versions of
> libvirt to the same standard we hold xen, vmware, hyperv, docker,
> ironic, etc.

That's a bit of a mis-characterization - in terms of functional test
coverage, the libvirt driver is the bar that all the other drivers
struggle to meet.

And I doubt any of us pay too close attention to the feature coverage
that the 3rd party CI test jobs have.

> I'm somewhat concerned that the -2 pile on in this review is a double
> standard of libvirt features, and features exploiting really new
> upstream features. I feel like a lot of the language being used here
> about the burden of doing this testing is exactly the same as was
> presented by the docker team before their driver was removed, which was
> ignored by the Nova team at the time.

Personally, I wasn't very comfortable with the docker driver move. It
certainly gave an outward impression that we're an unfriendly community.
The mitigating factor was that a lot of friendly, collaborative,
coaching work went on in the background for months. Expectations were
communicated well in advance.

Kicking the docker driver out of the tree has resulted in an uptick in
the amount of work happening on it, but I suspect most people involved
have a bad taste in their mouths. I guess there's incentives at play
which mean they'll continue plugging away at it, but those incentives
aren't always at play.

> It was the concern by the freebsd
> team, which was also ignored and they were told to go land libvirt
> patches instead.
> 
> I'm ok with us as a project changing our mind and deciding that the test
> bar needs to be taken down a notch or two because it's too burdensome to
> contributors and vendors, but if we are doing that, we need to do it for
> everyone. A lot of other organizations have put a ton of time and energy
> into this, and are carrying a maintenance cost of running these systems
> to get results back in a timely basis.

I don't agree that we need to apply the same rules equally to everyone.

At least part of the reasoning behind the emphasis on 3rd party CI
testing was that projects (Neutron in particular) were being overwhelmed
by contributions to drivers from developers who never contributed in any
way to the core. The corollary of that is the contributors who do
contribute to the core should be given a bit more leeway in return.

There's a natural building of trust and element of human relationships
here. As a reviewer, you learn to trust contributors with a good track
record and perhaps prioritize contributions from them.

> As we seem deadlocked in the review, I think the mailing list is
> probably a better place for this.
> 
> If we want to reduce the standards for libvirt we should reconsider
> what's being asked of 3rd party CI teams, and things like the docker
> driver, as well as the A, B, C driver classification. Because clearly
> libvirt 1.2.5+ isn't actually class A supported.

No, there are features or code paths of the libvirt 1.2.5+ driver that
aren't as well tested as the "class A" designation implies. And we have
a proposal to make sure these aren't used by default:

  https://review.openstack.org/107119

i.e. to stray off the "class A" path, an operator has to opt into it by
changing a configuration option that explains they will be enabling code
paths which aren't yet tested upstream.

These features have value to some people now, they don't risk regressing
the "class A" driver and there's a clear path to them being elevated to
"class A" in time. We should value these contributions and nurture these
contributors.

Appending some of my comments from the review below. The tl;dr is that I
think we're losing sight of the importance of welcoming and nurturing
contributors, and valuing whatever contributions they can make. That
terrifies me. 

Mark.

---

Compared to other open source projects, we have done an awesome job in
OpenStack of having good functional test coverage. Arguably, given the
complexity of the system, we couldn't have got this far without it. I
can take zero credit for any of it.

However, not everything is tested now, nor is the tests we have
foolproof. When you consider the number of configuration options we
have, the supported distros, the ranges of library versions we claim to
support, etc., etc. I don't think we can ever get to an "everything is
tested" point.

In the absence of that, I think we should aim to be more clear what *is*
tested. The config option I suggest does that, which is a big part of
its merit IMHO.

We've had some success with the "be nasty enough to driver contributors
and they'll do what we want" approach so far, but IMHO that was an
exceptional approach for an exceptional situation - drivers that were
completely broken, and driver developers who didn't contribute to the
core project or, worse, didn't maintain their drivers adequately.

Not every contributor can be bullied like this. And even where it
succeeds, I don't much love the perception it gives the project.

In this case, the effect I could imagine us having would be that rather
than us successfully bullying someone into setting up "latest libvirt
CI", we drive development of features requiring newer libvirt into a
separate tree.

There would even be a perfectly defensible rationale for doing this -
"we're building features that can't go upstream until a newer libvirt
shows up there". If these are NFV features, there'd likely be a bunch of
downstreams that ship it. And then when the newer libvirt shows up, all
the features requiring it get proposed together as fully-baked patch
sets? No thank you.

The desire to have the best possible test coverage is great. The desire
to document our policies around this stuff is great. But let's not
forget that we're talking about contributors that need to be nurtured
and contributions that should be valued.