<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Aug 30, 2014 at 9:08 AM, Mark McLoughlin <span dir="ltr"><<a href="mailto:markmc@redhat.com" target="_blank">markmc@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Hey<br>
<br>
The libvirt version_cap debacle continues to come up in conversation and<br>
one perception of the whole thing appears to be:<br>
<br>
A controversial patch was "ninjaed" by three Red Hat nova-cores and<br>
then the same individuals piled on with -2s when a revert was proposed<br>
to allow further discussion.<br>
<br>
I hope it's clear to everyone why that's a pretty painful thing to hear.<br>
However, I do see that I didn't behave perfectly here. I apologize for<br>
that.<br>
<br>
In order to understand where this perception came from, I've gone back<br>
over the discussions spread across gerrit and the mailing list in order<br>
to piece together a precise timeline. I've appended that below.<br>
<br>
Some conclusions I draw from that tedious exercise:<br></blockquote><div><br></div><div>Thank you for going through and doing this.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
- Some people came at this from the perspective that we already have<br>
a firm, unwritten policy that all code must have functional written<br>
tests. Others see that "test all the things" is interpreted as a<br>
worthy aspiration, but is only one of a number of nuanced factors<br>
that needs to be taken into account when considering the addition of<br>
a new feature.<br></blockquote><div><br></div><div>Confusion over our testing policy sounds like the crux of one of the issues here. Having so many unwritten policies has led to confusion in the past which is why I started <a href="http://docs.openstack.org/developer/nova/devref/policies.html">http://docs.openstack.org/developer/nova/devref/policies.html</a>, hopefully by writing these things down in the future this sort of confusion will arise less often.</div>
<div><br></div><div>Until this whole debacle I didn't even know there was a dissenting opinion on what our testing policy is. In every conversation I have seen up until this point, the question was always how to raise the bar on testing. I don't expect us to be able to get to the bottom of this issue in a ML thread, but hopefully we can begin the testing policy conversation here so that we may be able to make a breakthrough and the summit.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
i.e. the former camp saw Dan Smith's devref addition as attempting<br>
to document an existing policy (perhaps even a more forgiving<br>
version of an existing policy), whereas other see it as a dramatic<br>
shift to a draconian implementation of "test all the things".<br>
<br>
- Dan Berrange, Russell and I didn't feel like we were "ninjaing a<br>
controversial patch" - you can see our perspective expressed in<br>
multiple places. The patch would have helped the "live snapshot"<br>
issue, and has other useful applications. It does not affect the<br>
broader testing debate.<br>
<br>
Johannes was a solitary voice expressing concerns with the patch,<br>
and you could see that Dan was particularly engaged in trying to<br>
address those concerns and repeating his feeling that the patch was<br>
orthogonal to the testing debate.<br>
<br>
That all being said - the patch did merge too quickly.<br>
<br>
- What exacerbates the situation - particularly when people attempt to<br>
look back at what happened - is how spread out our conversations<br>
are. You look at the version_cap review and don't see any of the<br>
related discussions on the devref policy review nor the mailing list<br>
threads. Our disjoint methods of communicating contribute to<br>
misunderstandings.<br>
<br>
- When it came to the revert, a couple of things resulted in<br>
misunderstandings, hurt feelings and frayed tempers - (a) that our<br>
"retrospective veto revert policy" wasn't well understood and (b)<br>
a feeling that there was private, in-person grumbling about us at<br>
the mid-cycle while we were absent, with no attempt to talk to us<br>
directly.<br></blockquote><div><br></div><div>While I cannot speak for anyone else, I did grumble a bit at the mid-cycle about the behavior on Dan's first devref patch, <a href="https://review.openstack.org/#/c/103923">https://review.openstack.org/#/c/103923</a>/. This was the first time I saw 3 '-2's on a single patch revision. To me 1 or 2 '-2's gives the perception of 'hold on there, lets discuss this more first,' but 3 '-2's is just piling on and is very confrontational in nature. I was taken aback by this behavior and still don't know what to say or even if my reaction is justified.<br>
</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<br>
To take an even further step back - successful communities like ours<br>
require a huge amount of trust between the participants. Trust requires<br>
communication and empathy. If communication breaks down and the pressure<br>
we're all under erodes our empathy for each others' positions, then<br>
situations can easily get horribly out of control.<br>
<br>
This isn't a pleasant situation and we should all strive for better.<br>
However, I tend to measure our "flamewars" against this:<br>
<br>
<a href="https://mail.gnome.org/archives/gnome-2-0-list/2001-June/msg00132.html" target="_blank">https://mail.gnome.org/archives/gnome-2-0-list/2001-June/msg00132.html</a><br>
<br>
GNOME in June 2001 was my introduction to full-time open-source<br>
development, so this episode sticks out in my mind. The two individuals<br>
in that email were/are immensely capable and reasonable people, yet ...<br>
<br>
So far, we're doing pretty okay compared to that and many other<br>
open-source flamewars. Let's make sure we continue that way by avoiding<br>
letting situations fester.<br>
<br>
<br>
Thanks, and sorry for being a windbag,<br>
Mark.<br>
<br>
---<br>
<br>
= July 1 =<br>
<br>
The starting point is this review:<br>
<br>
<a href="https://review.openstack.org/103923" target="_blank">https://review.openstack.org/103923</a><br>
<br>
Dan Smith proposes a policy that the libvirt driver may not use libvirt<br>
features until they have been available in Ubuntu or Fedora for at least<br>
30 days.<br>
<br>
The commit message mentions:<br>
<br>
"broken us in the past when we add a new feature that requires a newer<br>
libvirt than we test with, and we discover that it's totally broken<br>
when we upgrade in the gate."<br>
<br>
which AIUI is a reference to the libvirt "live snapshot" issue the<br>
previous week, which is described here:<br>
<br>
<a href="https://review.openstack.org/102643" target="_blank">https://review.openstack.org/102643</a><br>
<br>
where upgrading to Ubuntu Trusty meant the libvirt version in use in the<br>
gate went from 0.9.8 to 1.2.2, which caused the "live snapshot" code<br>
paths in Nova for the first time, which appeared to be related to some<br>
serious gate instability (although the exact root cause wasn't<br>
identified).<br>
<br>
Some background on the libvirt version upgrade can be seen here:<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-March/thread.html#30284" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-March/thread.html#30284</a><br>
<br>
= July 1 - July 8 =<br>
<br>
Back and forth debate mostly between Dan Smith and Dan Berrange. Sean<br>
votes +2, Dan Berrange votes -2.<br>
<br>
= July 14 =<br>
<br>
Russell adds his support to Dan Berrange's position, votes -2. Some<br>
debate between Dan and Dan continues. Joe Gordon votes +2. Matt<br>
Riedemann expresses support-in-principal for Dan Smith's approach.<br>
<br>
= July 15 =<br>
<br>
Debate continues ...<br>
<br>
16:12 - I -2 the patch and attempt to take a step back and think about<br>
how we could have prevented (or at least mitigated against) the "live<br>
snapshot" issue and suggest the idea of adding a new configuration<br>
option:<br>
<br>
[libvirt]<br>
version_cap = 1.2.2<br>
<br>
which would mean we would not automatically start using new libvirt<br>
features in the gate because of a libvirt version upgrade, but instead<br>
the new features would only begin to be used when we merge a change to<br>
the default value of version_cap.<br>
<br>
16:31 - I leave a separate comment addressing the broader debate about<br>
our functional test coverage requirements.<br>
<br>
16:46 - Dan Berrange likes the version_cap idea<br>
<br>
15:37 - Dan Berrange posts an implementation of version_cap:<br>
<br>
<a href="https://review.openstack.org/107119" target="_blank">https://review.openstack.org/107119</a><br>
<br>
and links to it from in Dan Smith's libvirt testing policy review (#103923)<br>
<br>
21:49 - Matt expresses some support for the config option, but worries<br>
about the precedent being set.<br>
<br>
23:14 - Dan Berrange explains his point of view that a "test all the<br>
things" rule must mean "test all the things which can be practically<br>
tested by our current CI system".<br>
<br>
= July 16 =<br>
<br>
08:04 - I +2 the version_cap patch after Dan fixes up some issues I<br>
pointed out.<br>
<br>
13:44 - 14:28 - Sean and John Garbutt add further thoughts to the<br>
libvirt testing policy review without making any comment on the<br>
version_cap idea. Sean takes the debate to the mailing list:<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040421.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040421.html</a><br>
<br>
Debate continues in the thread, largely around the mechanics of how to<br>
allow a newer version of libvirt be used in the gate.<br>
<br>
15:08 - I mention the version_cap proposal on the thread for the first<br>
time:<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040436.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040436.html</a><br>
<br>
and the point I make is the configuration option makes it easier for<br>
operators to run only code paths that are tested by the gate.<br>
<br>
16:44 - Johannes notes that multiple issues with code paths not tested<br>
in the gate may need to be fixed as part of a future review to increase<br>
the default value of version_cap.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040456.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040456.html</a><br>
<br>
18:50 - Russell approves the version_cap patch.<br>
<br>
<a href="https://review.openstack.org/107119" target="_blank">https://review.openstack.org/107119</a><br>
<br>
= July 17 =<br>
<br>
05:38 - The version_cap patch merges.<br>
<br>
13:09 - Somewhat related, Dan Berrange and I explain we won't be at the<br>
mid-cycle issue for any test policy discussions. Sean makes a point that<br>
the discussion is best had on email/IRC where there is a permanent<br>
record.<br>
<br>
14:33 - Johannes expresses concern in gerrit that version_cap got merged<br>
too quickly.<br>
<br>
<a href="https://review.openstack.org/107119" target="_blank">https://review.openstack.org/107119</a><br>
<br>
15:17 - Dan Berrange responds to Johannes in gerrit, saying that he<br>
thinks version_cap is useful irrespective of the broader testing<br>
discussion.<br>
<br>
15:28 - Johannes disagrees, asks for a response to his concerns on the<br>
mailing list.<br>
<br>
15:40 - Dan Berrange responds on the mailing list to Johannes<br>
version_cap concerns.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040576.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040576.html</a><br>
<br>
18:15 - Russell also responds to Johannes.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040597.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040597.html</a><br>
<br>
18:31 - Johannes responds to Dan.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040602.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040602.html</a><br>
<br>
18:39 - Russell responds to Johannes again.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040604.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040604.html</a><br>
<br>
19:13 - Johannes responds again.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040608.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040608.html</a><br>
<br>
= July 18 =<br>
<br>
07:55 - Dan Berrange responds to Johannes again.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040647.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040647.html</a><br>
<br>
09:37 - Sean notices version_cap, thinks it's "interesting", wonders do<br>
we need similar for qemu versions.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-July/040533.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-July/040533.html</a><br>
<br>
= July 28 - 30 =<br>
<br>
The Nova mid-cycle meetup in Portland. Neither, Dan, Russell nor I are<br>
present, for various personal reasons.<br>
<br>
AIUI, a brief "hallway track" conversation lead to briefly discussing<br>
the version_cap patch in the main sessions which went along the lines of<br>
"does anyone find it concerning how this patch was merged" and the<br>
general consensus was "yes, it is concerning".<br>
<br>
= July 30 =<br>
<br>
Sean Dague proposes a revert of the patch because a) "it's a policy<br>
change" and b) it requires further discussion.<br>
<br>
<a href="https://review.openstack.org/110754" target="_blank">https://review.openstack.org/110754</a><br>
<br>
= Aug 7 =<br>
<br>
Russell -2s the revert.<br>
<br>
Matt raises the topic of the revert on the mailing list.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-August/042284.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-August/042284.html</a><br>
<br>
Dan Smith, Joe Gordon +2s the revert.<br>
<br>
= Aug 8 =<br>
<br>
Michael Still, Jay Pipes +2s the revert.<br>
<br>
Russell starts experimenting with a gate job to test with latest<br>
libvirt.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-August/042470.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-August/042470.html</a><br>
<br>
= Aug 11 =<br>
<br>
Dan Berrange and I return from our (separate :) vacations and -2 the<br>
revert.<br>
<br>
I describe the revert as a "proxy battle", avoiding the real debate<br>
around testing requirements.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-August/042546.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-August/042546.html</a><br>
<br>
Joe tries to capture the concerns about how the version_cap patch was<br>
originally merged.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-August/042653.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-August/042653.html</a><br>
<br>
= Aug 12 =<br>
<br>
I explicitly attempt to address any notion there is a "corporate agenda"<br>
at play here.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-August/042691.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-August/042691.html</a><br>
<br>
Dan again tries to explain why he thinks version_cap is orthogonal to<br>
the broader debate.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-August/042707.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-August/042707.html</a><br>
<br>
Michael Still - with Thierry cc-ed - privately emails me, Russell and<br>
Dan Berrange asking us to remove our -2s so that we can have a "more<br>
complete discussion". All three of us initially refuse and a bunch of<br>
hurt feelings are frankly expressed on all sides, with one theme being<br>
the perception that this had gotten out of control because we weren't at<br>
the mid-cycle.<br>
<br>
Thierry calmed things down be patiently expressing Michael's perspective<br>
that there is a process to be followed where patches that get merged<br>
quickly are later seen to be controversial. The red mist begins to<br>
clear. I propose that we document this "retrospective veto revert"<br>
procedure to head off any future misunderstanding.<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2014-August/042728.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-August/042728.html</a><br>
<br>
All three of us remove our -2s. The patch gets reverted.<br>
<br>
= Aug 14 =<br>
<br>
Dan Berrange re-proposes the patch, later changing the default<br>
version_cap value.<br>
<br>
<a href="https://review.openstack.org/114307" target="_blank">https://review.openstack.org/114307</a><br>
<br>
The patch has seen very little discussion since then and is still<br>
pending review.<br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div></div>