Open Stack

On Jul 28, 2014, at 2:52 AM, Angus Lees <gus at inodes.org> wrote:

> On Mon, 21 Jul 2014 04:39:49 PM David Kranz wrote:
>> On 07/21/2014 04:13 PM, Jay Pipes wrote:
>>> On 07/21/2014 02:03 PM, Clint Byrum wrote:
>>>> Thanks Matthew for the analysis.
>>>> 
>>>> I think you missed something though.
>>>> 
>>>> Right now the frustration is that unrelated intermittent bugs stop your
>>>> presumably good change from getting in.
>>>> 
>>>> Without gating, the result would be that even more bugs, many of them
>>>> not
>>>> intermittent at all, would get in. Right now, the one random developer
>>>> who has to hunt down the rechecks and do them is inconvenienced. But
>>>> without a gate, _every single_ developer will be inconvenienced until
>>>> the fix is merged.
>>>> 
>>>> The false negative rate is _way_ too high. Nobody would disagree there.
>>>> However, adding more false negatives and allowing more people to ignore
>>>> the ones we already have, seems like it would have the opposite effect:
>>>> Now instead of annoying the people who hit the random intermittent bugs,
>>>> we'll be annoying _everybody_ as they hit the non-intermittent ones.
>>> 
>>> +10
>> 
>> Right, but perhaps there is a middle ground. We must not allow changes
>> in that can't pass through the gate, but we can separate the problems
>> of constant rechecks using too many resources, and of constant rechecks
>> causing developer pain. If failures were deterministic we would skip the
>> failing tests until they were fixed. Unfortunately many of the common
>> failures can blow up any test, or even the whole process. Following on
>> what Sam said, what if we automatically reran jobs that failed in a
>> known way, and disallowed "recheck/reverify no bug"? Developers would
>> then have to track down what bug caused a failure or file a new one. But
>> they would have to do so much less frequently, and as more common
>> failures were catalogued it would become less and less frequent.
>> 
>> Some might (reasonably) argue that this would be a bad thing because it
>> would reduce the incentive for people to fix bugs if there were less
>> pain being inflicted. But given how hard it is to track down these race
>> bugs, and that we as a community have no way to force time to be spent
>> on them, and that it does not appear that these bugs are causing real
>> systems to fall down (only our gating process), perhaps something
>> different should be considered?
> 
> So to pick an example dear to my heart, I've been working on removing these 
> gate failures:
> http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA2NTI3OTA3NzkzfQ==
> 
> .. caused by a bad interaction between eventlet and our default choice of 
> mysql driver.  It would also affect any real world deployment using mysql.
> 
> The problem has been identified and the fix proposed for almost a month now, but 
> actually fixing the gate jobs is still no-where in sight.  The fix is (pretty 
> much) as easy as a pip install and a slightly modified database connection 
> string.
> I look forward to a discussion of the meta-issues surrounding this, but it is 
> not because no-one tracked down or fixed the bug :(

I believe the main blocking issue right now is that Oracle doesn’t upload that library to PyPI, and so our build-chain won’t be able to download it as it is currently configured. I think the last I saw someone was going to talk to Oracle about uploading the source. Have we heard back?

Doug

> 
> - Gus
> 
>>  -David
>> 
>>> Best,
>>> -jay
>>> 
>>>> Excerpts from Matthew Booth's message of 2014-07-21 03:38:07 -0700:
>>>>> On Friday evening I had a dependent series of 5 changes all with
>>>>> approval waiting to be merged. These were all refactor changes in the
>>>>> VMware driver. The changes were:
>>>>> 
>>>>> * VMware: DatastorePath join() and __eq__()
>>>>> https://review.openstack.org/#/c/103949/
>>>>> 
>>>>> * VMware: use datastore classes get_allowed_datastores/_sub_folder
>>>>> https://review.openstack.org/#/c/103950/
>>>>> 
>>>>> * VMware: use datastore classes in file_move/delete/exists, mkdir
>>>>> https://review.openstack.org/#/c/103951/
>>>>> 
>>>>> * VMware: Trivial indentation cleanups in vmops
>>>>> https://review.openstack.org/#/c/104149/
>>>>> 
>>>>> * VMware: Convert vmops to use instance as an object
>>>>> https://review.openstack.org/#/c/104144/
>>>>> 
>>>>> The last change merged this morning.
>>>>> 
>>>>> In order to merge these changes, over the weekend I manually submitted:
>>>>> 
>>>>> * 35 rechecks due to false negatives, an average of 7 per change
>>>>> * 19 resubmissions after a change passed, but its dependency did not
>>>>> 
>>>>> Other interesting numbers:
>>>>> 
>>>>> * 16 unique bugs
>>>>> * An 87% false negative rate
>>>>> * 0 bugs found in the change under test
>>>>> 
>>>>> Because we don't fail fast, that is an average of at least 7.3 hours in
>>>>> the gate. Much more in fact, because some runs fail on the second pass,
>>>>> not the first. Because we don't resubmit automatically, that is only if
>>>>> a developer is actively monitoring the process continuously, and
>>>>> resubmits immediately on failure. In practise this is much longer,
>>>>> because sometimes we have to sleep.
>>>>> 
>>>>> All of the above numbers are counted from the change receiving an
>>>>> approval +2 until final merging. There were far more failures than this
>>>>> during the approval process.
>>>>> 
>>>>> Why do we test individual changes in the gate? The purpose is to find
>>>>> errors *in the change under test*. By the above numbers, it has failed
>>>>> to achieve this at least 16 times previously.
>>>>> 
>>>>> Probability of finding a bug in the change under test: Small
>>>>> Cost of testing:                                       High
>>>>> Opportunity cost of slowing development:               High
>>>>> 
>>>>> and for comparison:
>>>>> 
>>>>> Cost of reverting rare false positives:                Small
>>>>> 
>>>>> The current process expends a lot of resources, and does not achieve
>>>>> its
>>>>> goal of finding bugs *in the changes under test*. In addition to
>>>>> using a
>>>>> lot of technical resources, it also prevents good change from making
>>>>> its
>>>>> way into the project and, not unimportantly, saps the will to live of
>>>>> its victims. The cost of the process is overwhelmingly greater than its
>>>>> benefits. The gate process as it stands is a significant net
>>>>> negative to
>>>>> the project.
>>>>> 
>>>>> Does this mean that it is worthless to run these tests? Absolutely not!
>>>>> These tests are vital to highlight a severe quality deficiency in
>>>>> OpenStack. Not addressing this is, imho, an existential risk to the
>>>>> project. However, the current approach is to pick contributors from the
>>>>> community at random and hold them personally responsible for project
>>>>> bugs selected at random. Not only has this approach failed, it is
>>>>> impractical, unreasonable, and poisonous to the community at large. It
>>>>> is also unrelated to the purpose of gate testing, which is to find bugs
>>>>> *in the changes under test*.
>>>>> 
>>>>> I would like to make the radical proposal that we stop gating on CI
>>>>> failures. We will continue to run them on every change, but only after
>>>>> the change has been successfully merged.
>>>>> 
>>>>> Benefits:
>>>>> * Without rechecks, the gate will use 8 times fewer resources.
>>>>> * Log analysis is still available to indicate the emergence of races.
>>>>> * Fixes can be merged quicker.
>>>>> * Vastly less developer time spent monitoring gate failures.
>>>>> 
>>>>> Costs:
>>>>> * A rare class of merge bug will make it into master.
>>>>> 
>>>>> Note that the benefits above will also offset the cost of resolving
>>>>> this
>>>>> rare class of merge bug.
>>>>> 
>>>>> Of course, we still have the problem of finding resources to monitor
>>>>> and
>>>>> fix CI failures. An additional benefit of not gating on CI will be that
>>>>> we can no longer pretend that picking developers for project-affecting
>>>>> bugs by lottery is likely to achieve results. As a project we need to
>>>>> understand the importance of CI failures. We need a proper negotiation
>>>>> with contributors to staff a team dedicated to the problem. We can then
>>>>> use the review process to ensure that the right people have an
>>>>> incentive
>>>>> to prioritise bug fixes.
>>>>> 
>>>>> Matt
>>>> 
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> -- 
> - Gus
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev