Open Stack

On 28.02.2019 17:35, Bogdan Dobrelya wrote:
> On 28.02.2019 17:22, Clark Boylan wrote:
>> On Thu, Feb 28, 2019, at 6:16 AM, Bogdan Dobrelya wrote:
>>>> On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote:
>>>>> On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote:
>>>>>
>>>>
>>>> snip
>>>>
>>>>> That said, I wouldn't push too hard in either direction until someone
>>>>> crunched the numbers and figured out how much time it would have saved
>>>>> to not run long tests on patch sets with failing unit tests. I feel 
>>>>> like
>>>>> it's probably possible to figure that out, and if so then we should do
>>>>> it before making any big decisions on this.
>>>>
>>>> For numbers the elastic-recheck tool [0] gives us fairly accurate 
>>>> tracking of which issues in the system cause tests to fail. You can 
>>>> use this as a starting point to potentially figure out how expensive 
>>>> indentation errors caught by the pep8 jobs ends up being or how 
>>>> often unittests fail. You probably need to tweak the queries there 
>>>> to get that specific though.
>>>>
>>>> Periodically I also dump node resource utilization by project, repo, 
>>>> and job [1]. I haven't automated this because Tobiash has written a 
>>>> much better thing that has Zuul inject this into graphite and we 
>>>> should be able to set up a grafana dashboard for that in the future 
>>>> instead.
>>>>
>>>> These numbers won't tell a whole story, but should paint a fairly 
>>>> accurate high level picture of the types of things we should look at 
>>>> to be more node efficient and "time in gate" efficient. Looking at 
>>>> these two really quickly myself it seems that job timeouts are a big 
>>>> cost (anyone looking into why our jobs timeout?).
>>>>
>>>> [0] http://status.openstack.org/elastic-recheck/index.html
>>>> [1] http://paste.openstack.org/show/746083/
>>>>
>>>> Hope this helps,
>>>> Clark
>>>
>>> Here is some numbers [0] extracted via elastic-recheck console queries.
>>> It shows 6% of wasted failures because of tox issues in general, and 3%
>>> for tripleo projects in particular.
>>
>> Are these wasted failures? The queries appear to be tracking valid 
>> failures of those jobs. These valid failures are then actionable 
>> feedback for developers to fix their changes.
> 
> I'll need more time to get through the ideas below and wrap it around my 
> heaf, but for this particular statement, I understand the CI pool 
> resources as "wasted" if executed for either of the failed tox jobs. The 
> reason why it fails is secondary from that standpoint...

ugh... I meant the CI resources spent for running other integration 
jobs, while still having an unmergable patch here. As I noted above, 
*some* of the integration jobs may still be there in early results, just 
that probably not *all* of them. That is I called "middle-ground".

> 
>>
>> If we expect these failures to go away we'll need to be much more 
>> forceful about getting developers to run tox locally before they push.
>>
>> We need to compare (and this is a rough example) the resource usage 
>> and developer time of complete batch results where you have a pep8 
>> issue and an integration job issue in patchset one, fix both and all 
>> tests pass in patchset two against pep8 failure in patchset one, 
>> integration failure in patchset two, all tests pass in patchset three.
>>
>> Today:
>>    patchset one:
>>      pep8 FAILURE
>>      unittest SUCCESS
>>      integration FAILURE
>>    patchset two:
>>      pep8 SUCCESS
>>      unittest SUCCESS
>>      integration SUCCESS
>>
>> Proposed Future:
>>    patchset one:
>>      pep8 FAILURE
>>      unittest SUCCESS
>>    patchset two:
>>      pep8 SUCCESS
>>      unittest SUCCESS
>>      integration FAILURE
>>    patchset three:
>>      pep8 SUCCESS
>>      unittest SUCCESS
>>      integration SUCCESS
>>
>> There are strictly more patchsets (developer roundtrips) and tests run 
>> in my contrived example. Reality will depend on how many iterations we 
>> actually see in the real world (eg are we fixing bugs reliably based 
>> on test feedback and how often do unittest and integration tests fail 
>> for different reasons).
>>
>> Someone with a better background in statistics will probably tell me 
>> this approach is wrong, but using the elasticsearch tooling one 
>> approach may be to pick say 1k changes, then for each change identify 
>> which tests failed on subsequent patchsets? Then we'd be able to infer 
>> with some confidence the behaviors we have in the test suites around 
>> catching failures, whether they are independent across jobs and 
>> whether or not we fix them in batches.
>>
>> Finally, I think we might also want to consider what the ideal is 
>> here. If we find we can optimize the current system for current 
>> behaviors, we also need to consider if that is worthwhile given an 
>> ideal. Should developers be expected to run unittests and linters 
>> locally before pushing? If so then optimizing for when they don't 
>> might be effort better spent on making it easier to run the unittests 
>> and linters locally and educating developers on how to do so. I think 
>> we'd also ideally expect our tests to pass once they run in the gate 
>> pipeline. Unfortunately I think our elastic-recheck data shows they 
>> often don't and more effort in fixing those failures would provide a 
>> dramatic increase in throughput (due to the compounding costs of 
>> subsequent gate resets).
>>
>>>
>>> My final take is, given some middle-ground solution, like I illustrated
>>> earlier this sub-thread, it might be worth it, and the effort for
>>> boosting up the total throughput of openstack CI system by a 6% is not
>>> so bad idea.
>>>
>>> [0] http://paste.openstack.org/show/746503/
>>>
>>>
>>> -- 
>>> Best regards,
>>> Bogdan Dobrelya,
>>> Irc #bogdando
>>>
>>>
>>
> 
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando