Open Stack

On Wed, Nov 17 2021 at 07:51:57 AM -0800, Clark Boylan 
<cboylan at sapwetik.org> wrote:
> On Wed, Nov 17, 2021, at 2:18 AM, Balazs Gibizer wrote:
>> 
> 
> Snip. I want to respond to a specific suggestion:
> 
>>  3) there was informal discussion before about a possibility to 
>> re-run
>>  only some jobs with a recheck instead for re-running the whole set. 
>> I
>>  don't know if this is feasible with Zuul and I think this only treat
>>  the symptom not the root case. But still this could be a direction 
>> if
>>  all else fails.
>> 
> 
> OpenStack has configured its check and gate queues with something 
> we've called "clean check". This refers to the requirement that 
> before an OpenStack project can be gated it must pass check tests 
> first. This policy was instituted because a number of these 
> infrequent but problematic issues were traced back to recheck 
> spamming. Basically changes would show up and were broken. They would 
> fail some percentage of the time. They got rechecked until they 
> finally merged and now their failure rate is added to the whole. This 
> rule was introduced to make it more difficult to get this flakyness 
> into the gate.
> 
> Locking in test results is in direct opposition to the existing 
> policy and goals. Locking results would make it far more trivial to 
> land such flakyness as you wouldn't need entire sets of jobs to pass 
> before you could land. Instead you could rerun individual jobs until 
> each one passed and then land the result. Potentially introducing 
> significant flakyness with a single merge.
> 
> Locking results is also not really something that fits well with the 
> speculative gate queues that Zuul runs. Remember that Zuul constructs 
> a future git state and tests that in parallel. Currently the state 
> for OpenStack looks like:
> 
>   A - Nova
>   ^
>   B - Glance
>   ^
>   C - Neutron
>   ^
>   D - Neutron
>   ^
>   F - Neutron
> 
> The B glance change is tested as if the A Nova change has already 
> merged and so on down the queue. If we want to keep these speculative 
> states we can't really have humans manually verify a failure can be 
> ignored and retry it. Because we'd be enqueuing job builds at 
> different stages of speculative state. Each job build would be 
> testing a different version of the software.
> 
> What we could do is implement a retry limit for failing jobs. Zuul 
> could rerun failing jobs X times before giving up and reporting 
> failure (this would require updates to Zuul). The problem with this 
> approach is without some oversight it becomes very easy to land 
> changes that make things worse. As a side note Zuul does do retries, 
> but only for detected network errors or when a pre-run playbook 
> fails. The assumption is that network failures are due to the dangers 
> of the Internet, and that pre-run playbooks are small, self 
> contained, unlikely to fail, and when they do fail the failure should 
> be independent of what is being tested.
> 
> Where does that leave us?
> 
> I think it is worth considering the original goals of "clean check". 
> We know that rechecking/rerunning only makes these problems worse in 
> the long term. They represent technical debt. One of the reasons we 
> run these tests is to show us when our software is broken. In the 
> case of flaky results we are exposing this technical debt where it 
> impacts the functionality of our software. The longer we avoid fixing 
> these issues the worse it gets, and this is true even with "clean 
> check".
> 
> Do we as developers find value in knowing the software needs 
> attention before it gets released to users? Do the users find value 
> in running reliable software? In the past we have asserted that "yes, 
> there is value in this", and have invested in tracking, 
> investigating, and fixing these problems even if they happen 
> infrequently. But that does require investment, and active 
> maintenance.

Thank you Clark! I agree with your view that the current setup provides 
us with very valuable information about the health of the software we 
are developing. I also agree that our primary goal should be to fix the 
flaky tests instead of hiding the results under any kind of rechecks.

Still I'm wondering what we will do if it turns out that the existing 
developer bandwidth shrunk to the point where we simply not have the 
capacity for fix these technical debts. What the stable team does on 
stable branches in Extended Maintenance mode in a similar situation is 
to simply turn off problematic test jobs. So I guess that is also a 
valid last resort move.

Cheers,
gibi




> 
> Clark
>