Open Stack

On Wed, Nov 17, 2021, at 2:18 AM, Balazs Gibizer wrote:
>

Snip. I want to respond to a specific suggestion:

> 3) there was informal discussion before about a possibility to re-run 
> only some jobs with a recheck instead for re-running the whole set. I 
> don't know if this is feasible with Zuul and I think this only treat 
> the symptom not the root case. But still this could be a direction if 
> all else fails.
>

OpenStack has configured its check and gate queues with something we've called "clean check". This refers to the requirement that before an OpenStack project can be gated it must pass check tests first. This policy was instituted because a number of these infrequent but problematic issues were traced back to recheck spamming. Basically changes would show up and were broken. They would fail some percentage of the time. They got rechecked until they finally merged and now their failure rate is added to the whole. This rule was introduced to make it more difficult to get this flakyness into the gate.

Locking in test results is in direct opposition to the existing policy and goals. Locking results would make it far more trivial to land such flakyness as you wouldn't need entire sets of jobs to pass before you could land. Instead you could rerun individual jobs until each one passed and then land the result. Potentially introducing significant flakyness with a single merge.

Locking results is also not really something that fits well with the speculative gate queues that Zuul runs. Remember that Zuul constructs a future git state and tests that in parallel. Currently the state for OpenStack looks like:

  A - Nova
  ^
  B - Glance
  ^
  C - Neutron
  ^
  D - Neutron
  ^
  F - Neutron

The B glance change is tested as if the A Nova change has already merged and so on down the queue. If we want to keep these speculative states we can't really have humans manually verify a failure can be ignored and retry it. Because we'd be enqueuing job builds at different stages of speculative state. Each job build would be testing a different version of the software.

What we could do is implement a retry limit for failing jobs. Zuul could rerun failing jobs X times before giving up and reporting failure (this would require updates to Zuul). The problem with this approach is without some oversight it becomes very easy to land changes that make things worse. As a side note Zuul does do retries, but only for detected network errors or when a pre-run playbook fails. The assumption is that network failures are due to the dangers of the Internet, and that pre-run playbooks are small, self contained, unlikely to fail, and when they do fail the failure should be independent of what is being tested.

Where does that leave us?

I think it is worth considering the original goals of "clean check". We know that rechecking/rerunning only makes these problems worse in the long term. They represent technical debt. One of the reasons we run these tests is to show us when our software is broken. In the case of flaky results we are exposing this technical debt where it impacts the functionality of our software. The longer we avoid fixing these issues the worse it gets, and this is true even with "clean check".

Do we as developers find value in knowing the software needs attention before it gets released to users? Do the users find value in running reliable software? In the past we have asserted that "yes, there is value in this", and have invested in tracking, investigating, and fixing these problems even if they happen infrequently. But that does require investment, and active maintenance.

Clark