[neutron][CI] How to reduce number of rechecks - brainstorming
skaplons at redhat.com
Wed Nov 17 08:13:34 UTC 2021
Recently I spent some time to check how many rechecks we need in Neutron to
get patch merged and I compared it to some other OpenStack projects (see 
TL;DR - results aren't good for us and I think we really need to do something
Of course "easiest" thing to say is that we should fix issues which we are
hitting in the CI to make jobs more stable. But it's not that easy. We are
struggling with those jobs for very long time. We have CI related meeting
every week and we are fixing what we can there.
Unfortunately there is still bunch of issues which we can't fix so far because
they are intermittent and hard to reproduce locally or in some cases the
issues aren't realy related to the Neutron or there are new bugs which we need
to investigate and fix :)
So this is never ending battle for us. The problem is that we have to test
various backends, drivers, etc. so as a result we have many jobs running on
each patch - excluding UT, pep8 and docs jobs we have around 19 jobs in check
and 14 jobs in gate queue.
In the past we made a lot of improvements, like e.g. we improved irrelevant
files lists for jobs to run less jobs on some of the patches, together with QA
team we did "integrated-networking" template to run only Neutron and Nova
related scenario tests in the Neutron queues, we removed and consolidated some
of the jobs (there is still one patch in progress for that but it should just
remove around 2 jobs from the check queue). All of that are good improvements
but still not enough to make our CI really stable :/
Because of all of that, I would like to ask community about any other ideas
how we can improve that. If You have any ideas, please send it in this email
thread or reach out to me directly on irc.
We want to discuss about them in the next video CI meeting which will be on
November 30th. If You would have any idea and would like to join that
discussion, You are more than welcome in that meeting of course :)
Principal Software Engineer
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 488 bytes
Desc: This is a digitally signed message part.
More information about the openstack-discuss