Open Stack

Hi,

Recently I spent some time to check how many rechecks we need in Neutron to 
get patch merged and I compared it to some other OpenStack projects (see [1] 
for details).
TL;DR - results aren't good for us and I think we really need to do something 
with that.

Of course "easiest" thing to say is that we should fix issues which we are 
hitting in the CI to make jobs more stable. But it's not that easy. We are 
struggling with those jobs for very long time. We have CI related meeting 
every week and we are fixing what we can there.
Unfortunately there is still bunch of issues which we can't fix so far because 
they are intermittent and hard to reproduce locally or in some cases the 
issues aren't realy related to the Neutron or there are new bugs which we need 
to investigate and fix :)
So this is  never ending battle for us. The problem is that we have to test 
various backends, drivers, etc. so as a result we have many jobs running on 
each patch - excluding UT, pep8 and docs jobs we have around 19 jobs in check 
and 14 jobs in gate queue.

In the past we made a lot of improvements, like e.g. we improved irrelevant 
files lists for jobs to run less jobs on some of the patches, together with QA 
team we did "integrated-networking" template to run only Neutron and Nova 
related scenario tests in the Neutron queues, we removed and consolidated some 
of the jobs (there is still one patch in progress for that but it should just 
remove around 2 jobs from the check queue). All of that are good improvements 
but still not enough to make our CI really stable :/

Because of all of that, I would like to ask community about any other ideas 
how we can improve that. If You have any ideas, please send it in this email 
thread or reach out to me directly on irc.
We want to discuss about them in the next video CI meeting which will be on 
November 30th. If You would have any idea and would like to join that 
discussion, You are more than welcome in that meeting of course :)

[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-November/
025759.html

-- 
Slawek Kaplonski
Principal Software Engineer
Red Hat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211117/73a61bde/attachment.sig>