Hi, Thank You all for all the ideas and discussion in this thread. With Yatin we prepared summary at https://etherpad.opendev.org/p/neutron-ci-improvements and we want to go over and discuss them on the today's CI meeting. It will be both video and irc meeting. We will talk on https:// meetpad.opendev.org/neutron-ci-meetings Agenda for the meeting is at https://etherpad.opendev.org/p/neutron-ci-meetings Everyone is welcome to join the discussion :) On środa, 17 listopada 2021 09:13:34 CET Slawek Kaplonski wrote:
Hi,
Recently I spent some time to check how many rechecks we need in Neutron to get patch merged and I compared it to some other OpenStack projects (see [1] for details). TL;DR - results aren't good for us and I think we really need to do something with that.
Of course "easiest" thing to say is that we should fix issues which we are hitting in the CI to make jobs more stable. But it's not that easy. We are struggling with those jobs for very long time. We have CI related meeting every week and we are fixing what we can there. Unfortunately there is still bunch of issues which we can't fix so far because they are intermittent and hard to reproduce locally or in some cases the issues aren't realy related to the Neutron or there are new bugs which we need to investigate and fix :) So this is never ending battle for us. The problem is that we have to test various backends, drivers, etc. so as a result we have many jobs running on each patch - excluding UT, pep8 and docs jobs we have around 19 jobs in check and 14 jobs in gate queue.
In the past we made a lot of improvements, like e.g. we improved irrelevant files lists for jobs to run less jobs on some of the patches, together with QA team we did "integrated-networking" template to run only Neutron and Nova related scenario tests in the Neutron queues, we removed and consolidated some of the jobs (there is still one patch in progress for that but it should just remove around 2 jobs from the check queue). All of that are good improvements but still not enough to make our CI really stable :/
Because of all of that, I would like to ask community about any other ideas how we can improve that. If You have any ideas, please send it in this email thread or reach out to me directly on irc. We want to discuss about them in the next video CI meeting which will be on November 30th. If You would have any idea and would like to join that discussion, You are more than welcome in that meeting of course :)
[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-November/ 025759.html
-- Slawek Kaplonski Principal Software Engineer Red Hat