Open Stack

Hi,

Thank You all for all the ideas and discussion in this thread.
With Yatin we prepared summary at https://etherpad.opendev.org/p/neutron-ci-improvements and we want to go over and discuss them on the today's CI 
meeting.
It will be both video and irc meeting. We will talk on https://
meetpad.opendev.org/neutron-ci-meetings
Agenda for the meeting is at https://etherpad.opendev.org/p/neutron-ci-meetings

Everyone is welcome to join the discussion :)

On środa, 17 listopada 2021 09:13:34 CET Slawek Kaplonski wrote:
> Hi,
> 
> Recently I spent some time to check how many rechecks we need in Neutron to
> get patch merged and I compared it to some other OpenStack projects (see [1]
> for details).
> TL;DR - results aren't good for us and I think we really need to do 
something
> with that.
> 
> Of course "easiest" thing to say is that we should fix issues which we are
> hitting in the CI to make jobs more stable. But it's not that easy. We are
> struggling with those jobs for very long time. We have CI related meeting
> every week and we are fixing what we can there.
> Unfortunately there is still bunch of issues which we can't fix so far
> because they are intermittent and hard to reproduce locally or in some cases
> the issues aren't realy related to the Neutron or there are new bugs which
> we need to investigate and fix :)
> So this is  never ending battle for us. The problem is that we have to test
> various backends, drivers, etc. so as a result we have many jobs running on
> each patch - excluding UT, pep8 and docs jobs we have around 19 jobs in 
check
> and 14 jobs in gate queue.
> 
> In the past we made a lot of improvements, like e.g. we improved irrelevant
> files lists for jobs to run less jobs on some of the patches, together with
> QA team we did "integrated-networking" template to run only Neutron and Nova
> related scenario tests in the Neutron queues, we removed and consolidated
> some of the jobs (there is still one patch in progress for that but it
> should just remove around 2 jobs from the check queue). All of that are good
> improvements but still not enough to make our CI really stable :/
> 
> Because of all of that, I would like to ask community about any other ideas
> how we can improve that. If You have any ideas, please send it in this email
> thread or reach out to me directly on irc.
> We want to discuss about them in the next video CI meeting which will be on
> November 30th. If You would have any idea and would like to join that
> discussion, You are more than welcome in that meeting of course :)
> 
> [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-November/
> 025759.html


-- 
Slawek Kaplonski
Principal Software Engineer
Red Hat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211207/297d38d6/attachment.sig>