January 2026 - service-discuss - lists.openstack.org

February 2026 OpenDev Service Coordinator Election
by Clark Boylan 17 Feb '26

17 Feb '26

It is that time of year again where we need to elect an OpenDev Service Coordinator [0] for the next six months. Nominations will be open from February 3, 2026 to February 17, 2026. If we have more than one candidate we will vote for our preferred candidate from February 18, 2026 to February 25, 2026. All dates are based on a UTC clock. Please send your self nominations to this mailing list during the nomination time frame. You are eligible to run and vote if you have contributed to OpenDev git repositories in the last 12 months (this period will be February 3, 2025 to February 2, 2026). I continue to encourage anyone who may be interested to reach out and ask questions. I am happy to support someone new in the role. [0] https://docs.opendev.org/opendev/system-config/latest/project.html#governan… Clark

1 1

Gerrit account creation issue
by Jad Haj Yahya 30 Jan '26

30 Jan '26

Hello, I am trying to create an OpenDev Gerrit account. - *Ub*untu One email: jad.haj(a)gmail.com - Ubuntu One username: jadhajyahy - error: Not Found page with url: https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator - - Thanks, - Jad -

2 2

Team Meeting Agenda for January 27, 2026
by Clark Boylan 26 Jan '26

26 Jan '26

We will meet with this agenda on January 27, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We have moved to Matrix *** Minimal traffic has remained on IRC *** Tools like Gerritbot, statusbot, and eavesdropping seem to be working well *** Have we noticed any problems? ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** Fixes for OCI image pruning have been put in place *** Rechecks of https://review.opendev.org/c/opendev/system-config/+/970451 show there may still be problems though ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** docs.opendev.org has a honeypot in place in its robots.txt file *** So far the only hits for this have been from fungi's test IP address *** We may need a more explicit url link in the content or maybe we need to apply the honeypot to a system that gets crawled more than docs.opendev.org ** Etherpad 2.6.1 Upgrade (clarkb 20260113) *** As soon as we upgraded to 2.6.0 there is a new 2.6.1 release. It needs a change and testing *** There is also a new etherpad golang rewrite: https://github.com/ether/etherpad-go/wiki/FAQ ** MariaDB backups over TCP instead of Unix Socket (clarkb 20260127) *** We noticed that if mariadb's container image is updated during mysqldumps for backups that backup may never exit and conclude *** Theory is that this is happening because the unix socket is held open by the linux kernel then nothing further is written to it so everything hangs around *** Proposed fix is to run mysqldump over TCP. We are testing this with lodgeit to start. *** https://review.opendev.org/c/opendev/system-config/+/973541 This will apply the change to all services if lodgeit looks happy. ** Service Coordinator Election (clarkb 20260120) *** Nomination period running February 3-17. Then if necessary an election will run February 18-25. * Open discussion

1 0

Team Meeting Agenda for January 20, 2026
by Clark Boylan 19 Jan '26

19 Jan '26

We will meet with this agenda on January 20, 2026 at 1900 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul Secret Rotation *** We gave Zuul a new brain and have been rotating secrets *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** At this point most secrets that need rotating are largely user managed. If OpenDev can help with any rotations let us know. ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 now running on Java 21 atop Debian Trixie **** Possible new warning for JDBC errors related to inserting non unique Primary keys to accountPatchReviewDb. Java 21 required a newer jdbc driver for mariadb which is the likely cause. **** Clarkb doesn't think this has a real effect on usage of Gerrit. *** Add Gerrit 3.12 Images, Testing, and build web artifacts with node 24 **** https://review.opendev.org/c/opendev/system-config/+/970173 Change stack starts here. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We are moving January 20, 2026 at 20:00 UTC (immediately after this meeting) *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Users can join the new room now as it is up and running. ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** Fixes for OCI image pruning have been put in place *** Rechecks of https://review.opendev.org/c/opendev/system-config/+/970451 show there may still be problems though ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** docs.opendev.org has a honeypot in place in its robots.txt file *** So far the only hits for this have been from fungi's test IP address *** We may need a more explicit url link in the content or maybe we need to apply the honeypot to a system that gets crawled more than docs.opendev.org ** Etherpad 2.6.0 Upgrade (clarkb 20260113) *** https://review.opendev.org/c/opendev/system-config/+/972001 ** Service Coordinator Election (clarkb 20260120) *** Proposal: Nomination period running February 3-17. Then if necessary an election will run February 18-25. * Open discussion

1 0

Team Meeting Agenda for January 13, 2026
by Clark Boylan 12 Jan '26

12 Jan '26

We will meet with this agenda on January 13, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul Secret Rotation *** We gave Zuul a new brain last week and have been rotating secrets *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** At this point we expect most of the system to be functional *** There is likely a long tail of secrets that need rotating and we'll do our best to help ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well **** Other containers have updated to Trixie and seem to be working well. **** https://review.opendev.org/c/opendev/system-config/+/970160 Switches Gerrit to Java 21 on Debian Trixie *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Rename happened December 12 at approximately 2000 UTC *** Discovered that gerrit2 doesn't own its own homedir impacting some Ansible tasks in the rename playbook **** https://review.opendev.org/c/opendev/system-config/+/970919 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We are moving January 20, 2026 at 20:00 UTC *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Users can join the new room now as it is up and running. ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** Fixes for OCI image pruning have been put in place *** Rechecks of https://review.opendev.org/c/opendev/system-config/+/970451 show there may still be problems though ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** docs.opendev.org has a honeypot in place in its robots.txt file *** So far the only hits for this have been from fungi's test IP address *** We may need a more explicit url link in the content or maybe we need to apply the honeypot to a system that gets crawled more than docs.opendev.org ** Gitea 1.25.3 Upgrade (clarkb 20260113) *** https://review.opendev.org/c/opendev/system-config/+/971469 ** Etherpad 2.6.0 Upgrade (clarkb 20260113) *** https://review.opendev.org/c/opendev/system-config/+/972001 * Open discussion

1 0

Team Meeting Agenda for January 6, 2026
by Clark Boylan 06 Jan '26

06 Jan '26

We will meet with this agenda on January 6, 2026 at 19:00 UTC in #opendev-meeting. I expect that this will be a quick meeting while we start to catch up on things in the new year. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well **** Other containers have updated to Trixie and seem to be working well. **** https://review.opendev.org/c/opendev/system-config/+/970160 Switches Gerrit to Java 21 on Debian Trixie *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Rename happened December 12 at approximately 2000 UTC *** Discovered that gerrit2 doesn't own its own homedir impacting some Ansible tasks in the rename playbook **** https://review.opendev.org/c/opendev/system-config/+/970919 *** Also decided we should have manage-projects record logs the same way as other infra-prod jobs **** https://review.opendev.org/c/opendev/system-config/+/970920 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** Statusbot has been added to matrix *** Looking at our work items https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… I believe the next step is to announce the move to Matrix then move. *** Thoughts on when we want to do that? ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** We (re)discovered that speculative images in the intermediate registry seem to get pruned too soon *** Fix for that introduced a new problem with unexpected 404 results for early pruned data *** https://review.opendev.org/c/zuul/zuul-registry/+/971559 should clean that up. ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** As the web crawlers go crazy on the Internet more and more people notice the disruption they cause *** We can continue to make our services faster and bigger to keep up, but pushing against some of the demand is also a good idea *** One idea from fungi is that we set up a honeypot of invalid Disallowed data (via robots.txt) and anyone that fetches it gets blocked *** https://review.opendev.org/c/opendev/system-config/+/970674 Implement a PoC honeypot against docs.opendev.org * Open discussion

1 0