- service-discuss - lists.openstack.org

February 2026 OpenDev Service Coordinator Election
by Clark Boylan 17 Feb '26

17 Feb '26

It is that time of year again where we need to elect an OpenDev Service Coordinator [0] for the next six months. Nominations will be open from February 3, 2026 to February 17, 2026. If we have more than one candidate we will vote for our preferred candidate from February 18, 2026 to February 25, 2026. All dates are based on a UTC clock. Please send your self nominations to this mailing list during the nomination time frame. You are eligible to run and vote if you have contributed to OpenDev git repositories in the last 12 months (this period will be February 3, 2025 to February 2, 2026). I continue to encourage anyone who may be interested to reach out and ask questions. I am happy to support someone new in the role. [0] https://docs.opendev.org/opendev/system-config/latest/project.html#governan… Clark

1 1

Team Meeting Agenda for February 17, 2026
by Jeremy Stanley 16 Feb '26

16 Feb '26

We will meet with this agenda on February 17, 2026 at 19:00 UTC in #opendev-meeting (OFTC IRC): == Agenda for next meeting == * Announcements ** Service Coordinator Nominations Open Until February 17 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Setuptools removed pkg_resources in 82.0.0 (clarkb 20260210) *** https://discuss.python.org/t/pkg-resources-removal-how-to-go-from-there/106… Upstream discussion happening here *** This broke our infra-prod-service-bridge job which was fixed by pinning Setuptools in the ansible venv *** We should consider upgrading Ansible as newer Ansible will use packaging instead of pkg_resources ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We have moved to Matrix *** Minimal traffic has remained on IRC *** Tools like Gerritbot, statusbot, and eavesdropping seem to be working well *** Have we noticed any problems? ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** https://review.opendev.org/c/opendev/system-config/+/974942 More aggressive lure setup via comment in the html ** MariaDB backups over TCP instead of Unix Socket (clarkb 20260127) *** This seems to be working *** The one issue we ran into was the keycloak's db listens on ::1 not 127.0.0.1. ** Updating All of Our Containers to Trixie (clarkb 20260203) *** Trixie has been working well on services like Etherpad and Gerrit *** Let's update all of our containers to Trixie so that we can drop bookworm builds *** https://review.opendev.org/q/hashtag:%22opendev-trixie%22+status:open *** Should we retire Gear rather than update its container image? Or maybe just stop building a container for geard? ** Pre PTG Planning (clarkb 20260203) *** Would March 2-4 or March 9-11 work for a few days of snyc up on meetpad? *** We would probably do two or three hours a day and the Tuesday sync up would replace our regularly scheduled team meeting. * Open discussion

1 0

Team Meeting Agenda for February 10, 2026
by Clark Boylan 09 Feb '26

09 Feb '26

We will meet with this agenda on February 10, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service Coordinator Nominations Open Until February 17 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Setuptools removed pkg_resources in 82.0.0 (clarkb 20260210) *** https://discuss.python.org/t/pkg-resources-removal-how-to-go-from-there/106… Upstream discussion happening here *** This broke our infra-prod-service-bridge job which was fixed by pinning Setuptools in the ansible venv *** We should consider upgrading Ansible as newer Ansible will use packaging instead of pkg_resources ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We have moved to Matrix *** Minimal traffic has remained on IRC *** Tools like Gerritbot, statusbot, and eavesdropping seem to be working well *** Have we noticed any problems? ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** https://review.opendev.org/c/opendev/system-config/+/974942 More aggressive lure setup via comment in the html ** MariaDB backups over TCP instead of Unix Socket (clarkb 20260127) *** This seems to be working *** The one issue we ran into was the keycloak's db listens on ::1 not 127.0.0.1. ** Updating All of Our Containers to Trixie (clarkb 20260203) *** Trixie has been working well on services like Etherpad and Gerrit *** Let's update all of our containers to Trixie so that we can drop bookworm builds *** https://review.opendev.org/q/hashtag:%22opendev-trixie%22+status:open *** Should we retire Gear rather than update its container image? Or maybe just stop building a container for geard? ** Pre PTG Planning (clarkb 20260203) *** Would March 2-4 or March 9-11 work for a few days of snyc up on meetpad? *** We would probably do two or three hours a day and the Tuesday sync up would replace our regularly scheduled team meeting. * Open discussion

1 0

Team Meeting Agenda for February 3, 2026
by Clark Boylan 02 Feb '26

02 Feb '26

We will meet with this agenda on February 3, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service Coordinator Nominations Opened Today (February 3) *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We have moved to Matrix *** Minimal traffic has remained on IRC *** Tools like Gerritbot, statusbot, and eavesdropping seem to be working well *** Have we noticed any problems? ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** The problem now is that we aren't handling manifest lists so they get pruned and 404 *** https://review.opendev.org/c/zuul/zuul-registry/+/975088 handle manifest lists in addition to manifests themselves. ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** https://review.opendev.org/c/opendev/system-config/+/974942 More aggressive lure setup via comment in the html ** MariaDB backups over TCP instead of Unix Socket (clarkb 20260127) *** We noticed that if mariadb's container image is updated during mysqldumps for backups that backup may never exit and conclude *** Theory is that this is happening because the unix socket is held open by the linux kernel then nothing further is written to it so everything hangs around *** Proposed fix is to run mysqldump over TCP. We are testing this with lodgeit to start. *** https://review.opendev.org/c/opendev/system-config/+/973541 This will apply the change to all services if lodgeit looks happy. ** Updating All of Our Containers to Trixie (clarkb 20260203) *** Trixie has been working well on services like Etherpad and Gerrit *** Let's update all of our containers to Trixie so that we can drop bookworm builds *** https://review.opendev.org/c/opendev/system-config/+/975319 Update the base python images First *** https://review.opendev.org/q/hashtag:%22opendev-trixie%22+status:open *** Should we retire Gear rather than update its container image? Or maybe just stop building a container for geard? ** Gerrit 3.11.8 (clarkb 20260203) *** There is a new Gerrit 3.11.8 bugfix release available. *** https://review.opendev.org/c/opendev/system-config/+/975176 Upgrade to Gerrit 3.11.8 *** This change also converts us from building from stable-3.11 to checking out the v3.11.8 tag *** The idea is that this will prevent unexpected Gerrit updates from sneaking into Gerrit image rebuilds if those happen for unrelated reasons. ** Gerrit change.maxUpdate (clarkb 20260203) *** Gerrit's notedb by default allows 1000 updates to a single change. This is configured by change.maxUpdates in the gerrit server config *** This Cinder change has hit that limit: https://review.opendev.org/c/openstack/cinder/+/926844 *** The limit is set for performance reasons. It looks like Cinder may have a number of possibly outdated and never succeeding third party CI systems. Maybe we can disable those before we think about updating limits. ** Checking CPU Counts in Our Base Zuul Job (clarkb 20260203) *** https://review.opendev.org/c/opendev/base-jobs/+/975474 ** Pre PTG Planning (clarkb 20260203) *** Would March 2-4 or March 9-11 work for a few days of snyc up on meetpad? *** We would probably do two or three hours a day and the Tuesday sync up would replace our regularly scheduled team meeting. * Open discussion

1 0

Gerrit account creation issue
by Jad Haj Yahya 30 Jan '26

30 Jan '26

Hello, I am trying to create an OpenDev Gerrit account. - *Ub*untu One email: jad.haj(a)gmail.com - Ubuntu One username: jadhajyahy - error: Not Found page with url: https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator - - Thanks, - Jad -

2 2

Team Meeting Agenda for January 27, 2026
by Clark Boylan 26 Jan '26

26 Jan '26

We will meet with this agenda on January 27, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We have moved to Matrix *** Minimal traffic has remained on IRC *** Tools like Gerritbot, statusbot, and eavesdropping seem to be working well *** Have we noticed any problems? ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** Fixes for OCI image pruning have been put in place *** Rechecks of https://review.opendev.org/c/opendev/system-config/+/970451 show there may still be problems though ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** docs.opendev.org has a honeypot in place in its robots.txt file *** So far the only hits for this have been from fungi's test IP address *** We may need a more explicit url link in the content or maybe we need to apply the honeypot to a system that gets crawled more than docs.opendev.org ** Etherpad 2.6.1 Upgrade (clarkb 20260113) *** As soon as we upgraded to 2.6.0 there is a new 2.6.1 release. It needs a change and testing *** There is also a new etherpad golang rewrite: https://github.com/ether/etherpad-go/wiki/FAQ ** MariaDB backups over TCP instead of Unix Socket (clarkb 20260127) *** We noticed that if mariadb's container image is updated during mysqldumps for backups that backup may never exit and conclude *** Theory is that this is happening because the unix socket is held open by the linux kernel then nothing further is written to it so everything hangs around *** Proposed fix is to run mysqldump over TCP. We are testing this with lodgeit to start. *** https://review.opendev.org/c/opendev/system-config/+/973541 This will apply the change to all services if lodgeit looks happy. ** Service Coordinator Election (clarkb 20260120) *** Nomination period running February 3-17. Then if necessary an election will run February 18-25. * Open discussion

1 0

Team Meeting Agenda for January 20, 2026
by Clark Boylan 19 Jan '26

19 Jan '26

We will meet with this agenda on January 20, 2026 at 1900 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul Secret Rotation *** We gave Zuul a new brain and have been rotating secrets *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** At this point most secrets that need rotating are largely user managed. If OpenDev can help with any rotations let us know. ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 now running on Java 21 atop Debian Trixie **** Possible new warning for JDBC errors related to inserting non unique Primary keys to accountPatchReviewDb. Java 21 required a newer jdbc driver for mariadb which is the likely cause. **** Clarkb doesn't think this has a real effect on usage of Gerrit. *** Add Gerrit 3.12 Images, Testing, and build web artifacts with node 24 **** https://review.opendev.org/c/opendev/system-config/+/970173 Change stack starts here. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We are moving January 20, 2026 at 20:00 UTC (immediately after this meeting) *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Users can join the new room now as it is up and running. ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** Fixes for OCI image pruning have been put in place *** Rechecks of https://review.opendev.org/c/opendev/system-config/+/970451 show there may still be problems though ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** docs.opendev.org has a honeypot in place in its robots.txt file *** So far the only hits for this have been from fungi's test IP address *** We may need a more explicit url link in the content or maybe we need to apply the honeypot to a system that gets crawled more than docs.opendev.org ** Etherpad 2.6.0 Upgrade (clarkb 20260113) *** https://review.opendev.org/c/opendev/system-config/+/972001 ** Service Coordinator Election (clarkb 20260120) *** Proposal: Nomination period running February 3-17. Then if necessary an election will run February 18-25. * Open discussion

1 0

Team Meeting Agenda for January 13, 2026
by Clark Boylan 12 Jan '26

12 Jan '26

We will meet with this agenda on January 13, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul Secret Rotation *** We gave Zuul a new brain last week and have been rotating secrets *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** At this point we expect most of the system to be functional *** There is likely a long tail of secrets that need rotating and we'll do our best to help ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well **** Other containers have updated to Trixie and seem to be working well. **** https://review.opendev.org/c/opendev/system-config/+/970160 Switches Gerrit to Java 21 on Debian Trixie *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Rename happened December 12 at approximately 2000 UTC *** Discovered that gerrit2 doesn't own its own homedir impacting some Ansible tasks in the rename playbook **** https://review.opendev.org/c/opendev/system-config/+/970919 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** We are moving January 20, 2026 at 20:00 UTC *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Users can join the new room now as it is up and running. ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** Fixes for OCI image pruning have been put in place *** Rechecks of https://review.opendev.org/c/opendev/system-config/+/970451 show there may still be problems though ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** docs.opendev.org has a honeypot in place in its robots.txt file *** So far the only hits for this have been from fungi's test IP address *** We may need a more explicit url link in the content or maybe we need to apply the honeypot to a system that gets crawled more than docs.opendev.org ** Gitea 1.25.3 Upgrade (clarkb 20260113) *** https://review.opendev.org/c/opendev/system-config/+/971469 ** Etherpad 2.6.0 Upgrade (clarkb 20260113) *** https://review.opendev.org/c/opendev/system-config/+/972001 * Open discussion

1 0

Team Meeting Agenda for January 6, 2026
by Clark Boylan 06 Jan '26

06 Jan '26

We will meet with this agenda on January 6, 2026 at 19:00 UTC in #opendev-meeting. I expect that this will be a quick meeting while we start to catch up on things in the new year. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well **** Other containers have updated to Trixie and seem to be working well. **** https://review.opendev.org/c/opendev/system-config/+/970160 Switches Gerrit to Java 21 on Debian Trixie *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Rename happened December 12 at approximately 2000 UTC *** Discovered that gerrit2 doesn't own its own homedir impacting some Ansible tasks in the rename playbook **** https://review.opendev.org/c/opendev/system-config/+/970919 *** Also decided we should have manage-projects record logs the same way as other infra-prod jobs **** https://review.opendev.org/c/opendev/system-config/+/970920 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** Statusbot has been added to matrix *** Looking at our work items https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… I believe the next step is to announce the move to Matrix then move. *** Thoughts on when we want to do that? ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** We (re)discovered that speculative images in the intermediate registry seem to get pruned too soon *** Fix for that introduced a new problem with unexpected 404 results for early pruned data *** https://review.opendev.org/c/zuul/zuul-registry/+/971559 should clean that up. ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** As the web crawlers go crazy on the Internet more and more people notice the disruption they cause *** We can continue to make our services faster and bigger to keep up, but pushing against some of the demand is also a good idea *** One idea from fungi is that we set up a honeypot of invalid Disallowed data (via robots.txt) and anyone that fetches it gets blocked *** https://review.opendev.org/c/opendev/system-config/+/970674 Implement a PoC honeypot against docs.opendev.org * Open discussion

1 0

Cancelling our Team Meetings December 23 and December 30, 2025
by Clark Boylan 19 Dec '25

19 Dec '25

Hello, Things seem to have slowed down quite a bit with the end of the year holidays. I'm going to go ahead and cancel our next meeting on December 23. We had previously decided to cancel the December 30 meeting as well. We will be back January 6, 2026. See you then. If anything comes up between now and then feel free to start a thread on this mailing list or jump onto IRC and bring it up there. Clark

1 0