March 2026 - service-discuss - lists.openstack.org

Team Meeting Agenda for March 23, 2026
by Jeremy Stanley 23 Mar '26

23 Mar '26

We will meet with this agenda on March 23, 2026 at 19:00 UTC in #opendev-meeting (OFTC IRC): == Agenda for next meeting == * Announcements ** OpenStack release is scheduled to happen April 1. https://releases.openstack.org/gazpacho/schedule.html * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** In addition to the backups servers and graphite, clarkb can work with mnasiadka to do some of the outstanding cleanup for the mirror nodes. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** A DDoS against static hosted sites resulted in new WAF rules and approaches on static02 *** It is possible we may wish to apply similar approaches to sites like lists though the specific details will be different *** https://review.opendev.org/q/hashtag:%22apache-waf%22+status:open *** There was also discussion about subscribing to common mod security rulesets that are already packaged and available via Ubuntu repos. *** Should we consider a larger static server, or multiple servers behind a load balancer? ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** https://review.opendev.org/c/opendev/system-config/+/976282 *** Based on Ansible's python support Matrix Ansible 9 gives us a good deal of flexibility for bridge and remote nodes *** Ansible 9 also fixes problems with the use of pkg_resources in the Ansible ip module *** Any concerns with proceeding with the upgrade since tests look good? ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.12 and 3.13 Upgrade Planning (clarkb 20260310) *** Targeting April 5/6 and April 12/13 for upgrade to 3.12 and 3.13 respectively. *** Goal is to catch back up to being only one release behind Gerrit upstream. 3.14 is expected to release in May *** Will probably need to start building 3.13 images earlier than usual and test both the 3.11 -> 3.12 and 3.12 ->3.13 upgrades *** Would rather not do them all in one go to simplify rollbacks if necessary and reduce the total downtime as >1 release upgrade requires offline reindexing. **** The big risk currently on the radar is that H2 is upgraded from v1 to v2 in 3.12. But will need to do more digging through release notes as well as testing. ** Purging backups on the smaller backup server (clarkb 20260310) *** Purging review02 and paste01 backups did free up some additional space *** Should we do the same for eavesdrop01 and refstack01 backups? ** Gitea 1.25.5 (clarkb 20260317) *** There is a new gitea bugfix release that we should update to. *** https://review.opendev.org/c/opendev/system-config/+/980808 * Open discussion -- Jeremy Stanley

1 1

Team Meeting Agenda for March 17, 2026
by Clark Boylan 17 Mar '26

17 Mar '26

We will meet with this agenda on March 17, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack release is scheduled to happen April 1. https://releases.openstack.org/gazpacho/schedule.html ** Clarkb will be out Monday March 23 * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** In addition to the backups servers and graphite, clarkb can work with mnasiadka to do some of the outstanding cleanup for the mirror nodes. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** A DDoS against static hosted sites resulted in new WAF rules and approaches on static02 *** It is possible we may wish to apply similar approaches to sites like lists though the specific details will be different *** https://review.opendev.org/q/hashtag:%22apache-waf%22+status:open *** There was also discussion about subscribing to common mod security rulesets that are already packaged and available via Ubuntu repos. *** Should we consider a larger static server, or multiple servers behind a load balancer? ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** https://review.opendev.org/c/opendev/system-config/+/976282 *** Based on Ansible's python support Matrix Ansible 9 gives us a good deal of flexibility for bridge and remote nodes *** Ansible 9 also fixes problems with the use of pkg_resources in the Ansible ip module *** Any concerns with proceeding with the upgrade since tests look good? ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.12 and 3.13 Upgrade Planning (clarkb 20260310) *** Targeting April 5/6 and April 12/13 for upgrade to 3.12 and 3.13 respectively. *** Goal is to catch back up to being only one release behind Gerrit upstream. 3.14 is expected to release in May *** Will probably need to start building 3.13 images earlier than usual and test both the 3.11 -> 3.12 and 3.12 ->3.13 upgrades *** Would rather not do them all in one go to simplify rollbacks if necessary and reduce the total downtime as >1 release upgrade requires offline reindexing. **** The big risk currently on the radar is that H2 is upgraded from v1 to v2 in 3.12. But will need to do more digging through release notes as well as testing. ** Purging backups on the smaller backup server (clarkb 20260310) *** Purging review02 and paste01 backups did free up some additional space *** Should we do the same for eavesdrop01 and refstack01 backups? ** Gitea 1.25.5 (clarkb 20260317) *** There is a new gitea bugfix release that we should update to. *** https://review.opendev.org/c/opendev/system-config/+/980808 * Open discussion

1 0

Team Meeting Agenda for March 10, 2026
by Clark Boylan 09 Mar '26

09 Mar '26

We will meet with this agenda on March 10, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** DST changes have begun. Double check meeting times haven't shifted against your local clock as a result. ** OpenStack release is scheduled to happen April 1 with RC1 expected this week. https://releases.openstack.org/gazpacho/schedule.html * Actions from last meeting * Specs Review * Topics ** Adding mnasiadka as a new root sysadmin (clarkb 20260310) *** https://review.opendev.org/c/opendev/system-config/+/978980 *** If others can review this change to ack it that would be great for record keeping *** Then clarkb is happy to work with mnasiadka to onboard on all the things. There is a lot we can potentially look at but replacing servers seems like a good first step since it helps to exercise a variety of tools/systems/access. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** In addition to the backups servers and graphite, clarkb can work with mnasiadka to do some of the outstanding cleanup for the mirror nodes. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** A DDoS against static hosted sites resulted in new WAF rules and approaches on static02 *** It is possible we may wish to apply similar approaches to sites like lists though the specific details will be different *** https://review.opendev.org/q/hashtag:%22apache-waf%22+status:open *** There was also discussion about subscribing to common mod security rulesets that are already packaged and available via Ubuntu repos. ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** https://review.opendev.org/c/opendev/system-config/+/976282 *** Based on Ansible's python support Matrix Ansible 9 gives us a good deal of flexibility for bridge and remote nodes *** Ansible 9 also fixes problems with the use of pkg_resources in the Ansible ip module *** Any concerns with proceeding with the upgrade since tests look good? ** Gerrit 3.12 and 3.13 Upgrade Planning (clarkb 20260310) *** Targeting April 5/6 and April 12/13 for upgrade to 3.12 and 3.13 respectively. *** Goal is to catch back up to being only one release behind Gerrit upstream. 3.14 is expected to release in May *** Will probably need to start building 3.13 images earlier than usual and test both the 3.11 -> 3.12 and 3.12 ->3.13 upgrades *** Would rather not do them all in one go to simplify rollbacks if necessary and reduce the total downtime as >1 release upgrade requires offline reindexing. **** The big risk currently on the radar is that H2 is upgraded from v1 to v2 in 3.12. But will need to do more digging through release notes as well as testing. ** Purging backups on the smaller backup server (clarkb 20260310) *** Each prune run on the smaller backup server is pruning less total data relative to the available disk capacity *** We can properly purge backups for these servers: **** eavesdrop01, review02, refstack01, and paste01 *** https://review.opendev.org/c/opendev/system-config/+/951041 Change to purge review02 and paste01 * Open discussion

1 0

OpenDevent Is Replacing Tomorrow's (March 3, 2026) Team Meeting
by Clark Boylan 03 Mar '26

03 Mar '26

Just a quick heads up that our typical weekly meeting is getting replaced this week by OpenDevent happening from 1900-2100 UTC tomorrow March 3, 2026. More details can be found in the planning and agenda etherpad: https://etherpad.opendev.org/p/opendevent-march-2026. See you there, Clark

1 0