- service-discuss - lists.openstack.org

Cancelling our Team Meetings December 23 and December 30, 2025
by Clark Boylan 20 Dec '25

20 Dec '25

Hello, Things seem to have slowed down quite a bit with the end of the year holidays. I'm going to go ahead and cancel our next meeting on December 23. We had previously decided to cancel the December 30 meeting as well. We will be back January 6, 2026. See you then. If anything comes up between now and then feel free to start a thread on this mailing list or jump onto IRC and bring it up there. Clark

1 0

Team Meeting Agenda for December 16, 2025
by Clark Boylan 16 Dec '25

16 Dec '25

We will meet with this agenda on December 16, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** December 30 meeting will be cancelled. Other meetings may be cancelled depending on attendance * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well **** Other containers have updated to Trixie and seem to be working well. **** https://review.opendev.org/c/opendev/system-config/+/970160 Switches Gerrit to Java 21 on Debian Trixie *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Rename happened December 12 at approximately 2000 UTC *** Discovered that gerrit2 doesn't own its own homedir impacting some Ansible tasks in the rename playbook **** https://review.opendev.org/c/opendev/system-config/+/970919 *** Also decided we should have manage-projects record logs the same way as other infra-prod jobs **** https://review.opendev.org/c/opendev/system-config/+/970920 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** Created a new @status account for statusbot. **** Gave it a valid email address using a plus address suffix and account creation succeeded but recorded an error setting the email address *** Private vars should be updated and we can land the change to update statusbot configs whenever we like **** https://review.opendev.org/c/opendev/system-config/+/969328 ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** We (re)discovered that speculative images in the intermediate registry seem to get pruned too soon *** Images older than 180 are expected to be pruned but these were ~2 days old and removed *** Investigation shows that the pruning routine handles docker image types properly but not oci image types. *** https://review.opendev.org/c/zuul/zuul-registry/+/970480/ This change fixes that and has a few parents/children addressing general CI concerns. ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** As the web crawlers go crazy on the Internet more and more people notice the disruption they cause *** We can continue to make our services faster and bigger to keep up, but pushing against some of the demand is also a good idea *** One idea from fungi is that we set up a honeypot of invalid Disallowed data (via robots.txt) and anyone that fetches it gets blocked *** https://review.opendev.org/c/opendev/system-config/+/970674 Implement a PoC honeypot against docs.opendev.org * Open discussion

1 0

Team Meeting Agenda for December 9, 2025
by Clark Boylan 09 Dec '25

09 Dec '25

We will meet with this agenda on December 9, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** December 30 meeting will be cancelled. Other meetings may be cancelled depending on attendance * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade (clarkb 20250401) *** Upgrade happened on December 7 *** We are running Gerrit 3.11 on top of Debian Bookworm with Java 17 *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Fungi is looking at renaming this project on December 12 at 2100 UTC ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** Current blocker is adding a new bot account for the status bot on matrix *** EMS requires email for all newly created accounts. We can use a beta dashboard to do so without validating the email address *** Another option would be to use plus addressing to give bots working email addresses *** It isn't clear if we can use a single email address with multiple accounts ** Zuul Launcher Updates (clarkb 20251125) *** We found a corrupted image upload in raxflex sjc3 again on November 21. Previous occurrence was discovered on October 6 *** https://review.opendev.org/c/zuul/zuul/+/968090 Will validate the glance checksum against our checksum and reject mismatches **** It isn't clear to clarkb if this is safe as glance may change the checksum value if it does a format conversion on the backend? *** Floating IPs attached to held nodes were deleted in raxflex. Likely due to our leaked fip cleanup routine. Debugging has been added to the launcher to track this down. *** Clarkb will send email to Rackspace today to follow up on this incase it got lost during the holiday week at the end of November. * Open discussion

1 0

Team Meeting Agenda for December 2, 2025
by Clark Boylan 02 Dec '25

02 Dec '25

We will meet with this agenda on December 2, 2025 at 19:00 UTC in #opendev-meeting == Agenda for next meeting == * Announcements ** December 30 meeting will be cancelled. Other meetings may be cancelled depending on attendance * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Upgrade is happening Sunday December 7, 2025 *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Held nodes here: **** 3.10: https://217.182.142.186 **** 3.11: server lost its floating ip again and needs a new one. *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Fungi indicates that he can try to drive the renaming week of December 8-12 after the Gerrit upgrade ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** The new room exists and gerritbot has joined it *** Statusbot needs to be configured for Matrix and we need to schedule a cut over day for eavesdrop and our usage of the room. *** https://review.opendev.org/q/hashtag:%22opendev-matrix%22+status:open ** Zuul Launcher Updates (clarkb 20251125) *** We found a corrupted image upload in raxflex sjc3 again on November 21. Previous occurrence was discovered on October 6 *** https://review.opendev.org/c/zuul/zuul/+/968090 Will validate the glance checksum against our checksum and reject mismatches **** It isn't clear to clarkb if this is safe as glance may change the checksum value if it does a format conversion on the backend? *** Floating IPs attached to held nodes were deleted in raxflex. Likely due to our leaked fip cleanup routine. Debugging has been added to the launcher to track this down. * Open discussion

1 0

Team Meeting Agenda for November 25, 2025
by Clark Boylan 25 Nov '25

25 Nov '25

We will meet on November 25, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Holiday weekend for many of us this weekend. * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon. *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Held nodes here: **** 3.10: https://217.182.142.186 **** 3.11: https://50.56.159.76/ *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** Proposing December 7 at 2100 UTC for the upgrade itself ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Biggest question is when do we want to try and do this? Before or after the 3.11 Upgrade? Also where does it fit with holidays etc. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** The new room exists (thank you tonyb) and updates to statusbot have begun (thank you corvus) *** https://review.opendev.org/q/hashtag:%22opendev-matrix%22+status:open ** Zuul Launcher Updates (clarkb 20251125) *** We found a corrupted image upload in raxflex sjc3 again on November 21. Previous occurrence was discovered on October 6 *** https://review.opendev.org/c/zuul/zuul/+/968090 Will validate the glance checksum against our checksum and reject mismatches **** It isn't clear to clarkb if this is safe as glance may change the checksum value if it does a format conversion on the backend? *** Floating IPs attached to held nodes were deleted in raxflex. Likely due to our leaked fip cleanup routine. Debugging has been added to the launcher to track this down. ** Gitea 1.25.2 Upgrade (clarkb 20251125) *** https://review.opendev.org/c/opendev/system-config/+/968245 Upgrade gitea to 1.25.2 ** December Meeting Planning (clarkb 20251125) *** Holidays are approaching for many of us. Do we want to proactively cancel meetings? *** December 23, 30, and January 6 are the dates most likely to be impacted *** clarkb can likely host a meeting on each of these days if we wish to have them. * Open discussion

1 0

Re: Issue Adding OpenStack-Helm Repository via Helm
by Clark Boylan 21 Nov '25

21 Nov '25

Note this email was originally sent to the incident list, but I'm following up here on the public list as this is not a security concern. On 11/21/25 8:05 AM, Francesco La Porta wrote: > I am experiencing an issue when trying to add the OpenStack-Helm > repository using Helm. Running the following command: helm repo add > openstack-helm https://tarballs.opendev.org/openstack/openstack-helm > <https://tarballs.opendev.org/openstack/openstack-helm> > > returns the following error: > > Error: looks like "https://tarballs.opendev.org/openstack/openstack-helm > <https://tarballs.opendev.org/openstack/openstack-helm>" is not a valid > chart repository or cannot be reached: context deadline exceeded > (Client.Timeout or context cancellation while reading body) > > > This occurs consistently and prevents me from using the OpenStack-Helm > charts via Helm. > > Could you please confirm if there is an ongoing issue with the > repository or advise on any alternative methods to access the charts? This is probably not an intended behavior, but is unfortunately an expected one. The issue here is an underlying artifact of using Apache2 mod_autoindex to build index.html content on the fly from data backed by OpenAFS in a directory with thousands of entries. Opening this URL with wget and my browser takes about 3 minutes and 12 seconds. OpenAFS performs poorly with many thousands of entries and to make things worse Apache is stat'ing the directory entries to include modification times and file sizes in the generated index which takes time. The good news is I think you can workaround this today using the --timeout option to `helm repo add` and the OpenStack Helm team can change how they manage their Helm repository to improve its performance. `helm repo add` has a default timeout of 2 minutes. Given my testing says fetching this data takes around 3 minutes 12 seconds I think if you set the timeout to say 5 minutes using --timeout you should be able to fetch the data (if slowly). Separately, the OpenStack Helm team needs to stop writing all of this data to a single directory. Currently there are just under 15k entries in the directory. In addition to slowness stat'ing thousands of files OpenAFS directories have a hard limit on total entries between 15k and 60k based on the lengths of the filenames. We estimate the current limit for that directory is about 32k. Instead what they should be doing instead is sharding the data into subdirectories. For example the current file at openstack/openstack-helm/nova-2025.2.0+f6be959ce.tgz could instead be stored at openstack/openstack-helm/nova/2025.2.0/nova-2025.2.0+f6be959ce.tgz then openstack/openstack-helm/index.yaml should be updated to match. Doing this will drastically cut down on the total number of entries in each directory which will improve performance and avoid the hard limits. Now I recognize that you aren't going to fix that for them, and I can't either. I think a good next step would be for you to engage with the OpenStack Helm team directly and point out that this is both problematic for you and that there is something they can do to fix it. That team lists communication options here: https://docs.openstack.org/openstack-helm/latest/readme.html#communication and you should feel free to refer to this email thread or quote it when you reach out to them. An archived version of this email should show up in https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/ shortly after you receive it. Clark

1 0

Team Meeting Agenda for November 18, 2025
by Clark Boylan 18 Nov '25

18 Nov '25

We will meet with this agenda on November 18, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon. *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes here: **** 3.10: https://217.182.142.186 **** 3.11: https://174.143.59.58 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** The new room exists (thank you tonyb) and updates to statusbot have begun (thank you corvus) *** https://review.opendev.org/q/hashtag:%22opendev-matrix%22+status:open ** Gitea Performance (clarkb 20251028) *** All traffic should be through the load balancer now *** Memcached size limits have been increased *** We should monitor and determine if these changes have improved performance * Open discussion

1 0

Team Meeting Agenda for November 11, 2025
by Clark Boylan 11 Nov '25

11 Nov '25

We will meet with this agenda on November 11, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon. *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** https://review.opendev.org/c/opendev/system-config/+/966083 Fix container bind mounts for Gerrit *** https://review.opendev.org/c/opendev/system-config/+/966084 Update to Gerrit 3.10.9 and 3.11.7 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Upgrade Zuul Zookeeper Cluster to 3.9 (clarkb 20251111) *** https://review.opendev.org/c/opendev/system-config/+/966612 *** This will add some new features Zuul can take advantage of ** Gitea 1.25.1 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.1 ** Gitea Performance (clarkb 20251028) *** All traffic should be through the load balancer now *** Memcached size limits have been increased *** We should monitor and determine if these changes have improved performance ** Raxflex DFW3 Disabled (clarkb 20251104) *** We discovered the raxflex dfw3 mirror had afs cache errors. fs flushall appeared to get stuck and the kernel oopsed. After forcing a shutdown via openstack the server refuses to boot *** We've asked dan_with what our next steps should be via IRC *** Worst case we rebuild the mirror with a new cinder volume and replace the whole thing. * Open discussion

1 0

Team Meeting Agenda for November 4, 2025
by Clark Boylan 04 Nov '25

04 Nov '25

We will meet with this agenda on November 4, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb out early Friday and all of Monday. Expect a meeting Tuesday with a possibly delayed agenda. * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes are now available. *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Gerrit Spontaneous Shutdown During Summit (clarkb 20251028) *** The Gerrit server was discovered to be in a shutdown state with no recorded errors *** The Nova team indicates this could happen if the hypervisor is unable to allocate memory. The host would be shutdown and Nova would notice but not record an error *** Its possible the underlying cause is something else; either way we need to work with Vexxhost to identify the source of the problem and mitigate it *** Is someone willing to figure out how to file a ticket with Vexxhost? This was their suggested approach last time we had problems. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content updates (clarkb 20250916) *** https://review.opendev.org/c/opendev/system-config/+/965334 Mirror trixie packages *** Rocky Linux didn't hit these mirror issues because we don't mirror any rocky linux releases. Debian expects mirror content for all releases. ** Zuul Launcher Updates (clarkb 20250923) *** Nodes can be from mixed providers again due to a bug when reassigning unassigned nodes. *** https://review.opendev.org/c/zuul/zuul/+/965954 Fix assignment of unassigned nodes. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Etherpad 2.5.2 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.2/CHANGELOG.md *** https://review.opendev.org/c/opendev/system-config/+/956593 *** This fixes our css issues and clarkb believes we are ready to upgrade. ** Gitea 1.25.0 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.0 ** Gitea Performance (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/964728 Don't allow direct backend access **** Some performance issues seem related to crawlers hitting backends directly so they aren't balanced properly **** Most recent issue was via traffic to the load balancer frontend so this won't fix everything **** Means we'll have to take extra steps to test backends directly (ssh port forwards) *** https://review.opendev.org/c/opendev/system-config/+/965420 Increase memcached cache size to mitigate effect of crawlers poisoning the cache ** Raxflex DFW3 Disabled (clarkb 20251104) *** We discovered the raxflex dfw3 mirror had afs cache errors. fs flushall appeared to get stuck and the kernel oopsed. After forcing a shutdown via openstack the server refuses to boot *** We've asked dan_with what our next steps should be via IRC *** Worst case we rebuild the mirror with a new cinder volume and replace the whole thing. * Open discussion

1 0

Team Meeting Agenda for October 28, 2025
by Clark Boylan 28 Oct '25

28 Oct '25

I know the PTG is happening this week which might make this a bit tough, but it has been a while since we had a meeting so I want to ensure we get a chance to cover what is happening in OpenDev. We will meet with this agenda on October 28, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Need to hold new test nodes after some Zuul launcher updates cleared out the old ones. *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Gerrit Spontaneous Shutdown During Summit (clarkb 20251028) *** The Gerrit server was discovered to be in a shutdown state with no recorded errors *** The Nova team indicates this could happen if the hypervisor is unable to allocate memory. The host would be shutdown and Nova would notice but not record an error *** Its possible the underlying cause is something else; either way we need to work with Vexxhost to identify the source of the problem and mitigate it ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Can we automate cleanups for reprepro content? In other words are the steps cheap enough during the noop case to run with our regular reprepro update script? *** Additional volumes to look into cleanups for include ceph, docker, and puppetlabs package mirrors as well as our python wheel caches *** CentOS 9 Stream grows at a rate that requires regular quota bumps. Is there anything we can do about that? *** With these cleanups and server upgrades completed we can probably start to consider adding new mirrors again. **** Trixie, CentOS 10 Stream, Rocky Linux ? ** Zuul Launcher Updates (clarkb 20250923) *** NODE_FAILURES created by a bug in zuul-launcher **** https://review.opendev.org/c/zuul/zuul/+/964893 is the fix which we should roll out as soon as it has merged ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Etherpad 2.5.1 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.1/CHANGELOG.md *** There is an even newer 2.5.1 release. We'll need to recycle the test nodes and check if CSS has improved *** Note we shouldn't upgrade until after the PTG as the event relies on etherpad. ** Gitea 1.24.7 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/964899/ Upgrade Gitea to 1.24.7 ** Gitea Performance (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/964728 Don't allow direct backend access **** Some performance issues seem related to crawlers hitting backends directly so they aren't balanced properly **** Most recent issue was via traffic to the load balancer frontend so this won't fix everything **** Means we'll have to take extra steps to test backends directly (ssh port forwards) ** PBR Updates to Handle Setuptools Deprecations (clarkb 20251028) *** Heads up setuptools has an October 31 deadline for no longer relying on easy_install tooling *** https://review.opendev.org/c/openstack/pbr/+/964712/ and children aim to address this * Open discussion

1 0