- service-discuss - lists.openstack.org

Team Meeting Agenda for May 21, 2024
by Clark Boylan 21 May '24

21 May '24

We will meet with this agenda on May 21, 2024 (today) at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP ends May 29, 2024 * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Tonyb looking at cacti and the wiki as next items. ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Mirrors for x86_64 and arm64 are in place. We should be able to add image builds to nodepool. ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** Planning to upgrade the Gerrit server on May 31, 2024 from 1600-1700 UTC ** Linaro Cloud SSL Cert Renewal (clarkb 20240521) *** It has been a couple of months and we need to renew the cert again. Should be documented now. Any volunteers? * Open discussion

1 0

Team Meeting Agenda for May 14, 2024
by Clark Boylan 14 May '24

14 May '24

We will meet with this agenda on May 14, 2024 at 19:00 UTC in #opendev-meeting: * Announcements ** Clarkb out 5/17 and 5/20 * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Meetpad is done. Just need to clean up old servers when we feel confident we won't rollback. *** What is next? ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Next step is to add Noble to our mirrors ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** Downgrade is possible with an offline reindex using the old war. *** Things to think about: **** Do we want to enable diff3 diffs on merge changes? **** Change topic limit seems to only apply to open changes. Being over the limit doesn't seem to impact the upgrade process **** https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts **** There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely. *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Cleaning up the openstack.org DNS zone (fungi 20240514) *** Proposed for deletion: https://paste.opendev.org/show/bVHJchKcKBnlgTNRHLVK/

1 0

Team Meeting Agenda for May 7, 2024
by Clark Boylan 06 May '24

06 May '24

We will meet with this agenda on May 7, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Everything but Gerrit is done now. When do we think we want to do Gerrit? ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Next step is to add Noble to our mirrors ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** https://review.opendev.org/c/opendev/system-config/+/918333 Need updated and promoted 3.9 image before doing more testing. *** Downgrade is possible with an offline reindex using the old war. *** Things to think about: **** Do we want to enable diff3 diffs on merge changes? **** Change topic limit seems to only apply to open changes. Being over the limit doesn't seem to impact the upgrade process **** https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts **** There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely. *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Wiki SSL Cert Renewal (clarkb 20240423) *** Clarkb will renew Friday ish (May 10) * Open discussion

1 0

Team Meeting Agenda for April 30, 2024
by Clark Boylan 30 Apr '24

30 Apr '24

We will meet with this agenda on April 30, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Everything but Gerrit is done now. When do we think we want to do Gerrit? ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Do we want to prefer removing projects from zuul's tenant config or propose changes to projects to cleanup Xenial usage then if/when they break we did our best? *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Changes needed in opendev nodepool elements and glean have all merged and DIB is testing Noble builds now *** Next step is to add Noble to our mirrors (do we have room within the existing Ubuntu Volume? Is that Volume still small enough for OpenAFS) then we can add image builds to Nodepool. ** Etherpad 2.0.3 Upgrade planning (clarkb 20240416) *** https://review.opendev.org/c/opendev/system-config/+/914119 *** Etherpad 2.0.2 removes the old APIKEY API auth method and now uses Oauth2.0 client_credentials instead. The above change updates docs and testing to accommodate this change. ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** Downgrade is possible with an offline reindex using the old war. *** Things to think about: **** Do we want to enable diff3 diffs on merge changes? **** Do we think 5000 changes per topic is too few? If so we need to update a new config option to set a larger value than the default. **** We can build the docs without external fonts, but the release target doesn't do this for us. I've asked how we might do that without forking the bazel configs. **** There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely. *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Wiki SSL Cert Renewal (clarkb 20240423) *** We have just under 4 weeks to renew the cert. This is one of the last (maybe the only) manually managed ssl cert. *** Will probably delay until we have less than a week left before processing it. * Open discussion

1 0

Team Meeting Agenda for April 23, 2024
by Clark Boylan 23 Apr '24

23 Apr '24

We will meet with this agenda on April 23, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad and mailman are done. Gerrit and Gitea are all that remain. ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial is next but currently busy with PTG, Release, and other tasks. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** https://review.opendev.org/c/opendev/glean/+/915907 Glean updates for python3.12 support *** Appears we will need a debootstrap from debian unstable to build noble ** Etherpad 2.0.x Upgrade planning (clarkb 20240416) *** https://review.opendev.org/c/opendev/system-config/+/914119 *** Etherpad 2.0.2 removes the old APIKEY API auth method and now uses Oauth2.0 client_credentials instead. This new system is not yet released, but we are working to determine how to use it pre release. ** Add reference to the project-team-guide in the fail ci msg (fungi 20240423) *** https://review.opendev.org/914189 ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** https://www.gerritcodereview.com/3.9.html Release Notes *** Downgrade is possible with an offline reindex using the old war. *** Need to compile an upgrade checklist (go over breaking changes and also test the downgrade). *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Wiki SSL Cert Renewal (clarkb 20240423) *** We have just under 4 weeks to renew the cert. This is one of the last (maybe the only) manually managed ssl cert. *** Will probably delay until we have less than a week left before processing it. * Open discussion

1 0

Team Meeting Agenda for April 16, 2024
by Clark Boylan 16 Apr '24

16 Apr '24

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/915183 Upgrade mailman3 mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial is next but currently busy with PTG, Release, and other tasks. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** https://review.opendev.org/c/opendev/glean/+/915907 Glean updates for python3.12 support *** Appears we will need a debootstrap from debian unstable to build noble ** Review02 had an oops last night (clarkb 20240326) *** Found the server was shutdown. After giving it a few minutes to potentially resolve itself (mostly worried about cloud action) clarkb proceeded to manually start the instance then start the containers. *** mnaser reports it may have been an OOM event on the hosting side. ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Need to set a hard time for the 19th and send an announcement. ** Etherpad 2.0.x Upgrade planning (clarkb 20240416) *** https://review.opendev.org/c/opendev/system-config/+/914119 *** Etherpad 2.0.2 removes the old APIKEY API auth method and now uses Oauth2.0 client_credentials instead. This new system is not yet released, but we are working to determine how to use it pre release. * Open discussion

1 1

Cancelling Tomorrow's (April 9, 2024) Team Meeting
by Clark Boylan 09 Apr '24

09 Apr '24

At a previous meeting several people indicated a preference to cancel this week's meeting due to the PTG. Let's make that official and go ahead and say the meeting is cancelled tomorrow April 9, 2024. You can always bring up discussion on this mailing list or on IRC in #opendev for items that need to be discussed broadly before next week. Clark

1 0

Team Meeting Agenda for March 26, 2024
by Clark Boylan 26 Mar '24

26 Mar '24

We will meet with this agenda on March 26, 2024 (today) at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack Release next week then PTG the week after ** Put your PTG agenda items on the etherpad: https://etherpad.opendev.org/p/apr2024-ptg-opendev * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial is next but currently busy with PTG, Release, and other tasks. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rebuilding Gerrit Images (clarkb 20240312) *** Gerrit 3.9.2 has been released finally. *** https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 **** This will also rebuild our 3.8.4 image so we should try and restart prod gerrit on the new 3.8.4 image when available. *** Sounds like there are a number of bugfixes that a rebuild will get us. May be worth doing this just after the openstack release completes? ** Review02 had an oops last night (clarkb 20240326) *** Found the server was shutdown. After giving it a few minutes to potentially resolve itself (mostly worried about cloud action) clarkb proceeded to manually start the instance then start the containers. *** mnaser reports it may have been an OOM event on the hosting side. ** Rackspace MFA Requirement (clarkb 20240312) *** MFA is enabled. Enforcement day is today. Please lookout for any issues. ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Penciled in April 19, 2024 submit your rename requests now. ** Nodepool image delete after upload (clarkb 20240319) *** Nodepool now has the ability to delete on disk files for images after they are uploaded. We could potentially keep only small qcow2s using this functionality to save disk space. * Open discussion

1 0

Team Meeting Agenda for March 19, 2024
by Clark Boylan 19 Mar '24

19 Mar '24

We will meet with this agenda on March 19, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Put your PTG agenda items on the etherpad: https://etherpad.opendev.org/p/apr2024-ptg-opendev * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** CentOS 7 is now gone. Next up is Ubuntu Xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rebuilding Gerrit Images (clarkb 20240312) *** Gerrit 3.9.2 has been released finally. *** https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 **** This will also rebuild our 3.8.4 image so we should try and restart prod gerrit on the new 3.8.4 image when available. ** Rackspace MFA Requirement (clarkb 20240312) *** We intend on manually enabling MFA on March 20, 2024 before the March 26, 2024 deadline. ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Penciled in April 19, 2024 submit your rename requests now. ** Linaro cloud cert renewal (clarkb 20240319) *** The cert for the cloud APIs expires on April 15 and will need to be manually rotated before then. ** Nodepool image delete after upload (clarkb 20240319) *** Nodepool now has the ability to delete on disk files for images after they are uploaded. We could potentially keep only small qcow2s using this functionality to save disk space. * Open discussion

1 0

Team Meeting Agenda for March 12, 2024
by Clark Boylan 12 Mar '24

12 Mar '24

We will meet with this agenda on March 12, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** DST changes are starting to occur for some of us. Double check your timezone hasn't moved relative to the 19:00 UTC meeting time. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** CentOS 7 removal is in progress: topic: drop-centos-7 **** Announced that we would finalize CentOS 7 removal on March 15 to give OpenStack time to clean stuff up and avoid unexpected release problems *** When removing Debian mirror content afs01.dfw.openstack.org lost a volume backing /dev/vicepa and had a big sad. Not sure if related or coincidence. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rebuilding Gerrit Images (clarkb 20240312) *** Gerrit 3.9.2 has been released finally. *** https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 **** This will also rebuild our 3.8.4 image so we should try and restart prod gerrit on the new 3.8.4 image when available. ** Rackspace MFA Requirement (clarkb 20240312) *** Rackspace is requiring all accounts switch to multi factor auth March 26, 2024 *** We moved our clouds.yaml auth over to `rackspaceauth` using an `api_key` as normal auth through the API will no longer work after MFA is enabled *** Still an open question if swift log uploads for zuul jobs need changing, but indications are they already use dedicated credentials which are unlikely to be affected. *** Should we go ahead and opt into MFA now ahead of the deadline so that we can control when our API interactions will be impacted? ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Penciled in April 19, 2024 submit your rename requests now. ** PTG Planning (clarkb 20240305) *** https://ptg.opendev.org/ptg.html *** Times have been selected. Feel free to add agenda items here: https://etherpad.opendev.org/p/apr2024-ptg-opendev * Open discussion

1 0