- service-discuss - lists.openstack.org

Team Meeting Agenda for March 12, 2024
by Clark Boylan 12 Mar '24

12 Mar '24

We will meet with this agenda on March 12, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** DST changes are starting to occur for some of us. Double check your timezone hasn't moved relative to the 19:00 UTC meeting time. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** CentOS 7 removal is in progress: topic: drop-centos-7 **** Announced that we would finalize CentOS 7 removal on March 15 to give OpenStack time to clean stuff up and avoid unexpected release problems *** When removing Debian mirror content afs01.dfw.openstack.org lost a volume backing /dev/vicepa and had a big sad. Not sure if related or coincidence. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rebuilding Gerrit Images (clarkb 20240312) *** Gerrit 3.9.2 has been released finally. *** https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 **** This will also rebuild our 3.8.4 image so we should try and restart prod gerrit on the new 3.8.4 image when available. ** Rackspace MFA Requirement (clarkb 20240312) *** Rackspace is requiring all accounts switch to multi factor auth March 26, 2024 *** We moved our clouds.yaml auth over to `rackspaceauth` using an `api_key` as normal auth through the API will no longer work after MFA is enabled *** Still an open question if swift log uploads for zuul jobs need changing, but indications are they already use dedicated credentials which are unlikely to be affected. *** Should we go ahead and opt into MFA now ahead of the deadline so that we can control when our API interactions will be impacted? ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Penciled in April 19, 2024 submit your rename requests now. ** PTG Planning (clarkb 20240305) *** https://ptg.opendev.org/ptg.html *** Times have been selected. Feel free to add agenda items here: https://etherpad.opendev.org/p/apr2024-ptg-opendev * Open discussion

1 0

Team Meeting Agenda for March 5, 2024
by Clark Boylan 05 Mar '24

05 Mar '24

We will meet with this Agenda on March 5, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** OpenSUSE Leap and Debian Buster are gone now *** Have started with CentOS 7 removal: topic: drop-centos-7 **** Announced that we would finalize CentOS 7 removal on March 15 to give OpenStack time to clean stuff up and avoid unexpected release problems *** When removing Debian mirror content afs01.dfw.openstack.org lost a volume backing /dev/vicepa and had a big sad. Not sure if related or coincidence. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. ** Project Renames (clarkb 20240227) *** Do we need to update the tooling to move the replication waiting queue aside during the rename process? *** Penciled in April 19, 2024 submit your rename requests now. ** PTG Planning (clarkb 20240305) *** https://ptg.opendev.org/ptg.html *** Very little schedule content is in place at this point. Clarkb proposes we do something like Wednesday 0400-0600 and Thursday 1400-1600UTC. Gives enough time between blocks to catch up on sleep. * Open discussion

1 0

Team Meeting Agenda for February 27, 2024
by Clark Boylan 27 Feb '24

27 Feb '24

We will meet with this agenda on February 27, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Service Coordinator Election Results (clarkb 20240227) *** Single nominee was clarkb *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Paste, Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. Starting with Paste due to its simplicity. *** https://review.opendev.org/c/opendev/system-config/+/909471 Upgrade paste.o.o mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** OpenSUSE Leap has been removed **** topic:drop-opensuse *** Debian Buster is in progress **** topic:drop-buster *** CentOS 7 and Ubuntu Xenial still to do. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. ** Gitea 1.21.7 Upgrade (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/909941 ** git-review vendoring the commit message hook (clarkb 20240227) *** https://review.opendev.org/c/opendev/git-review/+/910275 ** Project Renames (clarkb 20240227) *** Do we need to update the tooling to move the replication waiting queue aside during the rename process? * Open discussion

1 0

Clarkb's Service Coordinator Nomination for February 2024
by Clark Boylan 20 Feb '24

20 Feb '24

As previously mentioned on this list [0] it is Service Coordinator nomination time. In fact the nomination period ends about two hours after I send this email out. There have been no other nominees, and I am willing to continue operating in this capacity. Consider this my self nomination for the Service coordinator position. I continue to think it would be great to have others in the role, and I will happily support whoever that may be in the future should we have additional volunteers. Until that time, you can expect I'll continue to try and push for the sustainability of our services. We have been able to lean on Zuul, Ansible, Containers, and CI testing to reliably keep our services up to date, and we should continue to do this or similar. The consistent deployment practices reduces mental overhead understanding how things operate, and our CI testing gives us confidence when we make changes. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… Thank you for your consideration, Clark

1 0

Team Meeting Agenda for February 20, 2024
by Clark Boylan 20 Feb '24

20 Feb '24

We will meet with this agenda on February 20, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… *** No nominees as of the time the agenda was put together. Nominations stay open 4 hours beyond the end of our meeting. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookworm" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** PrePTG Summary (clarkb 20240220) *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… *** Code review practices are worth reviewing for accuracy *** Happy to followup on other topics or have others write up summaries from their perspective. ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Paste, Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. Starting with Paste due to its simplicity. ** AFS Mirror cleanups (clarkb 20240220) *** We will remove OpenSUSE Leap, Debian Buster, CentOS 7, and Ubuntu Xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. * Open discussion

1 0

Summary of Discussion from the OpenDev PrePTG
by Clark Boylan 16 Feb '24

16 Feb '24

This week's OpenDev PrePTG event resulted in a lot of successful discussion. We captured notes in the agenda Etherpad [0] alongside the agenda items and those interested should probably refer to that document after reading my summary. I'll try to keep this summary as brief as possible and will focus on actionable outcomes as a result. At a high level we are comfortable with the code review practices we have been employing and will not be making any drastic changes. We did note that we should codify these practices a bit more than we have though. My first attempt at this would be: * OpenDev code reviewers should be empowered to exercise judgement when applying single core reviewer approvals. It is OK to single core approve changes when reviewers are comfortable the change is low risk or low impact. * Holding off on approval despite a +2 is appropriate if the reviewer is unable to follow the change through merging and deployment. * It is generally a good idea to follow up on approved changes you have made to ensure they have deployed successfully. * Code pushed by another core reviewer is assumed to be implicitly approved by that approver unless they explicitly annotate the change with an indication it should not be merged (WIP/DNM/-2/etc). We also discussed the possibility of adding new code reviewers through the TaCT SIG. Hopefully, a larger group of reviewers looking at OpenStack related configs will enable more time for OpenDev work from OpenDev core reviewers. In the short term there are a number of tasks that we identified we could be working on: * Clarkb will be reaching out to our cloud providers to reengage with them. We'll use this as an opportunity to determine if there might be new tools we can take advantage of (Kubernetes as a service for example), or if expected updates have been made (has OpenMetal's deployment tooling been updated). * We will proceed with upgrading MariaDB containers using the MARIADB_AUTO_UPGRADE env var flag. MariaDB seems confident that upgrades between the versions we run should be quick and painless. * The planned Prometheus service deployment can be picked up and completed. This would be a good task for a new contribute. * The Grafana container image can be unpinned. This is also a good task for a new contributor. * OpenDev git repo migration into the OpenDev Zuul tenant should continue. We think that the vast majority of our repos can either be moved or retired. Then we'll coordinate moves for system-config, project-config, and the DNS zone repos. * We can clean up Nodepool images and mirror content for OpenSUSE LEAP, Debian Buster, CentOS 7, and Ubuntu Xenial. On a number of topics we realized that more investigation is required: * The Podman, Docker, Docker Compose situation needs to be further investigated and tested. In particular OpenDev is interested in finding the least painful way to host images off of docker hub while still enjoying speculative container testing within Zuul jobs. Corvus is hoping to be able to look into this again. * Fungi will check if storyboard's container deployments can be resurrected in order to redeploy storyboard on the latest version of storyboard. Once that is decided we will be sending email about plans to sunset and retire the storyboard service. * We can use mirror web server logs to determine which mirror paths are most actively used. This information can then be used to decide on additional mirror content to prune. We discussed a number of other topics without any solid conclusion. I want to call out two in particular that I would like the team to followup on, but please read the Etherpad nodes for complete coverage. First, is the question of how we want to handle email hosting for opendev.org. We can either host it ourselves or pay for hosted services which appear to be very reasonably priced. Second, is the management of some sort of backlog with indicators for new contributor and possible internship opportunities on the backlog items. Tonyb indicated a willingness to help mentor students involved in the Foundation's university partnerships through projects if we have good opportunities for them. I am happy to answer any questions related to the event that you might have, or followup on agenda items I haven't already covered above. There was more discussion than I managed to cover in this one email, but I think it is long enough already. It might also be worthwhile to get other perspectives as mine is sure to have biases related to my work within OpenDev that are different than yours. [0] https://etherpad.opendev.org/p/opendev-preptg-202402 Clark

1 0

Team Meeting Agenda for February 13, 2024
by Clark Boylan 13 Feb '24

13 Feb '24

We will meet with this agenda on February 13, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Keycloak upgrade struggles (clarkb 20240130) *** A new keycloak03 server is up and ready for use. This server is a redeployment of latest keycloak and a real database backend. ** Etherpad 1.9.7 Upgrade (clarkb 20240206) *** https://review.opendev.org/c/opendev/system-config/+/908328 ** Zuul Circular Dependency Refactor (clarkb 20240213) *** This has been deployed to OpenDev in a prerelease form. If you notice problems please let the Zuul project know *** Zuul has temporarily enabled this functionality within its tenant for testing/dogfooding purposes, but we don't expect OpenDev to enable this generally. * Open discussion

1 0

Team Meeting Agenda for February 6, 2024
by Clark Boylan 06 Feb '24

06 Feb '24

We will meet with this agenda on February 6, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** Python container updates (tonyb 20230718) *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Keycloak upgrade struggles (clarkb 20240130) *** Keycloak converted its runtime from wildfly to quarkus. This led to legacy wildfly images and modern quarkus images that are not super compatible with each other *** A switch from the legacy wildfly to quarkus image failed because the h2 database isn't very portable between the two. (Different file paths, filenames and db credentials). *** Pivoting to a new latest release deployment using a mariadb. Will require us to rebuild the zuul auth stuff if we can't port the db content across. ** Gitea Upgrade and DB Confg Changes (clarkb 20240206) *** https://review.opendev.org/c/opendev/system-config/+/907500 Increase DB connection limits after HTTP 500 errors due to hitting the limit *** https://review.opendev.org/c/opendev/system-config/+/907472 Upgrade to the latest bugfix release ** Etherpad 1.9.7 Upgrade (clarkb 20240206) *** No change yet. The changelog indicates this update impacts how plugins are installed. We should make sure the plugins we rely on function in our images builds. * Open discussion

1 0

OpenDev Team Meeting Agenda for January 30, 2024
by Clark Boylan 30 Jan '24

30 Jan '24

We will meet on January 30, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** Python container updates (tonyb 20230718) *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Broken wheel build issues (frickler 20231217) *** https://review.opendev.org/c/openstack/project-config/+/907150 Will fix centos 8 and 9 stream *** CentOS 7 appears to be failing now too. ** Keycloak upgrade struggles (clarkb 20240130) *** Keycloak converted its runtime from wildfly to quarkus. This led to legacy wildfly images and modern quarkus images that are not super compatible with each other *** A switch from the legacy wildfly to quarkus image failed because the h2 database isn't very portable between the two. (Different file paths, filenames and db credentials). *** Pivoting to a new latest release deployment using a mariadb. Will require us to rebuild the zuul auth stuff if we can't port the db content across. ** InMotion/OpenMetal Cloud RabbitMQ Problems (clarkb 20240130) *** RabbitMq crashed and got into an unhappy state. *** Naive restarts were not sufficient, but using some kolla rabbitmq restart made things happy again. * Open discussion

1 0

OpenDev Pre PTG Planning
by Clark Boylan 24 Jan '24

24 Jan '24

The OpenDev team is planning a pre PTG event where the OpenDev team can get together to prioritize its backlog and do some planning for the year. We will meet on February 14 and 15 from 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. An agenda is being built in this etherpad: https://etherpad.opendev.org/p/opendev-preptg-202402. We will use the corresponding meetpad room, https://meetpad.opendev.org/opendev-preptg-202402, for our meetings. Please feel free to add agenda items or add content to existing items. If there are specific topics you are particularly interested in, please note that on the etherpad so that we can do our best to accommodate time zones in our scheduling. This pre PTG is being aimed at existing contributors so that we can sync up on where we are today and what needs to be done tomorrow. If you would like to start contributing to OpenDev or have OpenDev usage questions, please reach out to us directly or join us at the PTG in April. We are happy to help. Finally, to avoid any confusion we do plan to participate in the PTG this April. Our focus at the April PTG will be more towards users and new contributors. This split allows us to dive into the pre PTG agenda earlier in the year, and many of our contributors are busy during the PTG which makes planning time during that week difficult. See you there, Clark

1 0