- service-discuss - lists.openstack.org

Team Meeting Agenda for October 22, 2024
by Clark Boylan 21 Oct '24

21 Oct '24

We will meet with this agenda on October 22, 2024 at 19:00 UTC in #opendev-meeting. I know that this is a busy week for many with the PTG, but we missed our meeting last week. I want to make sure we've got a bit of time to catch up on anything important that we may be ignoring due to distractions like the PTG. == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 ** PTG is happening this week. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 *** An alternative would be to attach a new empty volume and shuffle mounts around. Then let new backups fill the new space and leave old backups off to the side. ** Updating Gerrit Cache Sizes (clarkb 20241022) *** Noticed Gerrit pruning its caches on startup after a recent upgrade. Some of the sizes were quite large compared to their configured maximum *** https://review.opendev.org/c/opendev/system-config/+/932763 increase sizes for four gerrit caches ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. * Open discussion

1 0

Cancelling the October 15, 2024 OpenDev Team Meeting
by Clark Boylan 09 Oct '24

09 Oct '24

We are cancelling next week's team meeting that would normally occur on October 15, 2024. A number of us will be unavailable that day. It doesn't make sense to try and organize a meeting for one or two people. As usual feel free to bring up any items for discussion on this mailing list or on IRC in the #opendev channel. We can discuss things, as able, outside of the meeting via these channels. Clark

1 0

Team Meeting Agenda for October 8, 2024
by Clark Boylan 07 Oct '24

07 Oct '24

== Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** Switching OpenDev Zuul Tenants to Ansible 9 by Default (clarkb 20241008) *** Some tenants (opendev and zuul) have already switched. There were some minor issues that needed to be addressed *** Testing with devstack + tempest looks good: https://review.opendev.org/c/openstack/tempest/+/929684 *** https://review.opendev.org/c/openstack/project-config/+/931320 switches the remaining tenants (including openstack) to Ansible 9 by default. ** Rocky Package Mirror Creation (clarkb 20240924) *** No rsync script change pushed yet but one is expected soon. ** Rackspace's Flex Cloud (clarkb 20240604) *** Application credentials and general container/object creation works. *** Bug in openstack ansible modules prevents multipart uploads from reporting success within ansible (though it does upload behind the scenes) **** https://review.opendev.org/c/openstack/ansible-collections-openstack/+/9265… ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 is the last current open change related to this work ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 *** An alternative would be to attach a new empty volume and shuffle mounts around. Then let new backups fill the new space and leave old backups off to the side. ** Mailman 3 Upgrade (clarkb 20241008) *** https://review.opendev.org/c/opendev/system-config/+/930236 will upgrade our installation of mailman 3 to the latest versions *** The containers themselves should handle required db migrations on startup ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. * Open discussion

1 0

Team Meeting Agenda for October 1, 2024
by Clark Boylan 30 Sep '24

30 Sep '24

We will meet with this agenda on October 1, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** OpenStack Release Wednesday (clarkb 20241001) *** OpenStack will be publishing its next release on Wednesday from approximately 1000-1500 UTC. *** https://review.opendev.org/c/openstack/project-config/+/930709 will be landed prior to the release to remove a semaphore that slows things down. ** Rocky Package Mirror Creation (clarkb 20240924) *** No rsync script change pushed yet but one is expected soon. ** Rackspace's Flex Cloud (clarkb 20240604) *** Next up is figuring out how to use this cloud for swift storage of CI job logs. ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 is the last current open change related to this work ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? * Open discussion

1 0

Team Meeting Agenda for September 24, 2024
by Clark Boylan 23 Sep '24

23 Sep '24

We will meet with this agenda on September 24, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** Rocky Package Mirror Creation (clarkb 20240924) *** No rsync script change pushed yet but one is expected soon. ** Rackspace's Flex Cloud (clarkb 20240604) *** Next up is figuring out how to use this cloud for swift storage of CI job logs. ** Implementing DNS over TLS on Test Nodes (clarkb 20240924) *** OVN MitMs DNS traffic made in the clear this has created problems for some users **** https://serverfault.com/questions/1134180/how-to-disable-or-fix-openstack-i… *** We should be able to mitigate this using DNS over TLS now that some clouds appear to be using OVN *** https://review.opendev.org/c/opendev/base-jobs/+/929960 Adds this to configure-unbound role in base-jobs *** If that approach seems reasonable clarkb can split things up to go through a full round of base-test base job testing ** Etherpad 2.2.5 Upgrade (clarkb 20240820) *** Just wanted to close this out. We upgraded to a proper 2.2.5 release and meetpad seems to work. *** There is a new v2 API we can look into using but our tests indicate v1 still works for now. ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 is the last current open change related to this work ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Gitea 1.22.2 Upgrade (clarkb 20240924) *** https://review.opendev.org/c/opendev/system-config/+/930217 Change to upgrade to 1.22.2 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades * Open discussion

1 0

Team Meeting Agenda for September 17, 2024
by Clark Boylan 16 Sep '24

16 Sep '24

We will meet on September 17, 2024 at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rackspace's Flex Cloud (clarkb 20240604) *** Continue to keep an eye out for networking symptoms similar to slow ssh hostkey scanning *** This is a floating IP cloud and jobs may need to adjust their use of IP addresses *** We've added this cloud to the nested virt labels (as well as openmetal) *** Next up I think we can figure out how to use swift in this cloud for job logs ** Etherpad 2.2.4 Upgrade (clarkb 20240820) *** We upgraded but that broken the meetpad integration. *** Upstream had already fixed the issue but hadn't released it yet. We have deployed a development version to address the problem. *** Ideally we'll get a 2.2.5 release soon that we can quickly update to. ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** After updating our default nodeset to Noble we discovered that ansible-lint with all of our pins cannot run on Noble due to incompatibilities with python 3.12. *** Updating to latest ansible-lint and the current version of Zuul's Ansible install (>=8,<9) does work but introduces new rules and lint errors *** https://review.opendev.org/c/openstack/project-config/+/926848 *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job * Open discussion

1 0

Team Meeting Agenda for September 10, 2024
by Clark Boylan 09 Sep '24

09 Sep '24

We will meet with this agenda on September 10, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rackspace's Flex Cloud (clarkb 20240604) *** This cloud is now in use with max-servers set to 32. *** We've seen issues scanning ssh hostkeys which may be related to networking *** At least one job discovered the nodes use floating IPs that hard way trying to bind to the fip address. ** Etherpad 2.2.4 Upgrade (clarkb 20240820) *** Etherpad 2.2.2 has been released which breaks compatibility with our ep_headings plugin. **** ep_headings2 appears to work though *** https://review.opendev.org/c/opendev/system-config/+/926078 ** OSUOSL ARM Cloud Issues (clarkb 20240827) *** Kolla has discovered that jobs take significantly more time on OSUOSL nodes than they did on linaro and seem to fail more than they succeed. Other projects are having fewer issues. **** IO throughput seems to be the main culprit. OSUOSL has been improving the nodes and now kolla-build-debian-aarch64 builds succeed occasionally (not not consistently). ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** After updating our default nodeset to Noble we discovered that ansible-lint with all of our pins cannot run on Noble due to incompatibilities with python 3.12. *** Updating to latest ansible-lint and the current version of Zuul's Ansible install (>=8,<9) does work but introduces new rules and lint errors *** https://review.opendev.org/c/openstack/project-config/+/926848 *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 ** Zuul-launcher image builds (corvus 20240910) *** DIB roles/job landed in zuul-jobs *** In what repo should we define our central image build jobs? * Open discussion

1 0

Team Meeting Agenda For August 27, 2024
by Clark Boylan 26 Aug '24

26 Aug '24

== Agenda for next meeting == * Announcements ** Clarkb will miss the next meeting (September 3, 2024) due to the Summit. * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rackspace's Flex Cloud (clarkb 20240604) *** Fungi and Clarkb spoke to rax about this and tl;dr is it is an entirely new cloud generation and we can use our existing accounts *** Fungi got clouds.yaml updated and is able to talk to the new cloud region. *** Need to enroll the cloud into our cloud launcher to configure networks, subnets, routers, security groups, and ssh keys. **** https://review.opendev.org/c/opendev/system-config/+/927214 *** Need to figure out flavors and quotas. *** Optionally upload the upstream Ubuntu Noble Cloud Image ** Etherpad 2.2.2 Upgrade (clarkb 20240820) *** Etherpad 2.2.2 has been released which breaks compatibility with our ep_headings plugin. *** fungi tested etherpad 2.2.2 + ep_headings2 against our prod database and found that it seems to work. We can probably go ahead and upgrade but maybe be extra cautious and do a db backup before the upgrade? *** https://review.opendev.org/c/opendev/system-config/+/926078 ** Service Coordinator Election (clarkb 20240820) *** clarkb was the only nomination. Is Service Coordinator for six more months by default *** If a nomination was missed now is the time to bring that up. ** OSUOSL ARM Cloud Issues (clarkb 20240827) *** nb04 ran out disk on /opt which broke image builds *** Cleaning up /opt/dib_tmp didn't produce working builds because losetup was also out of device space *** Current status is server has been rebooted to clear out stale losetup loopback devices and leaked mounts with another round of disk cleaning in progress. *** Separately Kolla has discovered that jobs take significantly more time on OSUOSL nodes than they did on linaro and seem to fail more than they succeed. Other projects are having fewer issues. **** Not sure yet what the underlying issue is but hunches are that it is related to IO (since Neutron unittests don't seem super bothered but Kolla container builds are). ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** After updating our default nodeset to Noble we discovered that ansible-lint with all of our pins cannot run on Noble due to incompatibilities with python 3.12. *** Updating to latest ansible-lint and the current version of Zuul's Ansible install (>=8,<9) does work but introduces new rules and lint errors *** https://review.opendev.org/c/openstack/project-config/+/926848 *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 **** This change fails due to an inability to build openafs packages on CentOS 9 Stream arm64 images due to out of date kernels. Hopefully fixing those image builds will correct this problem. * Open discussion

1 0

August 2024 OpenDev Service Coordinator Election
by Clark Boylan 20 Aug '24

20 Aug '24

Hello everyone, It is that time of year again where we need to elect a new Service Coordinator. Last election held its candidate nomination period from February 6 to 20, 2024; due to having only a single candidate no voting was necessary. Six months from that time period is August 6 to 20, 2024. The August 2024 OpenDev Service Coordinator election will run on this timeline: candidate nominations open from August 6 to August 20, 2024. Then if we have more than one candidate we will vote to select the winning candidate from August 21 to August 28, 2024. The electorate will be contributors to OpenDev git repos since August 20, 2023 (basically the year prior to the election). Candidates can respond to this email thread indicating their self nomination or start a new thread on this mailing list to do so. We will start and end days at midnight UTC to avoid any confusion around specific timezones. I would encourage anyone who is interested in running to do so. I can answer questions about the role, and am happy to support a new Service Coordinator. Finally, we've got a bit of time before August 6. If there are questions, concerns or feedback on this process please feel free to bring them up here on the mailing list or more directly if necessary. Clark

1 1

OpenDev Team Meeting Agenda for August 20, 2024
by Clark Boylan 19 Aug '24

19 Aug '24

We will meet with this agenda on August 20, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb will miss the September 3, 2024 meeting due to the Summit. * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Clarkb is trying to meet with rax this week. ** Etherpad 2.2.2 Upgrade (clarkb 20240820) *** Etherpad 2.2.2 has been released which breaks compatibility with our ep_headings plugin. There is an ep_headings2 alternative plugin that is likely not compatible with ep_headings content, but it isn't clear if that will break etherpad entirely or just force people to re apply heading formatting to pads they care about. *** We need to do more testing of the behaviors so that we are not surprised if/when we upgrade. *** https://review.opendev.org/c/opendev/system-config/+/926078 ** Service Coordinator Nominations End August 20 (Day of the Meeting) (clarkb 20240820) *** No nominations have arrived yet. Is clarkb it by default again? *** clarkb can submit a nomination after the meeting to make it official if that is the path we're taking ** Updating the default Zuul Nodeset to Noble on August 21, 2024 (clarkb 20240820) *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** https://review.opendev.org/c/opendev/base-jobs/+/926360 *** Just a reminder that we are planning to make this change soon. Be on the lookout for unexpected fallout. ** openstack.org DNS Hosting Moving to CloudFlare (clarkb 20240820) *** Once this move is done we won't be able to update the zone in rax and have that reflect in global DNS for us. *** Currently fungi can make edits to the new zone contents. clarkb should be able to at some point after getting access. ** OpenMetal Cloud Cert Expiring Soon (clarkb 20240820) *** This cloud was deployed using Kolla's LetsEncrypt integration *** It is likely that we just need to retrigger kolla to request a new cert. *** Can we put that on a cron job? Maybe another item to feedback to OpenMetal so that this works for everyone and not just us. * Open discussion

1 0