- service-discuss - lists.openstack.org

Team Meeting Agenda for April 11, 2023
by Clark Boylan 10 Apr '23

10 Apr '23

We will meet on April 11, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** New Contributor Onboarding Wednesday April 12, 2023 at 14:00 UTC in https://meetpad.opendev.org/opendev-contributor-bootstrap-202304 * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230411) *** TODO testing of ianw's changes to generic container roles ** Bastion host (ianw 20230411) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230411) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230411) *** This was done successfully as far as I can tell *** Potentially need to manually run manage-projects to resync copyConditions for labels *** Should consider applying https://review.opendev.org/c/openstack/project-config/+/879906 or something like it when we do that to normalize indentation of configs to be what gerrit expects. *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230411) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** static.opendev.org has moved to static02. Still need to cleanup static01 when we are convinced we won't revert. **** static02 runs jammy's openafs-client package not our ppa package. *** Etherpad replacement is in process https://review.opendev.org/q/topic:add-static02 ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230411) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 *** I half expect 1.19.1 to be released soon. May make more sense to wait for that update and upgrade directly to 1.19.1 ** Quo vadis storyboard (frickler 20230411) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Wheel cache builds
by Ian Wienand 04 Apr '23

04 Apr '23

Hi there, The wheel cache builds are becoming harder and harder to maintain, so I think we need to re-evaluate what we're doing. To summarise; currently for every platform, every day * job starts with zuul clone of requirements * run bindep (for master only? ... probably wrong) and do some more (now-looking-a-bit-dubious) setup [1] * we iterate over master + stable/* and "pip wheel" build each item in requirements, putting it into a local wheel cache [2] * except for arm64, where we take the latest two branches (the choosing of which was recently broken by the change in sort ordering from the "YYYY.X" release format) * then we grep the build logs to find out which .whl files were downloaded from PyPI, and delete them from the local cache * then we move to the publish step, where we copy the wheels to AFS. This never removes, but it does overwrite (so the .whl is very likely to change every day, as timestamps, etc. mean .whl builds are not reproducable). [3] * then we make a pypi index from the files in afs [4]. * We wait for all the publishing jobs to complete successfully, then we release the AFS volumes. If any fail, we don't publish that day [5] This started a long time ago, when we had a few platforms and a few branches. We now have newton->2023.1 branches in requirements, and we currently do this for 15 different platforms. When you multiply that out, it's not sustainable. Daily build jobs are timing out now, which holds up all publishing (I think the latest release pushed us over the edge). For some years, we were not pruning wheels we downloaded from PyPI [6]. If a .whl is built and on PyPI we should get it directly from upstream -- we have a caching proxy setup for CI jobs. I have written a small tool to help us clean up our caches [7]. It would be good if we could audit this tool, and when we're happy with it's output we can look at clearing out our caches. But that still leaves what is going into them every day. Iterating every branch is fairly useless. Ideally, we'd have a matrix of platforms v branches that gave us an exact mapping of what platforms run jobs on what branches. This does not generally exist; we all have some vague ideas and the extremes are obvious (we are not running Zed jobs on centos-8, and we are not running newton jobs on Ubuntu Jammy) but the middle is fuzzy. I'd like to solicit opinions on what we want this cache to do? One compelling option is to just build master requirements into the cache. The theory being that as branches are made, the requirement must have passed through master; ergo as we have an additive cache we will have wheels built. This seems OK, but it also seems that we need a cut-off point. It doesn't seem useful to build building "master" on centos-8/xenial as the requirements are all pinning things for Python's way in advance of what's there. If we do this, how do we maintain where a platform stops building master? stable/* requirements shouldn't change much; but if they do, we should push new .whls into the cache -- how do we do that in this model? This also makes our cache "precious" in that we are never building old branches -- if we lose AFS for some reason, we have a job ahead of us to restore all the old wheels. I think a perfect solution here might involve making the entire publishing pipeline driven by changes to openstack/requirements. Firstly, we have a non-trivial amount of work to figure out modifying the release process from "everything builds and releases or nothing does" to individual builds. I think we can do this with Zuul semaphores, and there's a decent chance it was written like this because mutexes/sempahores weren't available. This would be a non-trivial amount of work, and also be handing off a significant amount of this from what has traditionally been an infra job to the requirements project. Is anyone interested on working on this? I welcome any and all suggestions on what we want out of the wheel cache and how we can achieve it :) -i [1] https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f… [2] https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f… [3] https://opendev.org/openstack/project-config/src/branch/master/roles/copy-w… [4] https://opendev.org/openstack/project-config/src/commit/6e4748ca35008a4c25e… [5] https://opendev.org/openstack/project-config/src/branch/master/playbooks/wh… [6] https://review.opendev.org/c/openstack/project-config/+/703487 [7] https://review.opendev.org/c/opendev/system-config/+/879239

3 3

Team Meeting Agenda for April 4, 2023
by Clark Boylan 03 Apr '23

03 Apr '23

We will meet with this agenda on April 4, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Docker Hub Team shutdown (clarkb 20230404) *** Docker has decided not to follow through with their plan afterall **** I think we should continue as planned just with a bit less urgency (which is nice) *** https://review.opendev.org/q/topic:tag-deletion Changes to handle publication to registries generically. *** TODO need to implement promote from intermediate registry path or use tag deletion role. ** Bastion host (ianw 20230404) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230404) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230404) *** Projects renames and Gerrit 3.7.2 upgrade April 6 at 22:00 UTC **** If we rename after upgrade then reindex for renames can happen online shortening downtime **** Are we comfortable relying on jeeypb's cache believing old project names are created in Gerrit preventing recreation? **** Has the 3.7 -> 3.6 revert path been tested? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230404) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Static and etherpad replacements are in process https://review.opendev.org/q/topic:add-static02 *** launch-env on bridge cannot list rax volumes *** reverse rax dns doesn't get automatically populated ** AFS volume quotas and utilization (clarkb 20230404) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230404) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 ** Quo vadis storyboard (frickler 20230404) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Skipping the team meeting on March 28, 2023
by Clark Boylan 27 Mar '23

27 Mar '23

Heads up, we will skip tomorrow's (March 28, 2023) team meeting. A number of us are participating in the virtual PTG this week and not having another hour of meetings would be nice. As always, if there is something important, urgent, or just interesting feel free to bring it up on this mailing list or on IRC. We can continue to work asynchronously as usual. Clark

1 0

Team Meeting Agenda for March 21, 2023
by Clark Boylan 20 Mar '23

20 Mar '23

We will meet on March 21, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack release happening this week (should happen tomorrow). ** PTG next week * Actions from last meeting * Specs Review * Topics ** Docker Hub Team shutdown (clarkb 20230321) *** https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo Notes on the situation and planning *** https://quay.io/organization/opendevorg Will become the new home for opendev images *** https://review.opendev.org/c/zuul/zuul-jobs/+/877834 Role to precreate public images on quay.io *** TODO Update zuul jobs for publication and promotion to quay.io *** TODO Copy historical content form docker hub to quay.io *** TODO publish new up to date images to quay.io ** Bastion host (ianw 20230321) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups *** Fungi needed to boot old bridge to recover gpg key material and generation tools. ** Mailman 3 (clarkb 20230321) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230321) *** https://review.opendev.org/c/opendev/system-config/+/868054 Better testing of submit-requirements and copyConditions *** https://bugs.chromium.org/p/gerrit/issues/detail?id=16765 Gerrit 3.7.1 related changes bug **** https://gerrit-review.googlesource.com/c/gerrit/+/364214 Gerrit 3.7.1 related changes bug fix ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230321) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-14 are the six production giteas behind haproxy *** Gitea01-04 are receiving Gerrit replication updates until we are confident we will not fallback to needing them any longer. **** Will cleanup after the openstack release. *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns ** AFS volume quotas and utilization (clarkb 20230321) *** Fedora 36 to be cleaned up in the near future. ** Scheduling project renames (clarkb 20230321) *** April 6, 2023 at 22:00 UTC *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Quo vadis storyboard (frickler 20230321) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** Gitea 1.19 Upgrade (clarkb 20230321) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 **** Probably best to wait for the openstack release to be finished before landing this change. * Open discussion

1 0

Team Meeting Agenda for March 14, 2023
by Clark Boylan 13 Mar '23

13 Mar '23

We will meet on March 14, 2023 at 19:00 UTC (note the DST change in some locations) in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Daylight Savings time has taken effect in North America and will in a few weeks for others. This may affect the time you attend our meeting in your local timezone, but we continue to meet at 19:00 UTC. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230314) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230314) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230314) *** https://review.opendev.org/q/topic:gerrit-s-r-3.7 Cleaning up deprecated Gerrit ACL stuff *** We discovered that 'and' and 'AND' are not equivalent in Gerrit search queries. *** Conversions to submit requirements and CopyConditions continue. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230314) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-14 are the six production giteas behind haproxy *** Gitea05-08 have been deleted and removed from DNS *** Gitea01-04 are receiving Gerrit replication updates until we are confident we will not fallback to needing them any longer. *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns ** AFS volume quotas and utilization (clarkb 20230314) *** Quotas increased last week on a number of volumes to avoid hitting limits *** Addition of Fedora 37 and removal of Fedora 35 seems to have resulted in a net increase in data storage. *** Do we need to be thinking about increasing the size of backing volumes? I was unable to find any short term deletion wins. ** Scheduling project renames (clarkb 20230314) *** April 7, 2023 are things still looking good for this date? *** We should pick a time of day and announce it. ** Quo vadis storyboard (frickler 20230314) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Modernising ACLs for Gerrit 3.7
by Ian Wienand 06 Mar '23

06 Mar '23

Hi, The changes to update copy-conditionals and submit-requirements have multiplied a bit, so I thought it might be good to have a bit of a written-down plan to get us ready for Gerrit 3.7. I think we should probably start with a manual update to All-Projects, as we don't really manage that in Ansible. The two reviews to update the bootstrap documentation should serve as review for the production change: https://review.opendev.org/c/opendev/system-config/+/876236 https://review.opendev.org/c/opendev/system-config/+/876237 I've consolidated those changes an actual diff to our production All-Projects/project.config shown in https://paste.opendev.org/show/bmy36m7TO2b4cpndO5gO/ I have that in a checkout, so if we're happy with that, I can manually push it (or will rework for comments). I think this gives us a good place to start and covers both the copy-conditions and submit-requirements changes for the top-level All-Projects. Updating the copy-conditions for the project ACL's is already reviewed in https://review.opendev.org/c/openstack/project-config/+/867931 I think I'll feel better about that having already done All-Projects and seen that working, but that just needs to be applied. Then we need to move onto submit-requirements for the project ACL's. Firstly https://review.opendev.org/c/openstack/project-config/+/875992/1 is just mechnical to handle the arguments in the normalize tool. The first change https://review.opendev.org/c/openstack/project-config/+/875804/3 updates labels using "function = NoOp" to also have a no-op submit-requirement. We know from testing that labels must still specify a function. Strictly, I don't think the s-r is necessary; but it does seem that we can use this to add a description explaining what the label is doing. I also think this future-proofs us against the removal of label functions (although it is currently unclear how future gerrit will deal with this). The second change is similar https://review.opendev.org/c/openstack/project-config/+/875993/2 except this is a large selection of Backport-Candidate labels using "function = NoBlock", which is alised to NoOp. For consistency, this is all changed to NoOp and, again, given a submit-requirement as well. Following that is https://review.opendev.org/c/openstack/project-config/+/875994/2 which is a mechanical change again for the normalize tool, and then the two final changes https://review.opendev.org/c/openstack/project-config/+/875995 https://review.opendev.org/c/openstack/project-config/+/875996 convert more complex labels that have blocking semantics submit-requirements. I think these just want careful review to ensure I've translated what they're doing correctly. At that point, I think we're fully migrated off the deprecated ACL options, clearing our path for a Gerrit 3.7 upgrade, general notes for which are kept at https://etherpad.opendev.org/p/gerrit-upgrade-3.7 -i

1 1

Team Meeting Agenda for March 7, 2023
by Clark Boylan 06 Mar '23

06 Mar '23

We will meet on March 7, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230307) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230307) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230307) *** https://review.opendev.org/q/topic:gerrit-s-r-3.7 Cleaning up deprecated Gerrit ACL stuff *** We'll defer on java 17 for now as there is no rush and that gives upstream time to fix the issues *** Working with upstream to test ianw's fix for ssh channel tracking. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230307) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-12 are the only four giteas behind the load balancer currently *** gitea01-04 remain in the replication config and can be reenabled in the load balancer via the control socket as they were merely disabled. *** Work to cleanup gitea05-08 is in progress *** Do we need gitea13-16? *** Name servers and etherpad next. **** Etherpad after the PTG? ** AFS volume quotas (clarkb 20230307) *** A number of mirror AFS volumes appear to have steady growth over time. Should we bump their quotas? **** debian-security in particular is right near the limit ** Scheduling project renames (clarkb 20230307) ** Quo vadis storyboard (frickler 20230307) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Re: Intent to grant control of x/virtualpdu to OpenStack community
by Stéphane Robert 01 Mar '23

01 Mar '23

Hi, I just added Riccardo Pittau to the virtualpdu-core and virtualpdu-release groups. I didn't do it sooner because I could not remember my INAP credentials and could not use the "Forgot Password" functionality... Finally, after trying several combinations I got through. Hope this helps. If there is anything else I can do, feel free to reach out. I'll try my best. Stephane Le mer. 1 mars 2023, à 15 h 50, Jeremy Stanley <fungi(a)yuggoth.org> a écrit : > The OpenStack Ironic project relies on VirtualPDU, which is no > longer actively developed. Ironic contributors reached out to the > VirtualPDU maintainers with an offer to officially assume > maintenance responsibilities in order to avoid forking it, and > received a response from one (Mathieu Mitchell) who indicated > support for that plan. Unfortunately, communication died off shortly > thereafter, and the Ironic team has been unable to raise any of the > current maintainers to actually add their access to the x/virtualpdu > project in OpenDev's Gerrit. > > We don't have a documented process since this is the first time it's > really come up, but I'm officially announcing that I intend to use > my administrative permissions as an OpenDev sysadmin to add > membership of an OpenStack Ironic team representative to the > following Gerrit groups if no objections are raised before > Wednesday, March 8: > > * virtualpdu-core > * virtualpdu-release > > This will effectively grant full control of the x/virtualpdu > repository to OpenStack Ironic contributors. Please follow up to > the service-discuss(a)lists.opendev.org or > openstack-discuss(a)lists.openstack.org mailing list with any concerns > as soon as possible, or you can feel free to reach out to me > directly by email or in IRC (fungi in the #opendev channel on the > OFTC network). > -- > Jeremy Stanley >

2 1

Team Meeting Agenda for February 28, 2023
by Clark Boylan 27 Feb '23

27 Feb '23

We will meet on February 28, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** clarkb was the only nominee in the Service Coordinator election. Due to lack of alternative options there will be no polls, and clarkb wins by default. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230214) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230214) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230214) *** https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs *** https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17 *** https://github.com/apache/mina-sshd/issues/319 Gerrit SSH issues with flaky networks. *** Clarkb stuck the last two issues on the Gerrit Community Meeting agenda and plans to attend March 2 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230214) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09 has been launched **** Ended up using a modern vexxhost v3 flavor with 32GB of memory and 8vcpus. No BFV. Can probably replace 8 old servers with ~4 new servers of this size. **** Ansible deployed a gitea on it **** clarkb manually copied the database over from gitea01 and applied it to gitea09 for redirects **** clarkb also copied logo files for unused org names to keep their logos showing up **** Gerrit is currently (should be done by the meeting) syncing git repos to gitea09 **** https://review.opendev.org/c/opendev/system-config/+/874340/ and child will make it easier to add new giteas and remove old ones. ** Quo vadis storyboard (frickler 20230214) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** Handing over x/virtualpdu to OpenStack Ironic PTL (fungi 20230228) *** The Ironic team has received agreement from a current maintainer, but since ghosted; proposing a communication plan to move forward. * Open discussion

1 0