- service-discuss - lists.openstack.org

Team Meeting Agenda for May 23, 2023
by Clark Boylan 23 May '23

23 May '23

We will meet with this agenda on May 23, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230523) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** Discovered a deficiency in Docker impacting speculative testing of container images **** https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW **** A number of issues have been found when setting up services with podman ***** No reliable source of packages for Ubuntu older than Jammy ***** No syslog support **** Migration to podman still a reasonable goal but a longer term one. In the interim suggestion is we revert quay.io image moves for opendev back to docker hub to keep speculative image testing. Then work towards a future where we use not docker and can switch to not docker.io more easily. ** Bastion host (ianw 20230523) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230523) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230523) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230523) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** Fedora cleanup (clarkb 20230523) *** Removal of Fedora 37 mirror impacted DIB jobs. DIB can fetch from upstream. *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Proceed with removal as it seems no one is asking for Fedora specifically. ** Quo vadis storyboard (frickler 20230523) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Team Meeting Agenda for May 16, 2023
by Clark Boylan 16 May '23

16 May '23

We will meet with this agenda on May 16, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230516) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** Discovered a deficiency in Docker impacting speculative testing of container images **** https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW **** clarkb's suggestion: we build all container images with buildx (it doesn't have this problem) and use skopeo to grab the correct images: https://review.opendev.org/c/opendev/system-config/+/882977 ** Bastion host (ianw 20230516) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230516) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230516) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230516) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** AFS volume quotas and utilization (clarkb 20230516) *** Utilization climbing again. *** https://review.opendev.org/c/opendev/system-config/+/883083 Cleanup unused fedora mirror content *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Quo vadis storyboard (frickler 20230516) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Wiki cert (clarkb 20230516) *** clarkb will try to get this done in a few days once there is about a week of validity remaining. * Open discussion

1 0

Future of Fedora and CentOS Stream Test Images
by Clark Boylan 10 May '23

10 May '23

Hello everyone, Over the last couple of years we've grown support for a number of RHEL like or RHEL adjacent distros in our CI system. We have added OpenEuler, Rocky Linux, and CentOS Stream (it replaced CentOS 8). At the same time we've fallen behind producing and updating images for Fedora. Currently we have disk images and test nodes for Fedora 35 and 36. We mirror packages for Fedora 36 and 37. In the past our intention was always to primarily have a single up to date Fedora image and mirror setup for our CI system which would be Fedora 38 as of April 18, 2023. One option we have is to drop Fedora 35 and 37 from Nodepool and our mirrors, add Fedora 38, and then drop Fedora 36. This would basically maintain the status quo of the last few years. That said I think this is probably a good opportunity to rethink our approach, and we are looking for your feedback. Currently the RHEL like distros are overlapping with each other a bit and for this reason we added Rocky Linux without adding extra mirroring for it. So far this seems to be stable enough as we don't run significant numbers of jobs on Rocky Linux. I think one option available to us would be to drop Fedora mirroring entirely, then add Fedora 38 images/nodes, and then drop Fedora 36 images/nodes. One upside to this approach is it will free up disk space on our OpenAFS fileservers which are under a space crunch currently. I also wonder if we can drop Fedora entirely. It seems like CentOS Stream now occupies a similar role within our CI system. It gives us forward looking updates that will eventually be in RHEL. This allows us to test against a future state in preparation for RHEL changes. The upside to this is that Fedora's support lifetime is about 13 months but CentOS Stream releases appear to have several years of support. This should reduce the amount of mirror and image work we need to perform. Are there specific use cases that Fedora serves that CentOS Stream does not within our CI system? If so are they important enough to continue to try and run on the Fedora release treadmill? If we do want to try and keep up with Fedora do we have any concerns dropping our local mirroring of Fedora packages? Please let us know, Clark

7 14

Team Meeting Agenda for May 9, 2023
by Clark Boylan 09 May '23

09 May '23

We will meet with this agenda on May 9, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230509) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** About half of the images that need moving have changes pushed and are largely ready to merge. *** Will use the halfway point to pause and make sure everything is going well and we aren't missing anything important. ** Bastion host (ianw 20230509) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230509) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230509) *** https://review.opendev.org/c/openstack/project-config/+/882075 list little bit of ACL cleanup work *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** We should restart on updated image to pick up container move as well as theming changes. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230509) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** AFS volume quotas and utilization (clarkb 20230509) *** Utilization is slightly down compared to last week. *** Proposed the removal of Fedora mirrors https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Quo vadis storyboard (frickler 20230509) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Re: [dev][infra][qa][tact-sig] Future of Fedora and CentOS Stream Test Images
by Kashyap Chamarthy 05 May '23

05 May '23

On Wed, May 03, 2023 at 03:28:00PM +0000, Jeremy Stanley wrote: > tl;dr is that the OpenDev Collaboratory is looking at potentially > scaling back on Fedora-based test platform support. Several options > are presented in this service-discuss post: > > https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… > > If you have an opinion on using Fedora in our CI jobs, please follow > up to the discussion there. Hi, Jeremy. FWIW, I think it is still valuable to have at least the latest Fedora-only image available. Often times kernel and virtualization fixes are first fixed on Fedora -- many upstream kernel developers happen to use Fedora. Another reason I wanted to mention is the 'virt-preview' repo[1]. But I see that 'virt-preview' is also available for CentOS stream. So that's a less strong reason. That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist. [1] https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/ -- /kashyap

4 6

Team Meeting Agenda for May 2, 2023
by Clark Boylan 02 May '23

02 May '23

We will meet with this agenda on May 2, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230502) *** zookeeper-statsd is now publishing to quay.io showing that this all seems to work for OpenDev *** Need to decide on an order of operations. **** Do we do python-builder/python-base/uwsgi-base first then the consuming images? This is likely less work, but does open us to potential problems if we need quick updates of the base images and rebuilds of everything. However we are effectively rebuilding anything anyway so maybe we do this and do our best to do it in a short timeframe. **** Maybe we can pick a couple of days to sprint this and try to get as much done as possible? **** Probably a good idea to write down a plan and todo list as soon as we have an approach to track progress. Clarkb can do this. **** Don't forget to resync any images from docker hub to quay.io prior to publishing to quay.io if images have updated since the initial sync. **** We may need to manually clean up the old docker hub images on hosts after we move images. Zuul and Nodepool at least appear to need this. **** Do we need to announce this move to service-announce? We may have external image consumers. ** Bastion host (ianw 20230502) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230502) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230502) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** Gerrit 3.8 images exist now and we don't build 3.6. A couple of changes were landed to 3.7 as well to make 3.8 happy should we restart on the newer image? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230502) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver replacement largely complete. **** Do we still have cleanup todo? Removing old nodes from inventory, deleting them from the cloud, AAAA glue records? **** https://etherpad.opendev.org/p/2023-opendev-dns *** Need help with the next servers in the list. ** AFS volume quotas and utilization (clarkb 20230502) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230502) *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.2 ** Quo vadis storyboard (frickler 20230502) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Team meeting agenda for April 25, 2023
by Clark Boylan 25 Apr '23

25 Apr '23

We will meet with this agenda on April 25, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230418) *** Images have been manually copied from docker hub to quay.io. As we update jobs for each image we should manually resync to catch up. *** https://review.opendev.org/c/opendev/system-config/+/881285 WIP change to convert an image to quay.io using new jobs. *** Four images are not being migrated **** opendev/bazel **** opendev/grafana **** opendev/jitsi-meet-prosody **** opendev/jitsi-meet-web ** Bastion host (ianw 20230418) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230418) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230418) *** ACL updates to better match upstream expectations **** https://review.opendev.org/c/openstack/project-config/+/879906 *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230418) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Etherpad has been replaced. Final cleanup of etherpad01 to commence shortly. ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230418) *** Waiting for 1.19.2 to include API fixes. *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.1 ** Quo vadis storyboard (frickler 20230418) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Re: Followup on https://review.opendev.org/c/openstack/cinder/+/853785
by Jean-Sébastien BEVILACQUA 19 Apr '23

19 Apr '23

Hello, I allow myself to contact you following the email below sent to me by Clark. Indeed, I want to add a new volume driver in Cinder [0] and for that I must have a working CI which validates the tempest tests with the Lustre [1] driver. I tried for 2 weeks to set up this CI with Software Factory but I didn't manage to make it work. However, the Lustre FS being 100% open-source, I was advised to use the upstream CI directly. In gerrit I have already created a service user with username lustreci. Could you tell me how to set up and run the tests directly in the upstream CI ? For information, I already have ansible playbooks allowing to set up a Lustre FS on CentOS 8. Thanks in advance, [0] https://review.opendev.org/c/openstack/cinder/+/853785 [1] https://www.lustre.org/ Jean-Sébastien Le 19/04/2023 à 18:01, Clark Boylan a écrit : > Hello, > > I wanted to followup on your questions in > https://review.opendev.org/c/openstack/cinder/+/853785 as well as your > query submitted to https://openinfra.dev/projects/contact/. > > As mentioned by Sean on the Gerrit change the Cinder team requires > working CI for all in tree Cinder drivers. Many of the systems that > Cinder integrates with are proprietary storage systems which > necessitates the use of external (to OpenDev) third party CI as > specialized hardware and licensing requirements don't allow us to run > these upstream. > > To make this happen you need a CI system that is capable of listening > to Gerrit events, triggering builds, and reporting the results back to > Gerrit (for example Zuul/Jenkins/etc). Both OpenDev [0] and the Cinder > team [1] attempt to provide documentation, but this will always be > incomplete as we won't be aware of your local network policies, > hardware peculiarities and so on. The OpenDev team can help with > connection and account details for Gerrit, and the Cinder team should > be able to help with test specific needs (like appropriate logging, > service behavior, etc). I cannot speak to Software Factory as I have > never personally used it. > > The good news is that Lustre is open source software which can be > deployed without proprietary licensing restrictions, and there don't > appear to be specialized hardware needs either. In this case I would > test Lustre + Cinder in the upstream CI system. It looks like Cinder > is already doing this with Ceph [2]. My recommendation would be that > you shift focus from attempting to running a third party CI system to > adding a new job that runs against Cinder changes to test Cinder + > Lustre integration. > > This is something that the Cinder team should be able to help with as > they have a number of Zuul jobs already including the one that tests > against Ceph. The OpenDev team can help with higher level concerns > like Gerrit accounts, general Zuul behaviors/syntax, and CI system > limitations. You can reach out to the OpenDev team either via > service-discuss(a)lists.opendev.org or in #opendev on the OFTC IRC > network (all of this info can be found in the footer of > https://opendev.org) > > [0] > https://docs.opendev.org/opendev/system-config/latest/third_party.html > [1] https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers > [2] > https://opendev.org/openstack/cinder/src/commit/2e8aff660b18d1e8f961d6de979… > > Clark > -- Jean-Sébastien BEVILACQUA Responsable technique OSSA Tél : 07 85 55 35 11 Courriel :jsbevilacqua@linagora.com www.linagora.com

2 1

Team Meeting Agenda for April 18, 2023
by Clark Boylan 18 Apr '23

18 Apr '23

We will meet with this agenda on April 18, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230418) *** Intermediate registry publication process appears to be working. Now we need to move content and update jobs? ** Bastion host (ianw 20230418) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230418) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230418) *** ACL updates to better match upstream expectations **** https://review.opendev.org/c/openstack/project-config/+/880115 **** https://review.opendev.org/c/openstack/project-config/+/879906 *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230418) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Etherpad replacement is in process https://review.opendev.org/q/topic:add-static02 **** https://paste.opendev.org/show/brRuhPssVLSi4UnF5hcN/ Etherpad data migration process expect db dump and restore to take about half an hour each. **** Planning for Wednesday at 22:00 to start the move. ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230418) *** Gitea 1.19.1 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.1 *** Changes to authentication. Do we care? ** Quo vadis storyboard (frickler 20230418) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Stx docs version menu
by Stone, Ronald 11 Apr '23

11 Apr '23

We are having issues with the version/branch menu on the starlingx docs site. From a release branch, e.g docs.starlingx.com/r/stx.6.0 trying to switch to another release branch append the "r/stx.<version>/" to the current URL instead of pointing to https://docs.starlingx.io/r/stx.<version<https://docs.starlingx.io/r/stx.%3cversion>>. i.e. docs.starlingx.com/r/stx.6.0/r/stx.7.0 is constructed instead of the correct docs.starlingx.com/r/stx.7.0. The settings in the corresponding Sphinx conf.py file are: starlingxdocs_plus_this_version = "r/stx.6.0" starlingxdocs_plus_other_versions = [("Version 7.0","r/stx.7.0"),("Version 8.0","r/stx.8.0"),("Latest","/")] Can we confirm if these conf.py settings are correct? If they are, then we will open a bug against the template. Thanks

2 2

Team Meeting Agenda for April 11, 2023
by Clark Boylan 11 Apr '23

11 Apr '23

We will meet on April 11, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** New Contributor Onboarding Wednesday April 12, 2023 at 14:00 UTC in https://meetpad.opendev.org/opendev-contributor-bootstrap-202304 * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230411) *** TODO testing of ianw's changes to generic container roles ** Bastion host (ianw 20230411) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230411) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230411) *** This was done successfully as far as I can tell *** Potentially need to manually run manage-projects to resync copyConditions for labels *** Should consider applying https://review.opendev.org/c/openstack/project-config/+/879906 or something like it when we do that to normalize indentation of configs to be what gerrit expects. *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230411) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** static.opendev.org has moved to static02. Still need to cleanup static01 when we are convinced we won't revert. **** static02 runs jammy's openafs-client package not our ppa package. *** Etherpad replacement is in process https://review.opendev.org/q/topic:add-static02 ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230411) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 *** I half expect 1.19.1 to be released soon. May make more sense to wait for that update and upgrade directly to 1.19.1 ** Quo vadis storyboard (frickler 20230411) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Wheel cache builds
by Ian Wienand 04 Apr '23

04 Apr '23

Hi there, The wheel cache builds are becoming harder and harder to maintain, so I think we need to re-evaluate what we're doing. To summarise; currently for every platform, every day * job starts with zuul clone of requirements * run bindep (for master only? ... probably wrong) and do some more (now-looking-a-bit-dubious) setup [1] * we iterate over master + stable/* and "pip wheel" build each item in requirements, putting it into a local wheel cache [2] * except for arm64, where we take the latest two branches (the choosing of which was recently broken by the change in sort ordering from the "YYYY.X" release format) * then we grep the build logs to find out which .whl files were downloaded from PyPI, and delete them from the local cache * then we move to the publish step, where we copy the wheels to AFS. This never removes, but it does overwrite (so the .whl is very likely to change every day, as timestamps, etc. mean .whl builds are not reproducable). [3] * then we make a pypi index from the files in afs [4]. * We wait for all the publishing jobs to complete successfully, then we release the AFS volumes. If any fail, we don't publish that day [5] This started a long time ago, when we had a few platforms and a few branches. We now have newton->2023.1 branches in requirements, and we currently do this for 15 different platforms. When you multiply that out, it's not sustainable. Daily build jobs are timing out now, which holds up all publishing (I think the latest release pushed us over the edge). For some years, we were not pruning wheels we downloaded from PyPI [6]. If a .whl is built and on PyPI we should get it directly from upstream -- we have a caching proxy setup for CI jobs. I have written a small tool to help us clean up our caches [7]. It would be good if we could audit this tool, and when we're happy with it's output we can look at clearing out our caches. But that still leaves what is going into them every day. Iterating every branch is fairly useless. Ideally, we'd have a matrix of platforms v branches that gave us an exact mapping of what platforms run jobs on what branches. This does not generally exist; we all have some vague ideas and the extremes are obvious (we are not running Zed jobs on centos-8, and we are not running newton jobs on Ubuntu Jammy) but the middle is fuzzy. I'd like to solicit opinions on what we want this cache to do? One compelling option is to just build master requirements into the cache. The theory being that as branches are made, the requirement must have passed through master; ergo as we have an additive cache we will have wheels built. This seems OK, but it also seems that we need a cut-off point. It doesn't seem useful to build building "master" on centos-8/xenial as the requirements are all pinning things for Python's way in advance of what's there. If we do this, how do we maintain where a platform stops building master? stable/* requirements shouldn't change much; but if they do, we should push new .whls into the cache -- how do we do that in this model? This also makes our cache "precious" in that we are never building old branches -- if we lose AFS for some reason, we have a job ahead of us to restore all the old wheels. I think a perfect solution here might involve making the entire publishing pipeline driven by changes to openstack/requirements. Firstly, we have a non-trivial amount of work to figure out modifying the release process from "everything builds and releases or nothing does" to individual builds. I think we can do this with Zuul semaphores, and there's a decent chance it was written like this because mutexes/sempahores weren't available. This would be a non-trivial amount of work, and also be handing off a significant amount of this from what has traditionally been an infra job to the requirements project. Is anyone interested on working on this? I welcome any and all suggestions on what we want out of the wheel cache and how we can achieve it :) -i [1] https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f… [2] https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f… [3] https://opendev.org/openstack/project-config/src/branch/master/roles/copy-w… [4] https://opendev.org/openstack/project-config/src/commit/6e4748ca35008a4c25e… [5] https://opendev.org/openstack/project-config/src/branch/master/playbooks/wh… [6] https://review.opendev.org/c/openstack/project-config/+/703487 [7] https://review.opendev.org/c/opendev/system-config/+/879239

3 3

Team Meeting Agenda for April 4, 2023
by Clark Boylan 04 Apr '23

04 Apr '23

We will meet with this agenda on April 4, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Docker Hub Team shutdown (clarkb 20230404) *** Docker has decided not to follow through with their plan afterall **** I think we should continue as planned just with a bit less urgency (which is nice) *** https://review.opendev.org/q/topic:tag-deletion Changes to handle publication to registries generically. *** TODO need to implement promote from intermediate registry path or use tag deletion role. ** Bastion host (ianw 20230404) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230404) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230404) *** Projects renames and Gerrit 3.7.2 upgrade April 6 at 22:00 UTC **** If we rename after upgrade then reindex for renames can happen online shortening downtime **** Are we comfortable relying on jeeypb's cache believing old project names are created in Gerrit preventing recreation? **** Has the 3.7 -> 3.6 revert path been tested? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230404) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Static and etherpad replacements are in process https://review.opendev.org/q/topic:add-static02 *** launch-env on bridge cannot list rax volumes *** reverse rax dns doesn't get automatically populated ** AFS volume quotas and utilization (clarkb 20230404) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230404) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 ** Quo vadis storyboard (frickler 20230404) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Skipping the team meeting on March 28, 2023
by Clark Boylan 28 Mar '23

28 Mar '23

Heads up, we will skip tomorrow's (March 28, 2023) team meeting. A number of us are participating in the virtual PTG this week and not having another hour of meetings would be nice. As always, if there is something important, urgent, or just interesting feel free to bring it up on this mailing list or on IRC. We can continue to work asynchronously as usual. Clark

1 0

Team Meeting Agenda for March 21, 2023
by Clark Boylan 21 Mar '23

21 Mar '23

We will meet on March 21, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack release happening this week (should happen tomorrow). ** PTG next week * Actions from last meeting * Specs Review * Topics ** Docker Hub Team shutdown (clarkb 20230321) *** https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo Notes on the situation and planning *** https://quay.io/organization/opendevorg Will become the new home for opendev images *** https://review.opendev.org/c/zuul/zuul-jobs/+/877834 Role to precreate public images on quay.io *** TODO Update zuul jobs for publication and promotion to quay.io *** TODO Copy historical content form docker hub to quay.io *** TODO publish new up to date images to quay.io ** Bastion host (ianw 20230321) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups *** Fungi needed to boot old bridge to recover gpg key material and generation tools. ** Mailman 3 (clarkb 20230321) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230321) *** https://review.opendev.org/c/opendev/system-config/+/868054 Better testing of submit-requirements and copyConditions *** https://bugs.chromium.org/p/gerrit/issues/detail?id=16765 Gerrit 3.7.1 related changes bug **** https://gerrit-review.googlesource.com/c/gerrit/+/364214 Gerrit 3.7.1 related changes bug fix ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230321) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-14 are the six production giteas behind haproxy *** Gitea01-04 are receiving Gerrit replication updates until we are confident we will not fallback to needing them any longer. **** Will cleanup after the openstack release. *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns ** AFS volume quotas and utilization (clarkb 20230321) *** Fedora 36 to be cleaned up in the near future. ** Scheduling project renames (clarkb 20230321) *** April 6, 2023 at 22:00 UTC *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Quo vadis storyboard (frickler 20230321) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** Gitea 1.19 Upgrade (clarkb 20230321) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 **** Probably best to wait for the openstack release to be finished before landing this change. * Open discussion

1 0

Team Meeting Agenda for March 14, 2023
by Clark Boylan 14 Mar '23

14 Mar '23

We will meet on March 14, 2023 at 19:00 UTC (note the DST change in some locations) in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Daylight Savings time has taken effect in North America and will in a few weeks for others. This may affect the time you attend our meeting in your local timezone, but we continue to meet at 19:00 UTC. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230314) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230314) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230314) *** https://review.opendev.org/q/topic:gerrit-s-r-3.7 Cleaning up deprecated Gerrit ACL stuff *** We discovered that 'and' and 'AND' are not equivalent in Gerrit search queries. *** Conversions to submit requirements and CopyConditions continue. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230314) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-14 are the six production giteas behind haproxy *** Gitea05-08 have been deleted and removed from DNS *** Gitea01-04 are receiving Gerrit replication updates until we are confident we will not fallback to needing them any longer. *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns ** AFS volume quotas and utilization (clarkb 20230314) *** Quotas increased last week on a number of volumes to avoid hitting limits *** Addition of Fedora 37 and removal of Fedora 35 seems to have resulted in a net increase in data storage. *** Do we need to be thinking about increasing the size of backing volumes? I was unable to find any short term deletion wins. ** Scheduling project renames (clarkb 20230314) *** April 7, 2023 are things still looking good for this date? *** We should pick a time of day and announce it. ** Quo vadis storyboard (frickler 20230314) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Modernising ACLs for Gerrit 3.7
by Ian Wienand 07 Mar '23

07 Mar '23

Hi, The changes to update copy-conditionals and submit-requirements have multiplied a bit, so I thought it might be good to have a bit of a written-down plan to get us ready for Gerrit 3.7. I think we should probably start with a manual update to All-Projects, as we don't really manage that in Ansible. The two reviews to update the bootstrap documentation should serve as review for the production change: https://review.opendev.org/c/opendev/system-config/+/876236 https://review.opendev.org/c/opendev/system-config/+/876237 I've consolidated those changes an actual diff to our production All-Projects/project.config shown in https://paste.opendev.org/show/bmy36m7TO2b4cpndO5gO/ I have that in a checkout, so if we're happy with that, I can manually push it (or will rework for comments). I think this gives us a good place to start and covers both the copy-conditions and submit-requirements changes for the top-level All-Projects. Updating the copy-conditions for the project ACL's is already reviewed in https://review.opendev.org/c/openstack/project-config/+/867931 I think I'll feel better about that having already done All-Projects and seen that working, but that just needs to be applied. Then we need to move onto submit-requirements for the project ACL's. Firstly https://review.opendev.org/c/openstack/project-config/+/875992/1 is just mechnical to handle the arguments in the normalize tool. The first change https://review.opendev.org/c/openstack/project-config/+/875804/3 updates labels using "function = NoOp" to also have a no-op submit-requirement. We know from testing that labels must still specify a function. Strictly, I don't think the s-r is necessary; but it does seem that we can use this to add a description explaining what the label is doing. I also think this future-proofs us against the removal of label functions (although it is currently unclear how future gerrit will deal with this). The second change is similar https://review.opendev.org/c/openstack/project-config/+/875993/2 except this is a large selection of Backport-Candidate labels using "function = NoBlock", which is alised to NoOp. For consistency, this is all changed to NoOp and, again, given a submit-requirement as well. Following that is https://review.opendev.org/c/openstack/project-config/+/875994/2 which is a mechanical change again for the normalize tool, and then the two final changes https://review.opendev.org/c/openstack/project-config/+/875995 https://review.opendev.org/c/openstack/project-config/+/875996 convert more complex labels that have blocking semantics submit-requirements. I think these just want careful review to ensure I've translated what they're doing correctly. At that point, I think we're fully migrated off the deprecated ACL options, clearing our path for a Gerrit 3.7 upgrade, general notes for which are kept at https://etherpad.opendev.org/p/gerrit-upgrade-3.7 -i

1 1

Team Meeting Agenda for March 7, 2023
by Clark Boylan 07 Mar '23

07 Mar '23

We will meet on March 7, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230307) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230307) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230307) *** https://review.opendev.org/q/topic:gerrit-s-r-3.7 Cleaning up deprecated Gerrit ACL stuff *** We'll defer on java 17 for now as there is no rush and that gives upstream time to fix the issues *** Working with upstream to test ianw's fix for ssh channel tracking. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230307) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-12 are the only four giteas behind the load balancer currently *** gitea01-04 remain in the replication config and can be reenabled in the load balancer via the control socket as they were merely disabled. *** Work to cleanup gitea05-08 is in progress *** Do we need gitea13-16? *** Name servers and etherpad next. **** Etherpad after the PTG? ** AFS volume quotas (clarkb 20230307) *** A number of mirror AFS volumes appear to have steady growth over time. Should we bump their quotas? **** debian-security in particular is right near the limit ** Scheduling project renames (clarkb 20230307) ** Quo vadis storyboard (frickler 20230307) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Re: Intent to grant control of x/virtualpdu to OpenStack community
by Stéphane Robert 02 Mar '23

02 Mar '23

Hi, I just added Riccardo Pittau to the virtualpdu-core and virtualpdu-release groups. I didn't do it sooner because I could not remember my INAP credentials and could not use the "Forgot Password" functionality... Finally, after trying several combinations I got through. Hope this helps. If there is anything else I can do, feel free to reach out. I'll try my best. Stephane Le mer. 1 mars 2023, à 15 h 50, Jeremy Stanley <fungi(a)yuggoth.org> a écrit : > The OpenStack Ironic project relies on VirtualPDU, which is no > longer actively developed. Ironic contributors reached out to the > VirtualPDU maintainers with an offer to officially assume > maintenance responsibilities in order to avoid forking it, and > received a response from one (Mathieu Mitchell) who indicated > support for that plan. Unfortunately, communication died off shortly > thereafter, and the Ironic team has been unable to raise any of the > current maintainers to actually add their access to the x/virtualpdu > project in OpenDev's Gerrit. > > We don't have a documented process since this is the first time it's > really come up, but I'm officially announcing that I intend to use > my administrative permissions as an OpenDev sysadmin to add > membership of an OpenStack Ironic team representative to the > following Gerrit groups if no objections are raised before > Wednesday, March 8: > > * virtualpdu-core > * virtualpdu-release > > This will effectively grant full control of the x/virtualpdu > repository to OpenStack Ironic contributors. Please follow up to > the service-discuss(a)lists.opendev.org or > openstack-discuss(a)lists.openstack.org mailing list with any concerns > as soon as possible, or you can feel free to reach out to me > directly by email or in IRC (fungi in the #opendev channel on the > OFTC network). > -- > Jeremy Stanley >

2 1

Team Meeting Agenda for February 28, 2023
by Clark Boylan 28 Feb '23

28 Feb '23

We will meet on February 28, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** clarkb was the only nominee in the Service Coordinator election. Due to lack of alternative options there will be no polls, and clarkb wins by default. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230214) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230214) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230214) *** https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs *** https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17 *** https://github.com/apache/mina-sshd/issues/319 Gerrit SSH issues with flaky networks. *** Clarkb stuck the last two issues on the Gerrit Community Meeting agenda and plans to attend March 2 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230214) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09 has been launched **** Ended up using a modern vexxhost v3 flavor with 32GB of memory and 8vcpus. No BFV. Can probably replace 8 old servers with ~4 new servers of this size. **** Ansible deployed a gitea on it **** clarkb manually copied the database over from gitea01 and applied it to gitea09 for redirects **** clarkb also copied logo files for unused org names to keep their logos showing up **** Gerrit is currently (should be done by the meeting) syncing git repos to gitea09 **** https://review.opendev.org/c/opendev/system-config/+/874340/ and child will make it easier to add new giteas and remove old ones. ** Quo vadis storyboard (frickler 20230214) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** Handing over x/virtualpdu to OpenStack Ironic PTL (fungi 20230228) *** The Ironic team has received agreement from a current maintainer, but since ghosted; proposing a communication plan to move forward. * Open discussion

1 0

Zuul issues in Publishing StarlingX docs
by Balaraj, Juanita 23 Feb '23

23 Feb '23

Hi, We are currently experiencing some Zuul issues in publishing StarlingX 8.0 docs on https://docs.starlingx.io/. Any help to solve this issue is appreciated. Thank you, Juanita

2 7

Self Nomination for OpenDev Service Coordinator
by Clark Boylan 15 Feb '23

15 Feb '23

I'm probably beginning to sound like an old record now. I'm nominating myself to be OpenDev Service Coordinator for the next six months. I intend to continue to move towards a sustainable base set of services for OpenDev. Updating configuration management where necessary, trimming services that don't quite fit anymore, and taking advantage of our CI system to build more trust in our deployments. Basically the things we've been working on for a few years now. I am also still happy for someone else to take on the role if they are interested, but the lack of nominations tells me I'm likely it again. Help is always appreciated whatever form it takes. Happy to work with others interested in other forms of contributions as well. Clark

1 0

Team Meeting Agenda for February 14, 2023
by Clark Boylan 14 Feb '23

14 Feb '23

We will meet on February 14, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Service Coordinator nominations now through February 14, 2023 https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** No Meeting February 21, 2023 due to lack of participants. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230214) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230214) *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/873337 Fix warnings about missing migrations *** Need to change site domains ** Gerrit Updates (ianw 20230214) *** https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs *** https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17 *** https://github.com/apache/mina-sshd/issues/319 Gerrit SSH issues with flaky networks. *** Clarkb stuck the last two issues on the Gerrit Community Meeting agenda and plans to attend March 2 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230214) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Clarkb has begun thinking about gitea backend server upgrades. **** Are we happy with the current flavor size? Do we want to continue doing BFV? **** Need to copy the database from an existing server or programmatically add redirect entries to the repo_redirect table in Gitea on new servers. ** Quo vadis storyboard (frickler 20230214) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Updating copyConditions for Gerrit 3.7
by Ian Wienand 08 Feb '23

08 Feb '23

Hello, Change [1] modifies our Gerrit ACL files to move from the deprecated copy conditionals flags. In simpler terms, these are the flags that set rules for labels about "+2/-2 votes aren't removed when new changesets are updated" or "keep votes when a trivial rebase is made". Projects that have made custom labels (things like "Review-Priority", "Backport-Candidate", etc.) will have specified these as part of the label configuration. There should be no change to the rules applied to labels from this transition; the old rules have been translated into the new format. However, a double check is always welcome; if you project uses custom labels you might like to quickly look at [1]. We plan to merge this change in a few days. In going through all projects, I did notice that projects have perhaps copy-pasted the label/rules and possibly they are not always working as intended. If you use custom labels, it might be an idea to just again double-check the rules are what you want for your project. The new format is quite simple; described at [2] -- basically if the query matches on the new changeset, using standard Gerrit searching rules, then the votes will be copied forward to the new changeset. We are aware that Gerrit 3.6 changed the UI for showing custom labels, and the custom labels are less visible by default. We have raised some of the issues upstream [3]. So far there hasn't been a lot of community engagement with this, but we will keep the discussion going. It is good to remember that Gerrit does have a quite powerful ability to make customised dashboards, and we have gerrit-dash-creator (see samples at [4]) to help as well. Thanks, -i [1] https://review.opendev.org/c/openstack/project-config/+/867931 [2] https://gerrit-review.googlesource.com/Documentation/config-labels.html#lab… [3] https://groups.google.com/g/repo-discuss/c/gYVD-iBR9Ow [4] https://gerrit-dash-creator.readthedocs.io/en/latest/dashboards/index.html

1 0

Team Meeting Agenda for February 7, 2023
by Clark Boylan 07 Feb '23

07 Feb '23

We will meet with this agenda on February 7, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service Coordinator nominations now through February 14, 2023 https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230207) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups *** Need to shutdown old bridge. ** Mailman 3 (clarkb 20230207) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Git updates on docker images (clarkb 20230207) *** https://review.opendev.org/q/topic:flip-to-distro-git+status:open *** Base python images have not updated yet. We should rebuild our opendev base images when they do. ** New Debuntu preventing sudo pip installs (clarkb 20230207) *** Bookworm and 24.04 will prevent sudo pip installs. We should start figuring out how big of an impact this will have on us. ** Gerrit Updates (ianw 20230207) *** Missed openssh-client in the base image conversion and broke jeepyb manage-projects *** https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs *** https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17 *** https://github.com/apache/mina-sshd/issues/319 Gerrit SSH issues with flaky networks. ** SqlAlchemy 2.0 (clarkb 20230207) *** Released last week and is not compatible without some forklifting *** Zuul forklifting in progress *** Lodgeit reported to be broken *** Anything else relying on sqlalchemy? Refstack? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230207) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Quo vadis storyboard (frickler 20230207) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** FIPS jobs *** Latest patches: **** https://review.opendev.org/c/openstack/project-config/+/872222 **** https://review.opendev.org/c/zuul/zuul-jobs/+/866881 **** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/872223 (cannot merge before project-config change) **** Summary of structure: https://paste.openstack.org/show/blE6XM5uo2Gy9ibMJICb/ * Open discussion

1 0

OpenDev's Next Service Coordinator Election
by Clark Boylan 01 Feb '23

01 Feb '23

Hello everyone, Every six months we elect a Service Coordinator. The first step in that process is opening up a nomination period for individuals to nominate themselves for the position. The last nomination period ran from August 2, 2022 to August 16, 2022. Counting forward from that time period January 31, 2023 to February 14, 2023 is approximately six months after the previous nomination period. We'll go ahead and run our next nomination period from January 31, 2023 to February 14, 2023. If we end up with more than one nominee we can hold an election that runs for an additional two weeks ending February 28, 2023. Normally I would suggest that we only run the election for a week, but I am traveling the week of the 21st and would prefer to avoid conflicts. Historically, we haven't had more than one nominee so I don't expect this to be a problem, but if it does happen we have a plan. Clark

1 1

Team Meeting Agenda for January 31, 2023
by Clark Boylan 31 Jan '23

31 Jan '23

Hello, We will meet on January 31, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Service Coordinator nominations begin January 31, 2023 https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230131) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups *** Need to shutdown old bridge. ** Mailman 3 (clarkb 20230131) *** https://etherpad.opendev.org/p/mm3migration *** Need to restart to set Site Owner *** Need to change site domains *** Need to fix root alias ** Git updates on docker images (clarkb 20230131) *** https://review.opendev.org/q/topic:flip-to-distro-git+status:open *** Base python images have not updated yet. We should rebuild our opendev base images when they do. ** Gerrit Updates (ianw 20230131) *** https://review.opendev.org/c/opendev/system-config/+/870114 Add Gerrit 3.6 -> 3.7 Upgrade test job *** https://review.opendev.org/c/opendev/system-config/+/872238 Fixup Gerrit Submit Requirements usage *** https://review.opendev.org/c/opendev/system-config/+/870874 Convert Gerrit to run on our base python image *** https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17 *** https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs ** Linaro Cleanup/Move (clarkb 20230131) *** New cloud is in use. *** How hands on do we want to be with the deployment? ** SqlAlchemy 2.0 (clarkb 20230131) *** Released last week and is not compatible without some forklifting *** Zuul forklifting in progress *** Lodgeit reported to be broken *** Anything else relying on sqlalchemy? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230131) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Quo vadis storyboard (frickler 20230131) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Team Meeting Agenda for January 24, 2023
by Clark Boylan 24 Jan '23

24 Jan '23

We will meet on January 24, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Service Coordinator nominations begin January 31, 2023 https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230124) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups *** Need to shutdown old bridge. ** Mailman 3 (clarkb 20230124) *** https://etherpad.opendev.org/p/mm3migration *** Need to restart to set Site Owner *** Need to change site domains *** Need to fix root alias ** Gerrit Updates (ianw 20230124) *** https://review.opendev.org/c/opendev/system-config/+/870114 Add Gerrit 3.6 -> 3.7 Upgrade test job *** https://review.opendev.org/c/opendev/system-config/+/870874 Convert Gerrit to run on our base python image *** https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17 *** https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs ** Pruning backups on server (clarkb 20230124) *** The backup server in rax is at 92% capacity and needs pruning ** Linaro Cleanup/Move (clarkb 20230124) *** https://review.opendev.org/c/openstack/project-config/+/871196 Remove old linaro cloud from Nodepool *** Update for new cloud status. ** Gitea 1.18 upgrade (clarkb 20230124) *** https://review.opendev.org/c/opendev/system-config/+/870851 Upgrade to 1.18.3 *** There is a held node that reviewers can use to double check things. *** Gitea provides cross repo searching now too. Should we consider dropping Hound? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230124) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Quo vadis storyboard (frickler 20230124) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Team Meeting Agenda for January 17, 2023
by Clark Boylan 17 Jan '23

17 Jan '23

Hello, We will meet with this agenda on January 17, 2023 in #opendev-meeting at 19:00 UTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230117) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230117) *** https://etherpad.opendev.org/p/mm3migration *** Need to restart to set Site Owner *** Need to change site domains ** Quo vadis storyboard (frickler 20230117) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** Gerrit 3.6 (ianw 20230117) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.6 *** https://review.opendev.org/c/opendev/system-config/+/869763 beginning of stack to remove 3.5 images and add 3.7 and associated testing. ** Service Coordinator Election *** Nomination Period January 31, 2023 - Feburary 14, 2023 (Roughly 6 months after the previous nomination period). ** Linaro Cleanup/Move (clarkb 20230117) *** New building is running in OSUOSL *** Getting jobs running in new cloud then can shutdown old cloud. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230117) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Nova server rescue behavior in vexxhost (clarkb 20230117) *** Clarkb needs to test this using info provided by jrosser **** https://paste.opendev.org/show/bxxFEEUWeUrkIVlBSGrw/ * Open discussion

1 0

Container registry at OpenDev
by Radosław Piliszek 10 Jan '23

10 Jan '23

Hi OpenDevers, Are there any plans to have a container registry at OpenDev? Is there any thread that could be followed? Kind regards, Radek -yoctozepto

4 7

Team Meeting Agenda for January 10, 2023
by Clark Boylan 10 Jan '23

10 Jan '23

We will meet on January 10, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230110) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230110) *** https://etherpad.opendev.org/p/mm3migration *** Need to restart to set Site Owner *** Need to change site domains ** Quo vadis storyboard (frickler 20230110) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** Gerrit 3.6 (ianw 20230110) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.6 *** Need to cleanup Gerrit 3.5 image and add a 3.7 image *** Update testing to test 3.6 to 3.7 upgrade. ** Nox (clarkb 20230110) *** Nox is a tox alternative. *** Uses simple directives and standard tools like pip and virtualenv to do installations *** Clarkb would like to convert OpenDev repos to Nox to avoid Tox v4 migration issues and problems. ** Service Coordinator Election *** Nomination Period January 31, 2023 - Feburary 14, 2023 (Roughly 6 months after the previous nomination period). ** Linaro Cleanup/Move (clarkb 20230110) *** We need to migrate the nodepool builder node. Less disk in new linaro cloud makes this tricky. We have asked if we can host this in OSUOSL. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230110) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Nova server rescue behavior in vexxhost (clarkb 20230110) *** Clarkb needs to test this using info provided by jrosser **** https://paste.opendev.org/show/bxxFEEUWeUrkIVlBSGrw/ * Open discussion

1 0

OpenDev's Zuul and an open-source project in GitHub
by Radosław Piliszek 05 Jan '23

05 Jan '23

Dear OpenDev-knowledgeable folks, First of all, Happy New Year! I hope you have a healthy and very satisfying 2023. Now onto the business. I have a cloud-related open source project that will be starting the upcoming months and we are looking for hosting. We will most likely decide to host our project on GitHub due to the popularity factor (both for consumers and developers; I am aware the mirroring would help for consumers but we prefer fewer bumps for contributors) [as a side note, this has not been decided yet and OpenDev+Gerrit is still in discussions; and yes, it hurts me that OpenDev and Gerrit are not nearly as popular as GitHub]. The question is whether we could utilise (with all possible acknowledgments) the Zuul deployment at OpenDev. If it is a technical matter, we will have some effort to help with those so it might be amendable. If it is the trust matter, then we will agree to trust the OpenDev<->GitHub boundary. However I am mostly not sure about your policy with this and I have not found a reference to this model of OpenDev usage. Kind regards, Radek -yoctozepto

2 2

2022 OpenDev Foundation Annual Report
by Clark Boylan 28 Dec '22

28 Dec '22

Hello everyone, I've been asked to help put together content for the 2022 Foundation Annual Report covering the work we've done with OpenDev. I've started a draft here: https://etherpad.opendev.org/p/2022-opendev-annual-report. Please feel free to add feedback, ideas, or even write some content yourselves. I've attempted to capture the years work looking back in git logs and my local notes, but a year is a long time and it wouldn't surprise me to find I've forgotten something note worthy. Also, please check what I have written for accuracy. Clark

1 0

OpenDev Holiday Party
by Clark Boylan 15 Dec '22

15 Dec '22

Hello everyone, We've accomplished much this year and now it is time for some fun. Instead of having our normal meeting next week on December 20th we'll play some board games via Board Game Arena. Join us on December 20th at 19:00 UTC in https://meetpad.opendev.org/opendev-holiday-party You will need a free Board Game Arena account to play. Sign up here: https://en.boardgamearena.com Most games do require a premium account to host the game; I'll upgrade my account to ensure we've got that covered. We'll use the Meetpad call to coordinate games and take it from there. It is possible that we might end up with too many players for a single game. If that happens we can take turns, we can run non premium games, or maybe more than one person will have a premium account and can host. In any case I expect we can work through it on the call. See you there! Clark

1 0

Team Meeting Agenda for December 13, 2022
by Clark Boylan 13 Dec '22

13 Dec '22

We will meet with this agenda on December 13, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221213) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20221213) *** https://etherpad.opendev.org/p/mm3migration *** We have seen a few small artifacts in the web UI around the vhosting. lists.opendev.org shows up in at least one place where it shouldn't *** Is site owner properly set yet? ** Quo vadis storyboard (frickler 20221213) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Followup sent to hash out options for OpenDev. Saw one response offering to help with the software, but unfortunately we need to start with the deployment. ** Gerrit 3.6 (ianw 20221213) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.6 *** Upgrade has been performed. ** Tox v4 (clarkb 20221213) *** Tox v4 recently released and is not compatible with many of our hosted repos. *** zuul-jobs pinned tox in ensure-tox to <4 with a plan to uncap on the 22nd of December *** We should proactively write changes to address the compatibility issues in preparation. **** https://review.opendev.org/c/opendev/system-config/+/867269 ** Winding down iweb nodepool provider (clarkb 20221213) *** We need to shut this down by the end of the year, but many of us will be out at the very end of the year so we should start nowish. *** https://review.opendev.org/q/topic:wind-down-iweb ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221213) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Nova server rescue behavior in vexxhost (clarkb 20221129) *** Clarkb needs to test this using info provided by jrosser **** https://paste.opendev.org/show/bxxFEEUWeUrkIVlBSGrw/ ** Holiday Party December 20th? (clarkb 20221213) *** Instead of a normal meeting why don't we have some fun *** We can use meetpad and run a game or three on Game Board Arena if people are interested. Only one person needs a premium account to host everyone else and clarkb can sign up for one. *** Open to other ideas too * Open discussion

1 0

Team meeting agenda for December 6, 2022
by Clark Boylan 06 Dec '22

06 Dec '22

We will meet on December 6, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221206) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv *** https://review.opendev.org/q/topic:bridge-backups ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221129) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221129) *** https://etherpad.opendev.org/p/mm3migration *** lists.opendev.org and lists.zuul-ci.org have migrated. ** Quo vadis storyboard (frickler 20221129) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Followup sent to hash out options for OpenDev. Saw one response offering to help with the software, but unfortunately we need to start with the deployment. ** Nova server rescue behavior in vexxhost (clarkb 20221129) *** Clarkb needs to test this using info provided by jrosser **** https://paste.opendev.org/show/bxxFEEUWeUrkIVlBSGrw/ ** Gerrit 3.6 (ianw 20221206) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.6 *** todo held node testing *** update time/date proposals * Open discussion

1 0

Storyboard Needs and Alternatives
by Clark Boylan 30 Nov '22

30 Nov '22

Hello, There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. The Problem(s) * Storyboard is slow. * Storyboard's deployment is aging and needs to be updated. * Storyboard is largely unmaintained at this point. * Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be deployed as we need to update the deployment platform to a newer version of python. Options * Update the Storyboard deployment and see if that improves things enough to make people happy. * Migrate projects onto one or more different issue trackers. The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion on what would be necessary to support projects that don't want to use Storyboard any longer. Clark

6 9

Team Meeting Agenda for November 29, 2022
by Clark Boylan 29 Nov '22

29 Nov '22

Hello, We will meet November 29, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221129) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221129) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221129) *** https://etherpad.opendev.org/p/mm3migration *** Ready to add a new jammy node to our inventory and let Ansible do its thing. **** https://review.opendev.org/c/opendev/system-config/+/865361 ** Quo vadis storyboard (frickler 20221129) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Followup sent to hash out options for OpenDev. ** Nova server rescue behavior in vexxhost (clarkb 20221129) *** Should we consider building our own rescue image for non BFV instances? ** Gerrit 3.6 (ianw 20221129) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.6 ** acme.sh failures (clarkb 20221129) *** Recently we've had a number of certs fail to renew and our checker is complaining about them. *** acme.sh did do a recent release and has since updated their default from rsa to ecdsa. * Open discussion

1 0

Team Meeting Agenda for November 22, 2022
by Clark Boylan 22 Nov '22

22 Nov '22

We will meet with this agenda on November 22, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221115) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221115) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Testing updated images with updated packages is underway. ** Updating base python docker images to use `pip wheel` (clarkb 20221115) *** The change to modify this has landed and new images have been promoted. We don't expect problems but please call them out if seen. ** Quo vadis storyboard (frickler 20221115) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** It has been two weeks since we asked for additional feedback and haven't received any. We should begin making plans with the feedback that we were given. ** Nova server rescue behavior in vexxhost (clarkb 20221115) *** Should we consider building our own rescue image for non BFV instances? * Open discussion

1 0

Team Meeting Agenda for November 15, 2022
by Clark Boylan 15 Nov '22

15 Nov '22

We will meet with this agenda on Tuesday, November 15, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221115) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221115) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Currently debugging why updated images aren't working as expected. Appears to be some problem with django. ** Updating base python docker images to use `pip wheel` (clarkb 20221115) *** Will aim to land this change soon, then double check images built with new assemble process are happy. ** Etherpad docker container logs growth (clarkb 20221115) *** https://review.opendev.org/c/opendev/system-config/+/864060 ** Quo vadis storyboard (frickler 20221115) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** No new movement on the mailing list. Maybe give it another week then we can consider feedback collected? ** Nova server rescue behavior in vexxhost (clarkb 20221115) *** There are image settings that need to be set for rescue images. A dedicated rescue image would also be able to set a non colliding disk label. ** Goodbye birdsite, hello tooters? (frickler 20221115) *** Twitter seems to be on a downward spiral and we should consider stopping using it *** OIF already has an account on fosstodon.org, also ianw and frickler *** Do we want to move the opendevinfra acc there, too? *** How much work is needed in terms of tooling? Any volunteers? * Open discussion

1 0

Team Meeting Agenda for November 8, 2022
by Clark Boylan 08 Nov '22

08 Nov '22

Hello, We will meet with this agenda on November 8, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221108) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv *** https://review.opendev.org/c/opendev/system-config/+/863564 *** https://review.opendev.org/c/opendev/system-config/+/863568 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221101) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Updating base python docker images to use `pip wheel` (clarkb 20221108) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Etherpad docker container logs growth (clarkb 20221101) *** Need change to log to syslog instead similar to other services. ** Quo vadis storyboard (frickler 20221107) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Many people interested in moving away from storyboard *** No volunteers to update our installation *** How long can we/do we want to commit to running an outdated installation on an EOL OS? ** Nova server rescue behavior in vexxhost (clarkb 20221108) *** Clarkb has tested this and the behaviors are surprising *** Rescuing a normal disk instance results in the instance boot with the rescue image kernel and the rescued instance's root device mounted on / instead of the rescue image mounted to /. *** Rescuing a BFV instance fails. First with the default microversion because old Nova API doesn't support this. When using microversion 2.88 it also fails, but only after attempting to rescue. **** The instance goes into an error state due to "cannot be rescued: Driver Error: Cannot access storage file" **** Attempting to unrescue the instance also fails because you cannot unrescue an instance that is in an error state * Open discussion

1 0

Team Meeting Agenda for November 1, 2022
by Clark Boylan 01 Nov '22

01 Nov '22

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221101) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221101) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/862835/ Disable phased package updates ** Removing snapd (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/862834 ** Mailman 3 (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Updating base python docker images to use `pip wheel` (clarkb 20221101) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Etherpad docker container logs growth (clarkb 20221101) *** Need change to log to syslog instead similar to other services. * Open discussion

1 0

RE: Storyboard Needs and Alternatives
by Michael Johnson 26 Oct '22

26 Oct '22

Sorry this post may be out of the existing thread, I wasn't aware of this list (just the service-announce) until now, so I just signed up. Here are my issues with Storyboard: 1. It is unmaintained and even contributed patches get ignored (6+ months with no review). 2. It is slow (frankly it seems like it is getting slower over time). 3. The search is horrible. 4. It is not flexible enough to work with teams processes. a. For example, we can't link duplicate stories, so it's a pain to piece all of the stories together. b. There is no lifecycle status that can be used for search, etc. Triaged, confirmed, etc. c. Object storage never became a thing, so no screenshots, no logs, etc. d. There are no "My bugs" quick links, etc. so stuck with the search capability. e. Tags are free-form, so you get "RFE" "rfe" "feature", etc. that are all different searches. f. No way to search by "authored by" g. No "importance" fields, again making search and sorting hard without creating ordered lists. 5. Not all projects are on Storyboard, so it is a huge pain to move a story from one project to the other. 6. The Storyboard community was very opinionated on "how" it should be used and was not open to enabling alternate workflows/processes. a. This causes problems when community members have downstream tools/processes they need to align stories with. b. This made maintaining stories/bugs very difficult and people stopped triaging stories. These are the top of head issues I have with Storyboard. I know from the Octavia team perspective, we have tried[1] to work with and help Storyboard become a thing, but it just isn't happening. At this point I think the Octavia team is favoring going back to launchpad. I think given how Storyboard has gone, I would not be in favor of an alternate tool unless the community was 100% onboard to all move inside a cycle. Michael [1] https://etherpad.opendev.org/p/storyboard-issues

1 0

Team Meeting Agenda for October 25, 2022
by Clark Boylan 25 Oct '22

25 Oct '22

We will meet with this agenda on October 25, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221025) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** Are console log files no longer being written? Did base-jobs playbook get updated too? *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221025) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** gitea-lb02 was launched using a Jammy image and is now in prod **** Used upstream image published by Ubuntu uploaded to vexxhost and converted to raw. This allows for max compatibility with bfv on vexxhost (gitea-lb02 is not bfv). **** Needed to use an up to date paramiko out of a virtualenv to run launch node as Jammy doesn't do SSH RSA + SHA1. ** Removing snapd (clarkb 20221025) *** In the past we removed snapd as we didn't use it for anything. *** Stopped removing snapd in order to install kubectl. *** Should we go back to removing snapd globally and only add it back in where needed? ** Mailman 3 (clarkb 20221025) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Switching the default base job's nodeset to Ubuntu Jammy on October 25 (day of the meeting). ** Updating base python docker images to use `pip wheel` (clarkb 20221025) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Dropping python 3.8 base images (clarkb 20221025) *** To make room for python 3.11 it would be good to drop our python3.8 images. *** https://review.opendev.org/q/status:open+(topic:use-new-python+OR+topic:doc… ** iweb cloud going away at the end of the year (clarkb 20221025) *** We've been told this will need to be shutdown by the end of the year. We are good to use it until then. *** Possibility for using cloudstack resources as backfill, but would require a nodepool driver. ** Etherpad docker container logs growth (clarkb 20221025) *** The etherpad docker container's logs grow over time when we don't update the container. *** Can/should we add the syslogging for containers to this service? * Open discussion

1 0

Cancelling the Team Meeting on October 18, 2022
by Clark Boylan 18 Oct '22

18 Oct '22

Hello, We've decided to cancel the team meeting tomorrow (October 18, 2022) as a number of us are participating in the PTG and that is consuming a fair bit of time. Feel free to bring up items of concern either here on the mailing list or in IRC in #opendev on OFTC. We'll meet at our usual time and location on October 25, 2022. Clark

1 0

OpenSSH 8.8, RSA keys, and Gerrit
by Clark Boylan 14 Oct '22

14 Oct '22

Hello, About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example). There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough. On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms. [0] https://issues.apache.org/jira/browse/SSHD-1141 [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation. Clark

1 1

Testing OpenStackSDK/OSC on real clouds
by Artem Goncharov 13 Oct '22

13 Oct '22

Hi all, Last years of my work on OpenStackSDK and CLI on one hand and being representative of a public cloud on the other hand made clear to me that not a single OpenStack cloud behaves like another one, doesn’t matter what you do and how hard you try. And sadly Interop is not helping here at all. The API calls you use to talk with one public cloud are guaranteed to work slightly differently on another one. This is clearly visible through the Storyboard of SDK, IRC and other offline discussions. Lot of customers are capable to work without issues with SDK/CLI on their cloud, while others report the same things are not working for them. This is a relatively complex discussion and Clark reserved a slot in the TC meeting during PTG https://etherpad.opendev.org/p/tc-2023-1-ptg#L38 <https://etherpad.opendev.org/p/tc-2023-1-ptg#L38> to describe his stake in the problem. For a quite some time I am trying to follow an approach of having possibility to run functional tests of both SDK and CLI on real clouds and not only on devstack (on devstack things are working fine but it means as much as nothing at all in real life). On multiple occurrences I was asking for credentials and I got at the moment credentials for 2 public clouds. This already helped both to find issues in SDK (i.e. RADOSGW != Swift) as well as to point clouds on real issues on their side. It should be actually in the interest of those clouds to have some useful checks that give guarantee to their users SDK and CLI (ideally also Ansible/TF) are working fine. This is even a better form of interoperability check than anything else what we have now. Since people in this ML are involved in maintaining resource donations to OpenDev I would like to ask you to consider allocating cloud credentials for the SDK/CLI team to ensure things are running smoothly for your users (pls do not mix with allocating real resources - we need only cloud credentials to test APIs and there are no static resources required). At the moment I maintain those credentials manually until we finally get post-review pipeline created in OpenStack tenant to fully automate those tests. If you have questions or suggestions please feel free to post here, join SDK/CLI team during PTG on Friday 27.10 13:00-15:00 or contact me directly for clarifications. Thanks a lot for support, Artem

2 1

Team Meeting Agenda for October 11, 2022
by Clark Boylan 11 Oct '22

11 Oct '22

We will meet with this agenda on October 11, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** PTG next week * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221011) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 Disable writing console log files on bridge and static **** Needs a second change pushed to opendev/base-jobs/playbooks/infra-prod/setup-keys.yaml *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221011) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Need to retest launching a Jammy node now that user management is updated. ** Mailman 3 (clarkb 20221011) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Ubuntu 22.04.1 is out for some months now and thus can be considered stable, no need to wait a full year like we did with 20.04 *** Patch to switch base-test job: https://review.opendev.org/c/opendev/base-jobs/+/860686/2 *** Plan for further steps? Timetable, which repos to test, announcements *** Devstack currently inherits the base nodeset (via multinode) without pinning * Open discussion

1 0

Team Meeting Agenda for October 4, 2022
by Clark Boylan 04 Oct '22

04 Oct '22

We will meet on October 4, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221004) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221004) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/860112 Update to launch node to handle jammy hosts ** Mailman 3 (clarkb 20221004) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Gitea connectivity issues (clarkb 20221004) *** Users in certain geographies have reported trouble with Gitea access *** Gitea changed http libraries which broke our ability to trace requests from haproxy -> apache -> gitea **** Partially worked around that by removing X-Forwarded-For headers. * Open discussion

1 0

Team Meeting Agenda for September 27, 2022
by Clark Boylan 27 Sep '22

27 Sep '22

We will meet with this agenda on September 27, 2022 at 19:00 UTC in #opendev-meeting. This agenda is a slightly edited version compared to the wiki as I anticipate we'll have less participation this week. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Mailman 3 (clarkb 20220927) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220927) *** Change has landed. ** Nodepool Builder Disk utilization (clarkb 20220927) *** Have we expanded disk space on existing builders yet? * Open discussion

1 0