- service-discuss - lists.openstack.org

Team Meeting Agenda for May 23, 2023
by Clark Boylan 23 May '23

23 May '23

We will meet with this agenda on May 23, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230523) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** Discovered a deficiency in Docker impacting speculative testing of container images **** https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW **** A number of issues have been found when setting up services with podman ***** No reliable source of packages for Ubuntu older than Jammy ***** No syslog support **** Migration to podman still a reasonable goal but a longer term one. In the interim suggestion is we revert quay.io image moves for opendev back to docker hub to keep speculative image testing. Then work towards a future where we use not docker and can switch to not docker.io more easily. ** Bastion host (ianw 20230523) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230523) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230523) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230523) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** Fedora cleanup (clarkb 20230523) *** Removal of Fedora 37 mirror impacted DIB jobs. DIB can fetch from upstream. *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Proceed with removal as it seems no one is asking for Fedora specifically. ** Quo vadis storyboard (frickler 20230523) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Team Meeting Agenda for May 16, 2023
by Clark Boylan 16 May '23

16 May '23

We will meet with this agenda on May 16, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230516) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** Discovered a deficiency in Docker impacting speculative testing of container images **** https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW **** clarkb's suggestion: we build all container images with buildx (it doesn't have this problem) and use skopeo to grab the correct images: https://review.opendev.org/c/opendev/system-config/+/882977 ** Bastion host (ianw 20230516) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230516) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230516) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230516) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** AFS volume quotas and utilization (clarkb 20230516) *** Utilization climbing again. *** https://review.opendev.org/c/opendev/system-config/+/883083 Cleanup unused fedora mirror content *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Quo vadis storyboard (frickler 20230516) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Wiki cert (clarkb 20230516) *** clarkb will try to get this done in a few days once there is about a week of validity remaining. * Open discussion

1 0

Future of Fedora and CentOS Stream Test Images
by Clark Boylan 10 May '23

10 May '23

Hello everyone, Over the last couple of years we've grown support for a number of RHEL like or RHEL adjacent distros in our CI system. We have added OpenEuler, Rocky Linux, and CentOS Stream (it replaced CentOS 8). At the same time we've fallen behind producing and updating images for Fedora. Currently we have disk images and test nodes for Fedora 35 and 36. We mirror packages for Fedora 36 and 37. In the past our intention was always to primarily have a single up to date Fedora image and mirror setup for our CI system which would be Fedora 38 as of April 18, 2023. One option we have is to drop Fedora 35 and 37 from Nodepool and our mirrors, add Fedora 38, and then drop Fedora 36. This would basically maintain the status quo of the last few years. That said I think this is probably a good opportunity to rethink our approach, and we are looking for your feedback. Currently the RHEL like distros are overlapping with each other a bit and for this reason we added Rocky Linux without adding extra mirroring for it. So far this seems to be stable enough as we don't run significant numbers of jobs on Rocky Linux. I think one option available to us would be to drop Fedora mirroring entirely, then add Fedora 38 images/nodes, and then drop Fedora 36 images/nodes. One upside to this approach is it will free up disk space on our OpenAFS fileservers which are under a space crunch currently. I also wonder if we can drop Fedora entirely. It seems like CentOS Stream now occupies a similar role within our CI system. It gives us forward looking updates that will eventually be in RHEL. This allows us to test against a future state in preparation for RHEL changes. The upside to this is that Fedora's support lifetime is about 13 months but CentOS Stream releases appear to have several years of support. This should reduce the amount of mirror and image work we need to perform. Are there specific use cases that Fedora serves that CentOS Stream does not within our CI system? If so are they important enough to continue to try and run on the Fedora release treadmill? If we do want to try and keep up with Fedora do we have any concerns dropping our local mirroring of Fedora packages? Please let us know, Clark

7 14

Team Meeting Agenda for May 9, 2023
by Clark Boylan 09 May '23

09 May '23

We will meet with this agenda on May 9, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230509) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** About half of the images that need moving have changes pushed and are largely ready to merge. *** Will use the halfway point to pause and make sure everything is going well and we aren't missing anything important. ** Bastion host (ianw 20230509) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230509) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230509) *** https://review.opendev.org/c/openstack/project-config/+/882075 list little bit of ACL cleanup work *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** We should restart on updated image to pick up container move as well as theming changes. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230509) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** AFS volume quotas and utilization (clarkb 20230509) *** Utilization is slightly down compared to last week. *** Proposed the removal of Fedora mirrors https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Quo vadis storyboard (frickler 20230509) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Re: [dev][infra][qa][tact-sig] Future of Fedora and CentOS Stream Test Images
by Kashyap Chamarthy 05 May '23

05 May '23

On Wed, May 03, 2023 at 03:28:00PM +0000, Jeremy Stanley wrote: > tl;dr is that the OpenDev Collaboratory is looking at potentially > scaling back on Fedora-based test platform support. Several options > are presented in this service-discuss post: > > https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… > > If you have an opinion on using Fedora in our CI jobs, please follow > up to the discussion there. Hi, Jeremy. FWIW, I think it is still valuable to have at least the latest Fedora-only image available. Often times kernel and virtualization fixes are first fixed on Fedora -- many upstream kernel developers happen to use Fedora. Another reason I wanted to mention is the 'virt-preview' repo[1]. But I see that 'virt-preview' is also available for CentOS stream. So that's a less strong reason. That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist. [1] https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/ -- /kashyap

4 6

Team Meeting Agenda for May 2, 2023
by Clark Boylan 02 May '23

02 May '23

We will meet with this agenda on May 2, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230502) *** zookeeper-statsd is now publishing to quay.io showing that this all seems to work for OpenDev *** Need to decide on an order of operations. **** Do we do python-builder/python-base/uwsgi-base first then the consuming images? This is likely less work, but does open us to potential problems if we need quick updates of the base images and rebuilds of everything. However we are effectively rebuilding anything anyway so maybe we do this and do our best to do it in a short timeframe. **** Maybe we can pick a couple of days to sprint this and try to get as much done as possible? **** Probably a good idea to write down a plan and todo list as soon as we have an approach to track progress. Clarkb can do this. **** Don't forget to resync any images from docker hub to quay.io prior to publishing to quay.io if images have updated since the initial sync. **** We may need to manually clean up the old docker hub images on hosts after we move images. Zuul and Nodepool at least appear to need this. **** Do we need to announce this move to service-announce? We may have external image consumers. ** Bastion host (ianw 20230502) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230502) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230502) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** Gerrit 3.8 images exist now and we don't build 3.6. A couple of changes were landed to 3.7 as well to make 3.8 happy should we restart on the newer image? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230502) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver replacement largely complete. **** Do we still have cleanup todo? Removing old nodes from inventory, deleting them from the cloud, AAAA glue records? **** https://etherpad.opendev.org/p/2023-opendev-dns *** Need help with the next servers in the list. ** AFS volume quotas and utilization (clarkb 20230502) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230502) *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.2 ** Quo vadis storyboard (frickler 20230502) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Team meeting agenda for April 25, 2023
by Clark Boylan 25 Apr '23

25 Apr '23

We will meet with this agenda on April 25, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230418) *** Images have been manually copied from docker hub to quay.io. As we update jobs for each image we should manually resync to catch up. *** https://review.opendev.org/c/opendev/system-config/+/881285 WIP change to convert an image to quay.io using new jobs. *** Four images are not being migrated **** opendev/bazel **** opendev/grafana **** opendev/jitsi-meet-prosody **** opendev/jitsi-meet-web ** Bastion host (ianw 20230418) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230418) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230418) *** ACL updates to better match upstream expectations **** https://review.opendev.org/c/openstack/project-config/+/879906 *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230418) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Etherpad has been replaced. Final cleanup of etherpad01 to commence shortly. ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230418) *** Waiting for 1.19.2 to include API fixes. *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.1 ** Quo vadis storyboard (frickler 20230418) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Re: Followup on https://review.opendev.org/c/openstack/cinder/+/853785
by Jean-Sébastien BEVILACQUA 19 Apr '23

19 Apr '23

Hello, I allow myself to contact you following the email below sent to me by Clark. Indeed, I want to add a new volume driver in Cinder [0] and for that I must have a working CI which validates the tempest tests with the Lustre [1] driver. I tried for 2 weeks to set up this CI with Software Factory but I didn't manage to make it work. However, the Lustre FS being 100% open-source, I was advised to use the upstream CI directly. In gerrit I have already created a service user with username lustreci. Could you tell me how to set up and run the tests directly in the upstream CI ? For information, I already have ansible playbooks allowing to set up a Lustre FS on CentOS 8. Thanks in advance, [0] https://review.opendev.org/c/openstack/cinder/+/853785 [1] https://www.lustre.org/ Jean-Sébastien Le 19/04/2023 à 18:01, Clark Boylan a écrit : > Hello, > > I wanted to followup on your questions in > https://review.opendev.org/c/openstack/cinder/+/853785 as well as your > query submitted to https://openinfra.dev/projects/contact/. > > As mentioned by Sean on the Gerrit change the Cinder team requires > working CI for all in tree Cinder drivers. Many of the systems that > Cinder integrates with are proprietary storage systems which > necessitates the use of external (to OpenDev) third party CI as > specialized hardware and licensing requirements don't allow us to run > these upstream. > > To make this happen you need a CI system that is capable of listening > to Gerrit events, triggering builds, and reporting the results back to > Gerrit (for example Zuul/Jenkins/etc). Both OpenDev [0] and the Cinder > team [1] attempt to provide documentation, but this will always be > incomplete as we won't be aware of your local network policies, > hardware peculiarities and so on. The OpenDev team can help with > connection and account details for Gerrit, and the Cinder team should > be able to help with test specific needs (like appropriate logging, > service behavior, etc). I cannot speak to Software Factory as I have > never personally used it. > > The good news is that Lustre is open source software which can be > deployed without proprietary licensing restrictions, and there don't > appear to be specialized hardware needs either. In this case I would > test Lustre + Cinder in the upstream CI system. It looks like Cinder > is already doing this with Ceph [2]. My recommendation would be that > you shift focus from attempting to running a third party CI system to > adding a new job that runs against Cinder changes to test Cinder + > Lustre integration. > > This is something that the Cinder team should be able to help with as > they have a number of Zuul jobs already including the one that tests > against Ceph. The OpenDev team can help with higher level concerns > like Gerrit accounts, general Zuul behaviors/syntax, and CI system > limitations. You can reach out to the OpenDev team either via > service-discuss(a)lists.opendev.org or in #opendev on the OFTC IRC > network (all of this info can be found in the footer of > https://opendev.org) > > [0] > https://docs.opendev.org/opendev/system-config/latest/third_party.html > [1] https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers > [2] > https://opendev.org/openstack/cinder/src/commit/2e8aff660b18d1e8f961d6de979… > > Clark > -- Jean-Sébastien BEVILACQUA Responsable technique OSSA Tél : 07 85 55 35 11 Courriel :jsbevilacqua@linagora.com www.linagora.com

2 1

Team Meeting Agenda for April 18, 2023
by Clark Boylan 18 Apr '23

18 Apr '23

We will meet with this agenda on April 18, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230418) *** Intermediate registry publication process appears to be working. Now we need to move content and update jobs? ** Bastion host (ianw 20230418) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230418) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230418) *** ACL updates to better match upstream expectations **** https://review.opendev.org/c/openstack/project-config/+/880115 **** https://review.opendev.org/c/openstack/project-config/+/879906 *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230418) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Etherpad replacement is in process https://review.opendev.org/q/topic:add-static02 **** https://paste.opendev.org/show/brRuhPssVLSi4UnF5hcN/ Etherpad data migration process expect db dump and restore to take about half an hour each. **** Planning for Wednesday at 22:00 to start the move. ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230418) *** Gitea 1.19.1 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.1 *** Changes to authentication. Do we care? ** Quo vadis storyboard (frickler 20230418) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Stx docs version menu
by Stone, Ronald 11 Apr '23

11 Apr '23

We are having issues with the version/branch menu on the starlingx docs site. From a release branch, e.g docs.starlingx.com/r/stx.6.0 trying to switch to another release branch append the "r/stx.<version>/" to the current URL instead of pointing to https://docs.starlingx.io/r/stx.<version<https://docs.starlingx.io/r/stx.%3cversion>>. i.e. docs.starlingx.com/r/stx.6.0/r/stx.7.0 is constructed instead of the correct docs.starlingx.com/r/stx.7.0. The settings in the corresponding Sphinx conf.py file are: starlingxdocs_plus_this_version = "r/stx.6.0" starlingxdocs_plus_other_versions = [("Version 7.0","r/stx.7.0"),("Version 8.0","r/stx.8.0"),("Latest","/")] Can we confirm if these conf.py settings are correct? If they are, then we will open a bug against the template. Thanks

2 2