- service-discuss - lists.openstack.org

Updates to meetpad's Jitsi Meet configuration
by Jeremy Stanley 19 Sep '22

19 Sep '22

Last week, we refreshed our configuration defaults for the Jitsi Meet services which provide the teleconferencing features of our meetpad.opendev.org service. The primary behavior change you're likely to notice is that meeting room URLs first take you to a pre-join page now. This is a workaround for some browsers, particularly Firefox, treating the audio stream as unsolicited and automatically blocking it. Previously, that was responsible for causing confusing "I can't hear anyone else" problems for many users. If you had trouble with sound on our meetpad service before, please do try it again at your convenience and let us know if it helped (or more importantly, if it didn't). -- Jeremy Stanley

1 1

Team Meeting Agenda for September 13, 2022
by Clark Boylan 13 Sep '22

13 Sep '22

Hello, We will meet with this agenda on September 13, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220913) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/zuul/zuul/+/855309/ *** https://review.opendev.org/c/opendev/system-config/+/855472 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220913) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Bulk migration identified a few things that have since been updated. Need another pass to check. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220913) *** https://review.opendev.org/c/opendev/system-config/+/855983 Adds ansible and docker deployment for Jaeger ** Fedora 36 Rollout (clarkb 20220913) *** https://review.opendev.org/c/zuul/nodepool/+/853914 Remove fedora 35 testing from nodepool ** Jitsi Meet Updates for Meetpad (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856553 Update to use colibri websockets and scale out JVBs *** Not sure if the java keystore in use by the JVBs need to share a CA or otherwise be trusted by something else. ** Stability of Zuul reboot playbook (clarkb 20220913) *** New instability identified caused by race between container shutdown and docker exec for graceful command *** https://review.opendev.org/c/opendev/system-config/+/857209 Fix for this race ** Python container image updates (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856537 *** Updates to latest python point releases. *** This pulls in newer glibc allowing us to remove the backport in Zuul images ** Improving Ansible task runtime (clarkb 20220913) *** Ansible tasks can be slow on some test nodes. *** Particularly problematic when loops are used that serially process a large set of items *** https://review.opendev.org/c/zuul/zuul-jobs/+/857228 *** https://review.opendev.org/c/opendev/system-config/+/857232 *** Be on the lookout for problematic tasks and updates to speed them up. * Open discussion

1 0

Team Meeting Agenda for September 6, 2022
by Clark Boylan 06 Sep '22

06 Sep '22

Hello, This agenda is getting sent out a bit late due to the holiday yesterday. We will meet at 19:00 UTC in #opendev-meeting with this agenda today, September 6, 2022: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220906) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220906) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220906) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** Migration testing looking good. Need to test updated database settings for large attachments (including that db backups function) and redirects for old pipermail archives. ** Jaeger tracing server (for Zuul) (corvus 20220906) *** Any changes need review yet? ** Fedora 36 Rollout (clarkb 20220906) *** Status Update ** Jitsi Meet Updates for Meetpad (clarkb 20220906) *** Configs updated to start with landing page for meetings. Should fix Firefox auto mute of auto playing audio *** Discovered that JVBs rely on newer colibri websocket system for scaling out. Our extra JVBs need to be updated to function under this system. ** Stability of Zuul reboot playbook (clarkb 20220906) *** Crashes if unattended-upgrades is running due to conflict over apt lock. *** Potentially crashes if services were already stopped outside of the playbook run. * Open discussion

1 0

Team Meeting Agenda for August 30, 2022
by Clark Boylan 30 Aug '22

30 Aug '22

We will meet with this agenda on August 30, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack feature freeze this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220830) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220830) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220830) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** Working through migration testing now. ** Gerrit 08:00 UTC load issues (clarkb 20220830) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220830) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html ** Fedora 36 Rollout (clarkb 20220830) *** Do we need to wait for OpenStack to release first? *** We'll update the nodeset and most jobs will switch automatically. *** Will we remove Fedora 35 next? * Open discussion

1 0

Team Meeting Agenda for August 23, 2022
by Clark Boylan 22 Aug '22

22 Aug '22

Hello, We will meet with this agenda on August 23, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clark appears to have been the only Service Coordinator nominee. ** OpenStack release is fast approaching. https://releases.openstack.org/zed/schedule.html feature freeze September 2 ish. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220823) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220823) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The test deployment is largely there now. The next step is testing data migration from mm2 to mm3. ** Gitea 1.17 upgrade (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** This should be ready to go now that 1.17.1 has been released. Just need to decide on a good window of time to deploy it. ** Gerrit 08:00 UTC load issues (clarkb 20220823) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220823) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html *** Implementation should be straightforward; does this warrant an OpenDev spec, or should I just propose the change? * Open discussion

1 0

Self Nomination for OpenDev Service Coordinator
by Clark Boylan 17 Aug '22

17 Aug '22

Hello, I am submitting my nomination for OpenDev Service Coordinator. I plan to continue to focus on building sustainable services with the current resources we have. Upgrading servers to Jammy and replacing old services like mailman 2 are high on my todo list. I have been doing this for a while and don't have a lot of new things to say. I think it is important that these community services/resources continue to be accessible to the community, and that requires ensuring they are sustainable. If you have been happy with the last six months then you can expect more of the same. I am still happy for someone else to volunteer as well. You have a few more hours if anyone else would like to give it a go. I think having a group that can share these responsibilities is a good thing. But I am happy to continue as well. Clark

2 1

Team Meeting Agenda for August 16, 2022
by Clark Boylan 16 Aug '22

16 Aug '22

We will meet August 16, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination period happening now until August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving Grafana management tooling (clarkb 20220816) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/851955 Last remaining cleanup change ** Bastion host (ianw 20220816) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220816) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220816) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The test deployment is largely there now. The next step is testing data migration from mm2 to mm3. ** Gitea 1.17 upgrade (clarkb 20220816) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** https://github.com/go-gitea/gitea/milestone/122 1.17.1 Milestone is in progress * Open discussion

1 0

Team Meeting Agenda for August 9, 2022
by Clark Boylan 09 Aug '22

09 Aug '22

We will meet on August 9, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination period happening now until August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving Grafana management tooling (clarkb 20220809) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220809) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220809) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220809) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The service deployment is just about all there. Need to figure out configuring the service and lists properly now. ** Gitea 1.17 upgrade (clarkb 20220809) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** https://github.com/go-gitea/gitea/milestone/122 1.17.1 Milestone is in progress * Open discussion

1 0

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

7 11

Service Coordinator Election August 2022 Edition
by Clark Boylan 03 Aug '22

03 Aug '22

It is almost that time again. Back in February I said that we'd have a Service Coordinator Election nomination period that runs from August 2, 2022 to August 16, 2022 [0]. If you'd like to run (I'm more than happy for someone else to do it) now is the time to start thinking about that. I'm giving everyone a head up a week in advance today, and will send a notice on August 2nd that the nomination period is beginning. If you'd like to take this on you can send your nomination as a reply to this email thread or in a new email thread to the service-discuss list. [0] https://lists.opendev.org/pipermail/service-discuss/2022-February/000318.ht… Clark

1 1

Infra Team Meeting Agenda for August 2, 2022
by Clark Boylan 02 Aug '22

02 Aug '22

We will meet with this agenda on August 2, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination time begins now (August 2, 2022) and ends August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220802) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Improving Grafana management tooling (clarkb 20220802) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220802) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220802) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/851094/2 Run system-config-run-base against Jammy *** https://review.opendev.org/c/opendev/system-config/+/851266/1 Fix install-docker role for Jammy *** https://review.opendev.org/c/opendev/system-config/+/851248 WIP change to run mailman3 on Jammy ** Gitea 1.17 upgrade (clarkb 20220802) *** https://review.opendev.org/c/opendev/system-config/+/847204 ** Rocky images no longer booting (clarkb 20220802) *** Lack of machine ID broke installation of kernels to /boot which broke our Grub config *** https://review.opendev.org/c/openstack/diskimage-builder/+/851687 * Open discussion

1 0

Team Meeting Agenda for July 26, 2022
by Clark Boylan 26 Jul '22

26 Jul '22

We will meet on July 26, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220726) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Improving Grafana management tooling (clarkb 20220726) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220726) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220726) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Zuul job POST_FAILURES (clarkb 20220726) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** We now log the target swift region before uploading logs. Next step is collecting info on where this occurs to determine if it is consistent. ** OpenDev Service coordinator elections beginning soon (clarkb 20220726) *** In February I stated August 2 - 16, 2022 would work as a nomination period. *** Will send email about the process after our meeting. * Open discussion

1 0

Team Meeting Agenda for July 19, 2022
by Clark Boylan 18 Jul '22

18 Jul '22

Hello, We will meet on July 19, 2022 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220719) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Improving Grafana management tooling (clarkb 20220719) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220719) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220719) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Zuul + Ansible v5 + Glibc deadlock (clarkb 20220719) *** Using backported Debian testing glibc on Zuul images now. ** Zuul job POST_FAILURES (clarkb 20220719) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** https://review.opendev.org/c/opendev/base-jobs/+/848881 Add remote log store location debug info to prod base job. ** New Gerrit 3.5 caches (clarkb 20220719) *** Gerrit 3.5 added some new caches that consume a fair bit more disk space. *** We've grown the disk for Gerrit to accommodate this change with newer Gerrit. *** https://review.opendev.org/c/opendev/system-config/+/849886 is our fallback which will disable the caches should their growth continue to be unsustainable. * Open discussion

1 0

Re: [service-announce] Updating Zuul's Default Ansible Version to Ansible v5
by Ian Wienand 11 Jul '22

11 Jul '22

On Wed, Jun 15, 2022 at 12:11:00PM -0700, Clark Boylan wrote: > The OpenDev team will be updating the default Ansible version in our > Zuul tenants from Ansible 2.9 to Ansible 5 on June 30, 2022. Zuul > itself will eventually update its default, but making the change in > our tenant configs allows us to control exactly when this happens. Note this has been merged with https://review.opendev.org/c/openstack/project-config/+/849120 Just for visibility I've cc'd this to openstack-discuss; but please subscribe to service-announce [1] if you're interested in such OpenDev infra updates. Thanks, -i [1] https://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

1 0

Team Meeting Agenda for July 5, 2022
by Clark Boylan 05 Jul '22

05 Jul '22

Hello, We will meet on July 5, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** clarkb missing July 12 meeting * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220705) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Auto upgrades of Zuul are in place now. ** Improving Grafana management tooling (clarkb 20220705) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Run a custom URL shortener service (frickler 20220705) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL ** Zuul job POST_FAILURES (clarkb 20220705) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** https://review.opendev.org/c/opendev/base-jobs/+/848027 Add remote log store location debug info to base jobs. ** Bastion host (ianw 20220628) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? * Open discussion Apologies for getting this agenda sent out late. Yesterday was a holiday, and I had less time indoors than anticipated.

1 0

New approaches for grafana.opendev.org
by Ian Wienand 01 Jul '22

01 Jul '22

Hello, 65;6800;1c Currently all graphs pushed to grafana.opendev.org are defined in project-config/graphana as YAML files consumed by grafyaml [1] The fundamental tension for grafyaml is that the upstream Grafana project do not document and publish a defined schema for dashboards and their components. grafyaml has a subset of the upstream data model which is incomplete, in some cases buggy -- but also, perhaps most importantly, undocumented [1]. We have over a million data points of interesting information in graphite but I feel there are significant barriers to new and interesting visualisations. With no clear documentation, either upstream or in grafyaml, where is somebody supposed to start? A series of changes have been reviewed and landed through grafyaml that allow it to upload dashboards exported directly from the Grafana UI in its native .json format. I would like to achieve some consensus that we use this feature in the OpenDev environment. I will leave aside the issues with the schema encoded in grafyaml; it's possible this might be fixed. AIUI the main reason for duplicating the schema in grafyaml was that it presented more reviewable YAML files. To this I would say: 1) Layout of the page; i.e. the rows, panels, nesting, etc. My argument here is that reviewers having to build a mental model of what a dashboard will look like -- from either YAML or json -- does not make for thorough reviews, especially if you're not already intimately familiar with the desired output. To this end, I have added a new job "project-config-grafana" which produces an artifact "Screenshots" that loads changed graphs into a Grafana instance and stores actual screenshots loaded from a headless browser. I believe this is a much more effective way to review proposed layout changes for both formats of input. 2) The data graphed. This comes down to the metric selected, and any functions applied. My argument here is that firstly the screenshot is a good way to evaluate this; for example you will see if you've accidentally treated milliseconds as seconds, etc. when the graph axis is wrong. As to the actual data -- ensuring we have the right metric, etc. -- I would say that the "raw" output of the exported graphs just isn't that hard to parse. It is unobfuscated and reads logically. I have proposed some examples: https://review.opendev.org/c/openstack/project-config/+/833213/6/grafana/in… https://review.opendev.org/c/openstack/project-config/+/848212/3/grafana/no… I think you can clearly see the metrics chosen and the functions applied. 3) Generally more confusing. This is true, as the .json file is meant for Grafana to read. However, for better or worse, this is the actual data model of your graph page. To this end, I have proposed documentation and a helper-script to start a Grafana instance in a local container, and load it with the defined dashboards: https://review.opendev.org/c/openstack/project-config/+/833214/ This is useful for interactive editing sessions to develop new dashboards, and if a reviewer wishes to examine a change more closely than the screenshots provided by CI, they can simply pull the change from gerrit and load it into a live instance using this simple method. I think this is a significantly lower barrier to get people developing new and interesting things against the data provided. I'm not proposing any existing graph need change [2] and grafyaml's features to setup datasources and load the graphs are still used. I'm not proposing we remove or even stop any development of the YAML schema if people want to work on that and prefer to keep their graphs that way. I think that there is a great resource here that is underutilised, and my hope is we have a path to greatly reduce the barriers to new contributions. Sorry for the long mail, -i [1] Grafana does a good job of backwards compatibility, so "old" dashboards work in new releases. Hence our extant graphs, though producing output that looks very different from what the UI produces now, generally work. Modulo some bugs where the "update" process doesn't work (thresholds was one I found), deprecations of features that will disappear (c.f. time-series graphs) and just the many panel types that are completely unsupported. [2] Though most extant graphs use deprecated panel types that will have to be updated one day; but that's an issue for another time.

1 0

Team Meeting Agenda for June 28, 2022
by Clark Boylan 28 Jun '22

28 Jun '22

Hello, We will meet with this agenda on June 28, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb missing July 12 meeting * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220628) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Auto Zuul upgrade and reboots didn't run due to a flock issue. Should be fixed for this next weekend. ** Gerrit 3.5 upgrade (ianw 20220628) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** #link https://review.opendev.org/c/opendev/system-config/+/847035 Removing Gerrit 3.4 and adding Gerrit 3.6 images and testing. ** Improving Grafana management tooling (clarkb 20220628) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://review.opendev.org/q/topic:grafana-screenshots Improved testing of our dashboards to see that they render as expected. ** Run a custom URL shortener service (frickler 20220628) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL ** Zuul job POST_FAILURES (clarkb 20220628) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** The cause seems to be post-run timeouts during swift log uploads *** Both OSA and tripleo upload quite a number of log files. It could be related to this, but we're yet to find a root cause due to difficulty of debugging things that don't log properly. ** Bastion host (ianw 20220628) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? * Open discussion

1 0

Team Meeting Agenda for June 21, 2022
by Clark Boylan 21 Jun '22

21 Jun '22

Hello, We will meet with this agenda on June 21, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220621) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://review.opendev.org/c/opendev/system-config/+/846195 Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220621) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB **** review02's mariadb was upgraded to 10.6 from 10.4 using the process described in https://etherpad.opendev.org/p/gerrit-upgrade-3.5 ** Gerrit 3.5 upgrade (ianw 20220621) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Couple of issues discovered **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16018 **** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** Time to look at removing our 3.4 image builds and adding 3.6 image builds with updated upgrade testing. ** Enable webapp on nodepool launchers? (frickler 20220621) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220621) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team meeting agenda for June 14, 2022
by Clark Boylan 14 Jun '22

14 Jun '22

We will meet on June 14, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220614) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220614) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220614) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 ** Zuul changing default Ansible version to v5 soon (clarkb 20220614) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Enable webapp on nodepool launchers? (frickler 20220611) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220611) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team Meeting Agenda for May 31, 2022
by Clark Boylan 31 May '22

31 May '22

Hello, We will meet on May 31, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** Summit happens June 7-9. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220531) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul_… Automated graceful Zuul upgrades and server reboots **** Hit a bug with zuul merger graceful shutdowns **** Hit a bug in zuul's model api 8 transition **** Otherwise seemed to work well. ** Container maintenance (clarkb 20220531) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220531) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Manually triggering periodic jobs (frickler 20220531) *** What is the correct process for manually triggering periodic jobs? ** Zuul changing default Ansible version to v5 soon (clarkb 20220531) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Removing Ethercalc (clarkb 20220531) *** Announced removal date was May 31, 2022 *** Last call before clarkb proceeds with this cleanup ** Do we want to hold a meeting June 7, 2022 (clarkb 20220531) *** clarkb and fungi are unlikely to be able to attend due to the summit acting as a conflict * Open discussion

1 0

Team Meeting Agenda for May 24, 2022
by Clark Boylan 24 May '22

24 May '22

Hello, We will meet with this agenda on May 24, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Summit happens June 7-9. No meeting June 7. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220524) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220524) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220524) *** Last major missing piece is wheel builds (and mirroring?) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/842839 *** https://review.opendev.org/c/openstack/project-config/+/842841 ** Gerrit 3.5 upgrade planning (ianw 20220524) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Move requirements propose-updates job to py3.8 + py3.9 (frickler 20220523) *** Initial change: https://review.opendev.org/c/openstack/project-config/+/837815 *** Build results: https://zuul.opendev.org/t/openstack/builds?job_name=propose-updates&projec… *** First attempt at fixing: https://review.opendev.org/c/openstack/project-config/+/842837 *** Second fix: https://review.opendev.org/c/openstack/project-config/+/842853 *** Related question: How to properly trigger manual runs of periodic pipeline? **** Using example from zuul docs is failing, e.g. https://zuul.opendev.org/t/openstack/build/3d28674ac1044c22956c20ed39c82a4a ** Zuul changing default Ansible version to v5 soon (clarkb 20220524) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Playbooks and roles likely need updating. We should look into that for our hosted projects. *** Tenants can override a default to older Ansible if necessary. *** Eventually Ansible will remove old Ansible support. * Open discussion

1 0

Team Meeting Agenda for May 17, 2022
by Clark Boylan 17 May '22

17 May '22

We will meet on May 17, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. ** ClarkB out May 18 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Gerrit 3.5 upgrade planning (ianw 20220517) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Scheduling * Open discussion

1 0

Team Meeting Agenda for May 10, 2022
by Clark Boylan 10 May '22

10 May '22

Hello, We will meet on May 10, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220510) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Automating the build process for our Ubuntu PPA packages (clarkb 20220510) *** ianw is adding more automation to our vhd-util and openafs package builds. This should make it easier for us to update the packages and add new distro releases. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Keycloak container image update (clarkb 20220510) *** Keycloak stopped publishing to Docker Hub. *** Keycloak has two versions of the image. The Legacy wildfly image (what we use) and a non wildfly image we should probably convert to. *** https://review.opendev.org/c/opendev/system-config/+/840529 Pull keycloak from quay.io * Open discussion

1 0

Team Meeting Agenda for May 3, 2022
by Clark Boylan 03 May '22

03 May '22

We will meet with this agenda on May 3, 2022 at 19:00 UTC in @opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220503) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220503) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220503) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220503) *** Ubuntu-ports mirroring is in progress *** Do we need to build our openafs package for jammy (this would be to build wheel mirrors to start, but would potentially be used on zuul executors etc in the future) ** Shutting down Ethercalc (clarkb 20220503) *** The software doesn't seem to be well maintained *** The software lacks history tracking and rollback like etherpad *** Is largely used for PTG and other tools could be used for that (like PTGbot) *** clarkb would like to propose shutting this service down at the end of the month. * Open discussion

1 0

Team Meeting Agenda for April 26, 2022
by Clark Boylan 26 Apr '22

26 Apr '22

Hello, We will meet on April 26, 2022 at 19:00 UTC in #opendev with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220426) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220426) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220426) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220425) *** Repo mirrors https://review.opendev.org/c/opendev/system-config/+/838923 *** DIB testing https://review.opendev.org/c/openstack/diskimage-builder/+/836228 *** Actual build https://review.opendev.org/c/openstack/project-config/+/839057 *** Nodepool release with fixes? *** Drop mirroring for Xenial in order to recover AFS quota? **** We cleaned up a bunch of centos, epel, opensuse, open euler, fedora, etc mirrors to remove unnecessary content which made more than enough room for Jammy. ** Deleting the subunit2sql MySQL Trove instance (clarkb 20220426) *** The subunit2sql workers and the health api service that sat in front of this database are gone now *** Currently using 286GB of disk. Do we need to backup/snapshot this or can we simply delete it and move on. * Open discussion

1 0

Skipping Tomorrow's (April 19, 2022) Team Meeting
by Clark Boylan 19 Apr '22

19 Apr '22

Hello everyone, Due to many of us being out with either holidays or errands I'll go ahead and cancel tomorrow's (April 19, 2022) meeting. If you have something worth discussing feel free to start a thread on this mailing list or join us in #opendev on OFTC and start a conversation there. Clark

1 0

Team Meeting Agenda for April 12, 2022
by Clark Boylan 12 Apr '22

12 Apr '22

We will meet with this agenda on April 12, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220412) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220412) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220412) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Scheduling project renames (clarkb 20220412) *** Penciled in April 15, 2022 we'll coordinate more as that date gets closer. ** Time for another git-review release (fungi 20220412) ** Time for another bindep release (clarkb 20220412) * Open discussion

1 0

Team Meeting Agenda for April 5, 2022
by Clark Boylan 05 Apr '22

05 Apr '22

The PTG is happening this week but we'll sneak a short meeting in. We will meet on April 5, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Virtual PTG is happening now. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220405) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220405) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220405) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Retiring opendev/gerrit repo (ianw 20220405) *** Decided to delete branches and then set HEAD content to explanation of why the content has gone away. ** Scheduling project renames (clarkb 20220405) *** Penciled in April 15, 2022 we'll coordinate more as that date gets closer. * Open discussion

1 0

Team Meeting Agenda for March 29, 2022
by Clark Boylan 29 Mar '22

29 Mar '22

We will meet with this agenda on March 29, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack "Yoga" release on Wednesday (tomorrow relative to this meeting) ** Virtual PTG next week. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220329) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220329) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Upgrading Gitea (clarkb 20220329) *** We have another Gitea update to apply. *** https://review.opendev.org/c/opendev/system-config/+/835307 ** Spring cleaning for old reviews? (frickler 20220329) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Retiring opendev/gerrit repo (ianw 20220329) *** Decided to delete branches and then set HEAD content to explanation of why the content has gone away. ** Scheduling project renames (clarkb 20220329) *** Penciled in April 15, 2022 we'll coordinate more as that date gets closer. * Open discussion

1 0

Regression in Gerrit regarding rebases from Gerrit UI
by Radosław Piliszek 22 Mar '22

22 Mar '22

Hi! It seems I am experiencing a regression in Gerrit regarding the rebase capability available from the Gerrit UI. Before, when a child patch (in a patch stack) had its parent patch modified, it was possible to quickly rebase on the current state of the parent patch via the rebase button. Now, Gerrit UI always claims "This change is up to date with its parent." and suggests rebasing on top of the branch or any other patch. Just reporting this issue to see if others also experience this behaviour or something broke on my side (however unlikely). -yoctozepto

2 2

Team Meeting Agenda for March 22, 2022
by Clark Boylan 21 Mar '22

21 Mar '22

We will meet with this agenda on March 22, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack "Yoga" release about a week away. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220322) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220322) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220322) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Clarkb has been looking at opendev/system-config changes **** Many old changes have been abandoned: https://review.opendev.org/q/project:opendev/system-config+status:abandoned **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Retiring opendev/gerrit repo (ianw 20220322) *** Had some discussion bit inconclusive *** Delete branches? Merge retirement change? *** Want to ensure this repo falls out of search engines ** Scheduling project renames (clarkb 20220322) *** We have a request to rename a repo. Should schedule renames for some time after the OpenStack Release and PTG? * Open discussion

1 0

Infra Team Meeting Agenda for March 15, 2022
by Clark Boylan 15 Mar '22

15 Mar '22

Hello, We will meet with this agenda on March 15, 2022 at 19:00 UTC in #opendev-meeting. Note that for many of us in North America the DST time change happened over the weekend and the meeting time may have shifted relative to your local timezone. == Agenda for next meeting == * Announcements ** OpenStack is a couple weeks away from releasing "Yoga". * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220315) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220315) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220315) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://review.opendev.org/c/openstack/project-config/+/831837 Finalize repo retirements. *** Next up looking at system-config. ** Gitea 1.16 (clarkb 20220315) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.x when we are ready. *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Backup server pruning (clarkb 20220315) *** One of our backup servers needs to be pruned for disk space reasons. * Open discussion

1 0

Team Meeting Agenda for March 1, 2022
by Clark Boylan 01 Mar '22

01 Mar '22

We will meet with this agenda March 1, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Skip meeting next week (March 8, 2022) due to travel. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220301) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220301) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220301) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. *** https://review.opendev.org/q/topic:retirement+status:open Changes to retire all unused the repos. *** Once the above changes land we'll abandon related changes and then remove the repos from zuul entirely. ** Gitea 1.16 (clarkb 20220301) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.x when we are ready. *** Bug fixes for diffs will be in 1.16.3 as well as some dependency updates. Probably worth waiting for 1.16.3. *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Rocky Linux 8 Images (clarkb 20220301) *** Need to update DIB in our nodepool images https://review.opendev.org/c/zuul/nodepool/+/831108 ** zuul-registry bugs affecting system-config (clarkb 20220301) *** There are issues with concurrent uploads to zuul-registry. We notice them when making updates to Gerrit because we build multiple images with shared layers. *** https://review.opendev.org/c/zuul/zuul-registry/+/831235 should hopefully address these problems in zuul-registry. * Open discussion

1 0

Team Meeting Agenda for February 22, 2022
by Clark Boylan 22 Feb '22

22 Feb '22

== Agenda for next meeting == * Announcements ** Service Coordinator election is over. Clarkb was the only nomination. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220215) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220215) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220222) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. *** https://review.opendev.org/q/topic:retirement+status:open Changes to retire all unused the repos. *** Once the above changes land we'll abandon related changes and then remove the repos from zuul entirely. ** Gitea 1.16.1 (clarkb 20220222) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.1 when we are ready *** https://104.130.74.7:3081/opendev/system-config Test site via held node here *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Gerrit gitea weblinks (clarkb 20220222) *** Gerrit now links to gitea ** Rocky Linux 8 Images (clarkb 20220222) *** Need EPEL support in DIB for this https://review.opendev.org/c/openstack/diskimage-builder/+/830278 * Open discussion

1 0

Service Coordinator Election Nominations
by Clark Boylan 16 Feb '22

16 Feb '22

Hello everyone, I've noticed that while I brought up the planned nomination period of January 25, 2022 to February 8, 2022 for our Service Coordinator election in our meeting agendas, and the meetings themselves; I never sent a proper announcement to the mailing list about this. We discussed this in today's meeting and there were no objections to extending the nomination period to 23:59 UTC February 15, 2022. This way everything is formally announced and anyone interested has time to sign up. Consider this the official call for nominations. You've got a week to put your name in the hat if you'd like to give the Service Coordinator hat a go. If we have more than one nomination at that point we'll figure out an election process that should run for a week. If we end up with one nomination that individual will win by default. Looking at the 2022 calendar for the next election: we should plan to have nominations from August 2 - 16, 2022. Then have an election the week after if necessary. Clark

2 2

Infra Team Meeting Agenda for February 15, 2022
by Clark Boylan 15 Feb '22

15 Feb '22

We will meet Tuesday, February 15, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** Service Coordinator nomination time has begun. You have until February 15, 2022 UTC end of day. That is today. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220215) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220215) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220215) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** https://review.opendev.org/c/opendev/system-config/+/829115 Removes fedora 34 mirror content ** Spring cleaning for old reviews? (frickler 20220215) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. *** https://review.opendev.org/c/opendev/system-config/+/829119 Removes unused repos from integration testing. *** https://review.opendev.org/c/openstack/project-config/+/829121 Sets noop jobs on unused repos so that they can be retired. ** Gitea 1.16.1 (clarkb 20220215) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.1 when we are ready *** https://104.130.74.7:3081/opendev/system-config Test site via held node here *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Gerrit gitea weblinks (clarkb 20220215) *** Fixes in progress upstream **** https://gerrit-review.googlesource.com/c/gerrit/+/329279 **** https://gerrit-review.googlesource.com/c/plugins/gitiles/+/330361 *** Not sure why the bazel test runs fail complaining about java runtimes. This happens for me too on the gerrit repo but only happens to gitiles upstream (not gerrit). * Open discussion

1 0

Team Meeting Agenda for February 8, 2022
by Clark Boylan 08 Feb '22

08 Feb '22

We will meet with this agenda on February 8, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP needs your input: https://openinfra.dev/summit/ Tomorrow (the 9th) is the last day to get your entries in. ** Service Coordinator nomination time is ending. You have until February 8, 2022. That is today. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220208) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220208) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220208) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** CentOS 8 is now gone. Any progress on Fedora 34 removal? ** Spring cleaning for old reviews? (frickler 20220208) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. ** Gerrit mergeabilityComputationBehavior (frickler 20220127) *** https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.0/con… *** Frickler volunteered to write a change for this. * Open discussion

1 0

Team Meeting Agenda for February 1, 2022
by Clark Boylan 31 Jan '22

31 Jan '22

We will meet at 19:00 UTC on February 1, 2022 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP needs your input: https://openinfra.dev/summit/ ** Service Coordinator nomination time has begun. You have until February 8, 2022 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220201) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220201) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220201) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Changes to remove CentOS 8 are up. Are we ready to go ahead with this? In order they should be merged: **** https://review.opendev.org/c/openstack/project-config/+/827183 **** https://review.opendev.org/c/opendev/base-jobs/+/827181 **** https://review.opendev.org/c/openstack/project-config/+/827184 **** https://review.opendev.org/c/opendev/system-config/+/827186 ** Spring cleaning for old reviews? (frickler 20220201) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… *** Clarkb to produce a list of repos that can be retired. We can then retire the repos and abandon their open reviews as step 0 ** Gerrit mergeabilityComputationBehavior (frickler 20220127) *** https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.0/con… *** This was disabled while upgrading to 3.4.0 since the default is "NEVER" *** We could set the option to "API_REF_UPDATED_AND_CHANGE_REINDEX" to restore the previous behavior *** It doesn't look like we have to have any concerns in regard to CPU load http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=70348… ** Gerrit bugs/issues that we are tracking (clarkb 20220201) *** ACLs for pushing signed tags are ignored. Should be fixed in 3.4 now. https://review.opendev.org/c/opendev/system-config/+/827153 will update our images. *** URL substititions for gitweb urls does not properly include the hash which is needed by gitea. https://bugs.chromium.org/p/gerrit/issues/detail?id=15589 *** Backlog of ssh git pulls against Gerrit slows Zuul down. https://bugs.chromium.org/p/gerrit/issues/detail?id=15649 * Open discussion

1 0

Infra Team Meeting Agenda for January 25, 2022
by Clark Boylan 25 Jan '22

25 Jan '22

Hello, We will meet with this agenda on January 25, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ ** Service Coordinator nomination time has begun. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220125) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220125) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Buster to Bullseye updates are complete. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220125) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Plan to remove CentOS 8 from Nodepool and Zuul first thing next week. ** Spring cleaning for old reviews? (frickler 20220125) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… *** (clarkb) Considering that we had a big shift in approach to config management about 2 years ago I would say abandoning anything prior to that is probably fine. I have a change or two since then that might be good to keep around. Maybe we start at 2 years and see what that looks like? We can always restore changes too. ** Updating Grafana (clarkb 20220125) *** https://review.opendev.org/c/opendev/grafyaml/+/825990 needed to fix api use against newer grafana *** https://review.opendev.org/c/opendev/system-config/+/825410 update to latest grafana * Open discussion

1 0

Team Meeting Agenda for January 18, 2022
by Clark Boylan 18 Jan '22

18 Jan '22

We will meet January 18, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220118) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220118) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Buster to Bullseye updates are complete. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220118) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Tumbleweed has been removed *** CentOS 8 removal process has begun. Gave projects until the end of the month before we remove the nodeset and nodepool configs. ** Scheduling Gerrit Project Renames (clarkb 20220118) *** Target January 24 around 22:00 UTC? ** Spring cleaning for old reviews? (frickler 20220113) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… * Open discussion

1 0

Re: [service-announce] Changes to OpenDev's stable of CI images
by Clark Boylan 11 Jan '22

11 Jan '22

On Tue, Dec 7, 2021, at 8:38 AM, Clark Boylan wrote: > Hello, > > CentOS 8 reaches end of life on December 31, 2021. This means now is > the time for you to convert jobs to CentOS 8|9 Stream or some other > appropriate platform. The OpenDev team will plan to remove CentOS 8 > images and mirrors from our infrastructure in early January, hopefully > by January 14, 2022. Again, please update your jobs now so that you are > prepared. These changes are necessary as the distro will no longer be > supported, and there is no way for us to ensure that the images will be > secure or that we'll even be able to build them. > > We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's > short support window we typically swap out one Fedora release with the > next when the next is ready. If you are using the fedora-34 label > please update it to fedora-35. If you are using fedora-latest we'll be > updating that to use fedora-35 and the swap will happen automatically > for you. We expect to have Fedora 34 cleaned up by the end of the year. > > Additionally both OpenSUSE 15 and Tumbleweed could use some help. For > OpenSUSE 15 we need to update it to 15.3 as well as updating the > associated mirrors if we plan to keep it around. I'm happy to keep this > image around if we can get volunteers in the community that can help > with that. If there isn't interest in modernizing the OpenSUSE 15 > images I'd like to propose we clean them up in January along with the > CentOS 8 images. For Tumbleweed, the images haven't received much use > or attention. I'd like to propose that we simply retire this image and > clean it up. We had thought it might be a good platform for monitoring > upcoming changes in the Linux world, but the reality is that only works > if you've got people active caring for it and adjusting to those > changes. The Tumbleweed cleanup will likely begin before the end of the > year. > > Finally, our Gentoo images bounce between being buildable and broken > with more time spent being broken than not. I'd like to propose that > this image also get cleaned up in the near future. Again, if there is > interest in the community for keeping this alive I think we can make > that happen. But as is this image is not in a sustainable place and it > would be better for us to stop running jobs on it. > > Let us know if you have any questions or comments, To followup on this we have found volunteers for Gentoo and OpenSUSE Leap 15 maintenance. This means those images are not immediately up for removal. However, if those volunteers find they aren't sustainable for one reason or another they may still be removed. OpenSUSE Tumbleweed has been removed and as far as we could tell nothing was using it when we removed it. Shouldn't create any problems. CentOS 8 removal has begun with a number of changes [0] to start removing some of the infrastructure bits (like wheel mirrors and openafs packages) that supported it. I will be reaching out to OpenStack as well as they seem to be the primary users of the centos-8 label. Hopefully we'll be able to remove all of the existing centos-8 jobs cleanly, but if not then we'll wait until a reasonable amount of time and cleanup has passed (a couple of weeks?) before force merging the removal of the nodeset and label from our zuul configs. Fedora 34 cleanup is still in progress, but should pick up again as people return from holiday. [0] https://review.opendev.org/q/topic:%22remove-centos-8%22+status:open

2 1

Infra Team Meeting Agenda for January 11, 2022
by Clark Boylan 10 Jan '22

10 Jan '22

Our first meeting of the year will be held January 11, 2022 at 19:00 UTC in #opendev-meeting. I expect it will still be pretty quiet due to holidays so we might get through it quickly. This is our agenda: == Agenda for next meeting == * Announcements ** OpenInfra Foundation Board Election happening this week. Don't forget to vote. * Actions from last meeting * Specs Review * Topics ** ansible-lint failures (clarkb 20220111) *** An unpinned dep (rich) in old ansible-lint has updated and broken ansible-lint. Need to pin rich or update ansible-lint. *** clarkb has a number of these changes up for review now. topic:ansible-lint ** Improving OpenDev's CD throughput (clarkb 20220111) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220111) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Updating Buster images to Bullseye **** https://review.opendev.org/q/hashtag:%22bullseye-image-update%22+status:open The uwsgi image update in particular could use review. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220111) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** CentOS 8 and OpenSUSE Tumbleweed need cleanup this month. ** Scheduling Gerrit Project Renames (clarkb 20220111) * Open discussion

1 0

No Team Meeting December 28, 2021
by Clark Boylan 28 Dec '21

28 Dec '21

Just a heads up that things have been quiet (as expected) the last week. I think we can go ahead and cancel tomorrow's meeting. See you all in a couple weeks as we return to normal scheduling in the new year. Clark

1 0

Team Meeting Agenda for December 14, 2021
by Clark Boylan 14 Dec '21

14 Dec '21

We will meet with this agenda on December 14, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Meetings Cancelled December 21, 2021 and January 4, 2022 for Holidays ** We may have a meeting on December 28, 2021 depending on who is around. * Actions from last meeting * Specs Review * Topics ** Log4j vulnerability fun (clarkb 20211214) *** Discuss what we've learned about this bug and its impact. ** Improving OpenDev's CD throughput (clarkb 20211214) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? ** Container maintenance (clarkb 20211214) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Updating Buster images to Bullseye *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211214) *** Ansible newlist invocations still struggling *** Stack at https://review.opendev.org/c/opendev/system-config/+/820900/ should correct this. ** Spring cleaning our Nodepool images (clarkb 20211214) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Will need volunteers for the work. Probably ok if this proceeds a little slower than advertised. * Open discussion

1 0

Re: [service-announce] Changes to OpenDev's stable of CI images
by Jack Morgan 07 Dec '21

07 Dec '21

Hey Clark, I'd like to help out / volunteer on whatever needs to be done. It doesn't have to be specific to image clean up. I'm not sure how to get started so thought we could meet for coffee but open to other suggestions. On 12/7/21 08:38, Clark Boylan wrote: > Hello, > > CentOS 8 reaches end of life on December 31, 2021. This means now is the time for you to convert jobs to CentOS 8|9 Stream or some other appropriate platform. The OpenDev team will plan to remove CentOS 8 images and mirrors from our infrastructure in early January, hopefully by January 14, 2022. Again, please update your jobs now so that you are prepared. These changes are necessary as the distro will no longer be supported, and there is no way for us to ensure that the images will be secure or that we'll even be able to build them. > > We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's short support window we typically swap out one Fedora release with the next when the next is ready. If you are using the fedora-34 label please update it to fedora-35. If you are using fedora-latest we'll be updating that to use fedora-35 and the swap will happen automatically for you. We expect to have Fedora 34 cleaned up by the end of the year. > > Additionally both OpenSUSE 15 and Tumbleweed could use some help. For OpenSUSE 15 we need to update it to 15.3 as well as updating the associated mirrors if we plan to keep it around. I'm happy to keep this image around if we can get volunteers in the community that can help with that. If there isn't interest in modernizing the OpenSUSE 15 images I'd like to propose we clean them up in January along with the CentOS 8 images. For Tumbleweed, the images haven't received much use or attention. I'd like to propose that we simply retire this image and clean it up. We had thought it might be a good platform for monitoring upcoming changes in the Linux world, but the reality is that only works if you've got people active caring for it and adjusting to those changes. The Tumbleweed cleanup will likely begin before the end of the year. > > Finally, our Gentoo images bounce between being buildable and broken with more time spent being broken than not. I'd like to propose that this image also get cleaned up in the near future. Again, if there is interest in the community for keeping this alive I think we can make that happen. But as is this image is not in a sustainable place and it would be better for us to stop running jobs on it. Thanks, -- Jack Morgan

2 1

Parallel production jobs changes
by Ian Wienand 07 Dec '21

07 Dec '21

Hi, To recap: currently production deployment jobs run sequentially. Zuul starts the job on an executor, which is setup to log into the bastion host. The job sets up the system-config playbooks on the bastion host and Ansible is run from there against the production server. To run in parallel, each job needs to not assume it owns the system-config playbooks on the bastion host. Each Zuul *buildset* can use the same system-config playbook checkout though. To achieve this we need to rework the dependencies; each production job needs to depend on a common source-setup job. Once the source is setup on the bastion host, the actual production jobs can run in parallel. To the changes... Firstly, I believe we're doing the setup steps for the executor to log into bridge twice: https://review.opendev.org/c/opendev/system-config/+/818190 removes this duplication, and should be safe to merge. As pointed out in prior reviews when running in the periodic or hourly pipelines each job overrides that bastion host checkout to master. https://review.opendev.org/c/opendev/base-jobs/+/818189 moves this step into base-jobs, in preparation for only being done once by the separate source-setup job. I believe this will be safe to merge; system-config will just do it again in an idempotent way, until: https://review.opendev.org/c/opendev/system-config/+/818191 merges, which drops this step from system-config. We can then merge the system-config job dependency updates in https://review.opendev.org/c/opendev/system-config/+/807672 This should mean that all jobs not only rely on the correct base jobs, but jobs that need certificates, etc. will be relying on the letsencrypt job, etc. This should be safe to merge as nothing should actually change, we just have stricter dependencies. After this, I think we are ready to refactor the base jobs into the two separate steps -- firstly setup the keys on the executor to log into the bastion host, then setup the source to use on the bastion host: https://review.opendev.org/c/opendev/base-jobs/+/807807 This initial refactor should be safe to merge as it creates two new jobs, but the existing base job keeps running both steps as-is. Then we are ready for the penultimate change: https://review.opendev.org/c/opendev/system-config/+/807808 This updates the system-config jobs to all depend on "infra-prod-setup-src" which will be the canonical job that sets up the source repository on bridge.o.o. All other jobs in the buildset will depend on this job, ensuring consistency for a run. This should also be safe, as it again doesn't actually change ordering. Once all this is in, we need the final change to enable parallel running (and think about correct semaphores between periodic/hourly and regular runs). That is yet to be written, but we have enough to get to that point! -i

1 1

Team Meeting Agenda for December 7, 2021
by Clark Boylan 06 Dec '21

06 Dec '21

Hello, We will meet with this agenda on December 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211207) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies *** We ended up in a weird spot after relying on the per buildset update source job: **** We weren't syncing updated repo content to bridge but to a single use test node **** We weren't hard depending on the job so if it didn't run we would run the other jobs without updated sources **** We weren't adding the update source job to hourly and daily periodic buildsets *** Next steps happening in https://review.opendev.org/c/opendev/system-config/+/820320 ** User management on our systems (clarkb 20211207) *** Still struggling with matrix-gerritbot. Now openssh on it complains that uid 11000 doesn't exist. *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211207) *** Ansible newlist invocations still struggling *** https://review.opendev.org/c/opendev/system-config/+/820397 and parent are a stack to investigate and fix this. ** keycloak.opendev.org exists now (clarkb 20211207) *** This can be used to gain more familiarity with the tool to implement our authentication spec. *** Are we treating this as production ready? Might be good to avoid that for now while we investigate how to apply this to our previous spec? ** Gerrit User Summit (clarkb 20211207) *** Gerrit 3.2 is EOL *** We can integrate with the new Checks UI stuff if we implement a Checks plugin for Zuul. **** This plugin would run within Gerrit and poll an API on the Zuul side for CI results and render them in a consistent manner. *** Composable Submit Requirements **** This is a new system for expressing what is required to merge a change **** Relies on boolean query language using Gerrit's existing query predicates. **** Mostly we should be careful to avoid accidentally allowing too much access when this shows up. *** ChronicleMap libmodule plugin persistent caches **** This replaces H2 caches for a number of things and gives the caches persistence. Can apparently have a big impact on performance **** One suggestion was that Gerrit incorporate this by default. ** Spring cleaning our Nodepool images (clarkb 20211207) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** https://etherpad.opendev.org/p/xvu2oKUQVLiHIHsvGRUt draft communications for proposed cleanups. * Open discussion

1 0

UbuntuOne/Launchpad two-factor authentication
by Jeremy Stanley 01 Dec '21

01 Dec '21

For a little over a year Ian, Clark and I have been using the multi-factor authentication feature of UbuntuOne SSO (i.e. Launchpad) in order to more strongly secure the accounts we rely on for OpenID logins to the Web interfaces of our services like Gerrit and StoryBoard. It's gone smoothly, I think, and so we're probably overdue on our plan to offer this capability to other OpenDev users. Support for this is not enabled by default, your SSO account needs to be a member of a group which is granted the feature. We have one such group authorized for this purpose, which can be found here: https://launchpad.net/~opendev-2fa Please see the information and important caveats documented in the group description. I expect the process would be something like using the LP group members page to request membership for your account, and then one of the group administrators would approve the request, after which you would be able to proceed with configuration of your token or other HOTP/TOTP authenticator. I'm bringing it up here first for discussion, in order to see if anyone has any concerns or related suggestions, but barring none I'd like to move forward with a "soft" (quiet) call for wider testing the first week of December. -- Jeremy Stanley

3 5

Team Meeting Agenda for November 30, 2021
by Clark Boylan 29 Nov '21

29 Nov '21

We will meet with this agenda on November 30, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Gerrit User Summit happening December 2&3 virtually. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211130) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies *** https://review.opendev.org/c/opendev/system-config/+/807808 Update system-config once per buildset. *** https://review.opendev.org/c/opendev/base-jobs/+/818297/ Reduce actions needed to be taken in base-jobs. *** Next up is thinking about semaphores and actually trying to run jobs in parallel. Dependencies should be correct. ** Zuul multi scheduler setup (clarkb 20211130) *** Possible for zuul updates to break rolling system updates. Not sure if we've got signal on when that happens yet. ** User management on our systems (clarkb 20211130) *** Ran into problems with matrix-gerritbot. There is an updated image we can pull and use to try again. *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** UbuntuOne/Launchpad two-factor OpenID authentication availability (fungi 20211130) *** http://lists.opendev.org/pipermail/service-discuss/2021-November/000298.html ** Adding a lists.openinfra.dev mailman site (fungi 20211130) *** https://review.opendev.org/818826 ** Proxying and caching Ansible Galaxy in our providers (fungi 20211130) *** https://review.opendev.org/818787 ** Gerrit Account cleanups (clarkb 20211130) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml * Open discussion

1 0

Infra Team Meeting Agenda for November 16, 2021
by Clark Boylan 16 Nov '21

16 Nov '21

We will meet with this agenda on November 16, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Gerrit User Summit happening December 2&3 virtually. ** clarkb out next week. Should we skip the meeting on November 23? * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211116) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211116) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Zuul multi scheduler setup (clarkb 20211116) *** Zuul is currently running with two schedulers (zuul01.o.o and zuul02.o.o with zuul02.o.o being "primary") *** Did first rolling restart of schedulers over the weekend. *** Zuul-web should return consistent results now as it talk to ZooKeeper directly. ** User management on our systems (clarkb 20211116) *** Give gerritbot and matrix-gerritbot a shared user: https://review.opendev.org/c/opendev/system-config/+/816769/ *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** Caching openstack/openstack on our DIB images (clarkb 20211116) *** There are semi frequent errors when updating the DIB cache for openstack/openstack *** Seems related to verifying or updating submodule content. *** Should we simply stop caching this repo entirely? It isn't really used for much. * Open discussion

1 0

Team Meeting Agenda for November 9, 2021
by Clark Boylan 09 Nov '21

09 Nov '21

Hello, we will meet on November 9, 2021 at 19:00UTC in #opendev-meeting with this agenda (note the DST change in many parts of the world): == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211109) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211109) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Zuul multi scheduler setup (clarkb 20211109) *** Zuul is currently running with two schedulers (zuul01.o.o and zuul02.o.o with zuul02.o.o being "primary") *** We have tracked down a number of bugs Monday with corvus fixing many of them. *** Overall seems stable enough. *** Note the "flapping" status page can be weird. ** User management on our systems (clarkb 20211109) *** Be explicit about uid/gid ranges: https://review.opendev.org/c/opendev/system-config/+/816869/ **** 0-999 system, 1000-1999 unallocated, 2000-2999 for infra-root users, 3000-9999 host level users, 10k - 64k container users that need uids on the host as well for bind mounts. *** Clean up unused bootstrapping users: https://review.opendev.org/c/opendev/system-config/+/816771 *** Give gerritbot and matrix-gerritbot a shared user: https://review.opendev.org/c/opendev/system-config/+/816769/ *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. * Open discussion

1 0

Infra Team Meeting Agenda for November 2, 2021
by Clark Boylan 01 Nov '21

01 Nov '21

We will meet on November 2, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Gerrit User Summit details arriving soon. I've been told they would be interested to hear from us on how we do automated testing and management of Gerrit. * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211102) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211102) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Fedora 34 test node booting problems (clarkb 20211102) *** Changes to Fedora's kernel packaging broke Xen *** Not yet sure if that may have also somehow broken OVH and iweb. ** Zuul multi scheduler setup (clarkb 20211102) *** Zuul ran with two schedulers for the first time over the last weekend. *** First jobs started by one scheduler and completed by another ran. *** Had to revert due to bugs in caching. *** Expect Zuul restarts as the scale out scheduler work in Zuul progresses. ** FIPS testing in our CI system (clarkb 20211102) *** There is interest in testing various pieces of software against FIPS enabled systems in our Zuul. *** We are not building special FIPS images instead a Zuul role exists to update supported platforms and boot them into FIPS mode. *** Some tests have problems with ephemeral state being lost if the FIPS role runs too late. Then tests fail for unexpected reasons. Worth checking on reboot ordering relative to other test setup if there are problems. * Open discussion

1 0

Team Meeting Agenda for October 26, 2021
by Clark Boylan 26 Oct '21

26 Oct '21

We will meet on October 26, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211026) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211026) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Fedora 34 test node booting problems (clarkb 20211026) *** Changes to Fedora's kernel packaging broke Xen *** Not yet sure if that may have also somehow broken OVH and iweb. ** Begin planning for Gerrit 3.4 upgrade (clarkb 20211026) *** Read through release notes and identify areas of concern. **** https://www.gerritcodereview.com/3.4.html *** Do we need to modify system-config-run-review-3.4 or system-config-upgrade-review jobs to test any specific behaviors? *** Probably worth holding a 3.4 test node and doing a skim for any unexpected behaviors. * Open discussion

1 0

Team Meeting Agenda for October 19, 2021
by Clark Boylan 19 Oct '21

19 Oct '21

We will meet October 19, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** PTG this week. *** OpenDev session Wednesday October 20, 2021 at 14:00 - 16:00 UTC in https://meetpad.opendev.org/oct2021-ptg-opendev *** Zuul session Thursday October 21, 2021 at 14:00 UTC in https://meetpad.opendev.org/zuul-2021-10-21 * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211019) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211019) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Gerrit project renames (clarkb 20211019) *** Overall went well. *** Possible trouble renaming secrets and doing ZK secrets backups? *** We accidentally updated all Gitea projects. Should we just do that by default: https://review.opendev.org/c/opendev/system-config/+/814443 ? ** Improve zuul restarts (frickler 20211014) *** Docs at https://docs.opendev.org/opendev/system-config/latest/zuul.html#restarting-… need updating **** What to restart (scheduler,web,fingergw?) and how (docker restart vs. docker-compose) **** When to run the re-enqueue **** Collecting debug information *** Don't reenqueue periodic jobs * Open discussion Note the PTG is happening, if there are conflicts we can probably focus on the most important items (zuul restart docs and post project rename stuff) and keep the meeting short.

1 0

Gerrit 3.3 upgrade: dashboards and attention sets
by Ian Wienand 13 Oct '21

13 Oct '21

Hello You may have noticed the recent upgrade of Gerrit to version 3.3 on review.opendev.org. We note two things users should be aware of - Dashboard links If your dashboard uses https://review.opendev.org/#/dashboard/... it may no longer work, despite being the documented way to create dashboards. We are currently unclear if this is a bug or a feature [1] but in the mean time, removing the "/#/" (i.e. just use https://review.opendev.org/dashboard/...) makes things work. - Attention sets You have hopefully seen a new section on the main page "Your Turn". This is the Gerrit "Attention Set" feature [2]. This is an optional feature, but we feel it is worth spending some time with it to see how this can fit into our workflows. One thing to be aware of is that adding a vote by default adds the uploader in the attention set. While this makes sense in some models (particuarly non-gating where you are expected to merge yourself), in our environment usually adding a +1 or +2 with no comments is not something that needs the attention of the uploader -- it's either waiting for more review, or Zuul will merge it after gating tests. At the bottom of the modal box where you are inputting your votes you can easily modify the attention set. Look out for it next time you're reviewing! Consider removing others if your comments do not require new attention. If we pollute other people's attention lists with changes that don't actually need their attention the feature will not be useful. So it is worth us all trying to work with the system and be mindful of who and when we add people (we have raised an issue to discuss our workflow with this feature, but it will not be an instant fix [3]). Thanks! OpenDev Admins [1] https://groups.google.com/g/repo-discuss [2] https://gerrit-review.googlesource.com/Documentation/user-attention-set.html [3] https://bugs.chromium.org/p/gerrit/issues/detail?id=15154

1 0

OpenDev PTG Planning
by Clark Boylan 13 Oct '21

13 Oct '21

Hello, The OpenDev team is participating in next week's PTG by holding office hours during a two hour block of time. Join us October 20, 2021 at 14:00 - 16:00 UTC in https://meetpad.opendev.org/oct2021-ptg-opendev to discuss OpenDev related questions, concerns, services and whatever else you might want to talk about. We do ask that you add your topics to our etherpad, https://etherpad.opendev.org/p/oct2021-ptg-opendev, to help us prepare, prioritize, and do simple scheduling during our block of time. See you there, Clark

1 0

Team Meeting Agenda for October 12, 2021
by Clark Boylan 12 Oct '21

12 Oct '21

We will meet on October 12, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** This appears to have the votes. Plan to merge it Thursday 23:00UTC ish if no objections show up. ** Mailman 3 spec https://review.opendev.org/810990 *** Can use more reviews, but looks good to clarkb. * Topics ** Improving OpenDev's CD throughput (clarkb 20211012) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211012) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Scheduling Gerrit project renames (clarkb 20211012) *** October 15 at 18:00UTC. *** Now is the time to make sure projects have their changes in order and that we are ready. *** Updating the process to update project metadata on renamed projects ** Upgrading to Gerrit 3.3 (ianw 20211012) *** Upgrade happened over the weekend. Went very smoothly *** clarkb has been using hashtag:gerrit-3.3 to track post upgrade changes that we want to land related to the upgrade *** Are we ready to drop the 3.2 images? https://review.opendev.org/c/opendev/system-config/+/813074 *** Next up 3.4! * Open discussion

1 0

Team Meeting Agenda for October 5, 2021
by Clark Boylan 05 Oct '21

05 Oct '21

Hello, we will meet with this agenda on October 5, 2021 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack Release tomorrow * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to update to propose running node exporter from their shipped binary. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211005) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211005) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Debian Buster to Bullseye Updates (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/809269 Gitea bullseye update *** https://review.opendev.org/c/opendev/system-config/+/809286 Gerrit bullseye update ** Gitea 1.15.3 Upgrade (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Upgrading to Gerrit 3.3 (ianw 20211005) *** 3.4 is a much bigger jump and will need more care. Clarkb suggests we get to 3.3 soon then plan 3.4. ** Scheduling Gerrit project renames (clarkb 20211005) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion

1 0

Team Meeting Agenda for September 28, 2021
by Clark Boylan 28 Sep '21

28 Sep '21

We will meet on September 28, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Fix for Gerrit logo hosting https://review.opendev.org/c/opendev/system-config/+/811233 ** Gitea 1.15.3 Upgrade (clarkb 20210928) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion

1 0

Infra Team Meeting Agenda for September 21, 2021
by Clark Boylan 21 Sep '21

21 Sep '21

We will meet on September 21, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210921) *** lists.openstack.org is running focal now **** Kernel packages have been pinned. If you update the kernel you need to decompress it. **** Fungi is working on a mailman3 spec. We can decide the best route for replacing the server with that context. ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Stack at https://review.opendev.org/c/opendev/system-config/+/809510 migrates services to the new asset locations. ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 ** Gerrit replication "leaks" (clarkb 20210921) *** We've added a 15 minute network timeout to the gerrit config. Need to restart the server to pick that up. ** InMotion cloud has been scaled up (clarkb 20210921) *** We need to work on tuning the over subscription ratios. * Open discussion

1 0

Mailing lists offline 2021-09-12 for server upgrade
by Jeremy Stanley 14 Sep '21

14 Sep '21

On Sunday, September 12, the OpenDev sysadmins will be performing operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Expect extended outages for message delivery and access to archives between 15:00 UTC and 21:00 UTC, though messages sent to the mailing lists at these sites should queue normally and be delivered automatically after maintenance activities have concluded. If you have any questions, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley

1 2

Team Meeting Agenda for September 14, 2021
by Clark Boylan 13 Sep '21

13 Sep '21

We will meet September 14, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210914) *** lists.openstack.org is running focal now **** Number of manual edits made to services captured in https://review.opendev.org/c/opendev/system-config/+/808570 **** Ran into issues with Xen being able to boot lz4 compressed kernels **** We decompressed the kernel but that still failed **** We installed grub-xen then set up a chainload situation to a xen specific grub preloader via /boot/grub/menu.lst **** This chainload situation may not need a compressed kernel? If it doesn't this server may be viable long term and we can document things. **** We should consider replacing the server to get away from Xen PV weirdness and to maybe use a floating IP which can be moved between upgraded instances in the future. ** Improving OpenDev's CD throughput (clarkb 20210914) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210914) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210914) *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. ** Working with InMotion Friday to increase IP addr allocation and scale up (clarkb 20210914) *** Will try to debug instance launch failures there prior to Friday. ** Scheduling Gerrit project renames (clarkb 20210914) *** We've got a few of these showing up now. We should look at a calendar and pencil in some plans. * Open discussion

1 0

Team meeting agenda for September 7, 2021
by Clark Boylan 07 Sep '21

07 Sep '21

We will meet with this agenda September 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210907) *** lists.openstack.org upgrade scheduled for September 12 beginning 15:00UTC ** Improving OpenDev's CD throughput (clarkb 20210907) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210907) *** 33 conflicts remain. Need to investigate and email their users about planned next steps. ** OpenDev logo hosting (clarkb 20210907) *** https://review.opendev.org/c/opendev/system-config/+/805932/ ready to switch to using the assets container for gitea *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. * Open discussion

1 0

Team Meeting Agenda for August 31, 2021
by Clark Boylan 31 Aug '21

31 Aug '21

Sorry, this is going out a little late this week. We will meet on August 31, 2021 (today) at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Feature Freeze has begun. * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210831) *** lists.kc.io has been upgraded to Focal. Seems to be happy. *** Do resource consumption numbers look different compared to pre upgrade state? *** Time to schedule a lists.o.o upgrade? ** Improving OpenDev's CD throughput (clarkb 20210831) *** Quick check on how things look after our quick and easy updates. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210831) *** Need to delete external ids for the recent batch of acount retirements. ** OpenDev logo hosting (clarkb 20210831) *** https://etherpad.opendev.org/p/logo-locations *** Are we happy with an approach yet? * Open discussion

1 0

Infra Team Meeting Agenda for August 24, 2021
by Clark Boylan 23 Aug '21

23 Aug '21

We will meet on August 24, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Zuul is now on our matrix homeserver at #zuul:opendev.org ** OpenStack Feature Freeze starts next week * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210824) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o *** Will coordinate with Kata to do the upgrade this week hopefully. ** Improving OpenDev's CD throughput (clarkb 20210824) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/805487 Be more specific about when inventory changes force jobs to run. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210824) *** Need to delete external ids for the recent batch of acount retirements. ** Gitea 1.15.0 upgrade (clarkb 20210824) *** This release exists now *** How do we want to handle the opendev logo image hosting for the gerrit theme? * Open discussion

1 0

Team Meeting Agenda for August 17, 2021
by Clark Boylan 17 Aug '21

17 Aug '21

We will meet with this agenda on August 17, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210817) *** Clark was the only nomination. Takes the seat by default. ** Matrix eavesdrop and gerritbot bots (clarkb 20210817) *** Zuul's next steps ** Gitea01 backups to one of two servers are unhappy (clarkb 20210817) *** Reboots didn't help *** On Friday we had a set of servers that backup at the same time with similar issues but to the other backup server. Doesn't seem to be a persistent issue. ** Mailman Ansible and Server Upgrades (clarkb 20210817) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o ** Improving OpenDev's CD throughput (clarkb 20210817) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/804795 Run cloud launcher daily not hourly. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. * Open discussion

1 0

Service Coordinator Nominations and Election
by Clark Boylan 11 Aug '21

11 Aug '21

Hello everyone, It has been about six months since the previous election for the OpenDev Service Coordinator spot. When we did the last round I said we could do about 2 weeks of nomination period then follow that with a week long election if necessary. Considering this the official open call for nominations which will stay open until 23:59 UTC on August 10, 2021. If we need an election that will run from 00:01 UTC August 11, 2021 to 23:59 UTC August 17, 2021 UTC. If anyone else is remotely interested please feel free to step up. I've been doing this for a bit now and would be happy to have someone else take on the role. Let me know if you have questions about what is required. I am also more than happy to help in any transition that happens. Looking ahead to the next election I think we'll run those with nominations January 25, 2022 - February 8, 2022. The election would run February 9, 2022 - February 15, 2022. Clark

3 3

Team meeting agenda for August 10, 2021
by Clark Boylan 10 Aug '21

10 Aug '21

We will meet August 10, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210810) *** Nominations end 23:59 UTC on August 10, 2021. *** That is about 4 hours after the meeting ends. ** review upgrade (ianw 20210810) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Project Renames (clarkb 20210810) *** Need to update docs ** Matrix eavesdrop and gerritbot bots (clarkb 20210810) *** Debugging matrix gerritbot connection problems to review02. *** What are the next steps? ** Gitea01 backups to one of two servers are unhappy (clarkb 20210810) *** Reboots didn't help *** Now lists has the same problem? ** Gitea 1.15.0 prep *** Updating our Gitea management to only use the REST API https://review.opendev.org/c/opendev/system-config/+/803367/ ** Mailman Ansible and Server Upgrades (clarkb 20210810) *** Should we use lists.kc.io as the snapshot image and new test booted node? *** Need to pick this work up again. * Open discussion

1 0

Team Meeting Agenda for August 3, 2021
by Clark Boylan 03 Aug '21

03 Aug '21

Hello, We will meet with this agenda on August 3, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210803) *** Nominations end 23:59 UTC on August 10, 2021. ** review upgrade (ianw 20210803) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Gerrit User Cleanups (clarkb 20210803) *** Next batch of cleanups proposed in files on review02 *** Hoping this is the second to last batch with last batch getting direct cleanups that we can push. ** Project Renames (clarkb 20210803) *** Rename went well *** Need to update docs *** Need to manage ssh known_hosts for review02 users https://review.opendev.org/c/opendev/system-config/+/802922 *** Further testing improvements https://review.opendev.org/c/opendev/system-config/+/803266 ** Matrix eavesdrop and gerritbot bots (clarkb 20210803) *** Deployments are in progress ** Gitea01 backups to one of two servers are unhappy (clarkb 20210727) *** Reboots didn't help ** Gitea 1.15.0 prep *** Upstream no longer compatible with nodejs 16 to build the front end https://review.opendev.org/c/opendev/system-config/+/803265 *** Working through changes to gitea in https://review.opendev.org/c/opendev/system-config/+/803231 ** Mailman Ansible and Server Upgrades (clarkb 20210803) *** Need to fix newlist command invocation for the prod servers https://review.opendev.org/c/opendev/system-config/+/803263 *** Should we use lists.kc.io as the snapshot image and new test booted node? *** Need to pick this work up again. * Open discussion

1 0

Team Meeting Agenda for July 27, 2021
by Clark Boylan 27 Jul '21

27 Jul '21

Hello, We will meet with this agenda on July 27, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210727) *** Missed nomination period start by about 2 weeks *** Suggest we start nomination period now for 2 weeks then have an election after if necessary. ** review upgrade (ianw 20210727) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Gitea01 backups to one of two servers are unhappy (clarkb 20210727) *** Reboots didn't help ** Scheduling project renames (clarkb 20210727) *** July 30 at 15:00 UTC *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/802112 Test the rename playbook. *** Zuul making change that will impact rename procedures for Zuul keys. Keep an eye on that. ** PTG participation (clarkb 20210727) *** Signed up for 14:00 - 16:00 UTC Wednesday October 20, 2021 *** Please register if you plan to participate: https://openinfra-ptg.eventbrite.com/ * Open discussion

1 0

Team Meeting Agenda for July 20, 2021
by Clark Boylan 20 Jul '21

20 Jul '21

We will meet with this agenda on July 20, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** https://review.opendev.org/796156 Supporting communications on our very own Matrix homeserver * Topics ** review upgrade (ianw 20210720) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings ** Gerrit Account cleanup (clarkb 20210720) *** Plan to delete conflicting external ids early next week (gives about 3 weeks of time for people to raise concerns with their accounts). ** Gitea01 backups to one of two servers are unhappy (clarkb 20210720) ** Gitea 1.14.4 upgrade scheduling (clarkb 20210720) *** Wednesday July 21? ** High AFS utilization (clarkb 20210720) *** mirror.debian has only 5GB of headroom to its quota limit *** afs01.dfw.o.o is using more than 90% of its vicepa device *** Discussion around new distros like Alma and Rocky linux *** mirror.yum-puppetlabs is fairly large. Is this still used? *** Do we need to expand vicepa on our two primary fileservers by another terabyte? ** Scheduling project renames (clarkb 20210720) *** We need to schedule this for next week. Clarkb leaning towards July 30. *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/797990 ** PTG participation (clarkb 20210720) *** We need to fill out the PTG survey by July 21st if we intend on participating. *** Last PTG's office hours worked well for the first 2 hours. Maybe we should limit to that this time around? * Open discussion

1 0

review.opendev.org server upgrade 18/19th July 2021
by Ian Wienand 13 Jul '21

13 Jul '21

Hello, Around 2021-07-18 23:00 UTC we plan to provision an upgrade to the review.opendev.org server which hosts our Gerrit instance. We are aware that some users may require firewall updates or similar to access the new server. The new addresses will be: 199.204.45.33 2604:e100:1:0:f816:3eff:fe52:22de We hope the downtime will be limited to a few hours as we migrate and index the data. User-visible changes be should minimal. Due to incompatible schemas between the old hosted database and new database instance we will not be transitioning the table that holds the information about what files you have reviewed (gerrit shows a light-grey "Reviewed" when you have viewed a file). Note this is *not* review comments or code, but just the per-user flag in the UI if you have viewed a file. You can reach admins via email at service-discuss(a)lists.opendev.org or in OFTC #opendev for any further questions or queries. Thank you, -i

1 1

Team Meeting Agenda for July 13, 2021
by Clark Boylan 13 Jul '21

13 Jul '21

Hello, We will meet with this agenda on July 13, 2021 at 19:00UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** https://review.opendev.org/796156 Supporting communications on our very own Matrix homeserver * Topics ** Gerrit Account cleanup (clarkb 20210713) *** External ID cleanup for last batch of conflicts to happen after Gerrit server move. *** efoley has account problems. Batch refs/meta/external-ids update into server move? ** review upgrade (ianw 20210713) *** Scheduled for July 18 at 23:00 UTC *** upgrade checklist ready for review *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 ** Gitea01 backups to one of two servers are unhappy (clarkb 20210713) ** Gitea 1.14.4 upgrade scheduling (clarkb 20210713) ** Scheduling project renames (clarkb 20210713) *** We have penciled in the week after the server upgrade. Will decide on specific time later. *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/797990 * Open discussion

1 0

[UX] I can not find a way to create a new repository using the web UI
by hexagon-recursion＠posteo.net 09 Jul '21

09 Jul '21

Hello. I wanted to give opendev.org a try as an alternative to github for my own free software projects unrelated to OpenStack. Your homepage says that this is allowed and encouraged https://opendev.org/#can-i-host-my-project-on-opendev . I am currently stuck unable to figure out how to create a git repository. What I have done so far: 1. Clicked on "Get Started" 2. Created an Ubuntu One account Where is my "create a git repository" button? -- Andrey Bienkowski

2 1

Team Meeting Agenda for July 6, 2021
by Clark Boylan 06 Jul '21

06 Jul '21

We will meet with this agenda on July 6, 2021 (today) at 19:00UTC in #opendev-meeting on OTFC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Gerrit Account cleanup (clarkb 20210706) *** Clarkb going to retire the next large set of accounts today. External id cleanup to follow in a few weeks. ** review upgrade (ianw 20210706) *** Scheduled for July 18 at 23:00 UTC *** upgrade checklist ready for review *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 ** Draft matrix spec (corvus 20210706) *** https://review.opendev.org/796156 *** Discussion with EMS went well. They don't seem opposed to our idea. *** Next step is to start a trial account. clarkb will coordinate this. ** Gitea01 backups to one of two servers are unhappy (clarkb 20210706) ** Scheduling project renames (clarkb 20210706) *** We have penciled in the week after the server upgrade. Will decide on specific time later. *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/797990 ** Do we want to consider a new meeting time? (clarkb 20210706) *** Current time is too late for frickler and very early for ianw. *** Happy to shift a little later to include ianw at a better time. * Open discussion

1 0

Team meeting agenda for June 29, 2021
by Clark Boylan 28 Jun '21

28 Jun '21

We will meet on June 29, 2021 at 19:00 UTC in #opendev-meeting on OFTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Eavesdrop and Limnoria updates (clarkb 20210629) *** Fixed crontab bug for text to html conversion of logs. This may be the (last?) fix we needed for lag in logs. ** Gerrit Account cleanup (clarkb 20210629) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** review upgrade (ianw 20210628) *** upgrade checklist ready for review *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** date/time for upgrade? 19th july maybe, .au time? ** Server Upgrades (clarkb 20210629) *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. ** Draft matrix spec (corvus 20210629) *** https://review.opendev.org/796156 *** Reaching out to ESM ** Scheduling project renames (clarkb 20210629) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. ** Gitea01 backups to one of two servers are unhappy (clarkb 202106029) * Open discussion

1 0

StarlingX Docs promote job failing
by Camp, MaryX 24 Jun '21

24 Jun '21

Hi, Since the security updates to Zuul today, the StarlingX docs promote job is failing on every review. [1] The last successful promote job ran yesterday. [2] Here is the error: The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'targets' The error appears to be in '/var/lib/zuul/builds/e93712cdb47b4b519d2f9f1202a7f2e1/trusted/project_0/opendev.org/opendev/base-jobs/playbooks/docs/promote.yaml': line 51, column 7, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: target_dict: "{{ afs.targets.master }}" - name: Set target directory if not master ^ here Is anyone else affected by this? thanks, Mary Camp Kelly Services Technical Writer | maryx.camp(a)intel.com<mailto:maryx.camp@intel.com> [1] https://review.opendev.org/q/project:starlingx/docs+status:closed [2] https://review.opendev.org/c/starlingx/docs/+/797756

2 2

FLOSS MOOC ML
by Kendall Nelson 22 Jun '21

22 Jun '21

Hello :) I was wondering what the process was for getting a ML setup? Ildiko and I have been attending regular meetings and advising on the development of a MOOC on FLOSS[1] and they were interested in starting an accompanying ML. I mentioned opendev maybe being a place for that and explained that they could be setup as admins to manage the ML, etc. I am all ears about what to do to get started! -Kendall Nelson (diablo_rojo) [1] https://gitlab.com/mooc-floss/mooc-floss

3 3

Team Meeting Agenda for June 15, 2021
by Clark Boylan 15 Jun '21

15 Jun '21

We will meet on June 15, 2021 at 19:00UTC in #opendev-meeting on OFTC with this agenda: == Agenda for next meeting == * Announcements ** clarkb out next week. Will need a volunteer meeting chair or we can skip. * Actions from last meeting * Specs approval * Topics ** Eavesdrop and Limnoria updates (clarkb 20210615) *** Fixed a bug upstream when joining many channels *** Still running a fork to include setuptools updates ** Gerrit Account cleanup (clarkb 20210615) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Server Upgrades (clarkb 20210615) *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. *** Review02 getting into shape ** Cleaning up PPC packages in our CentOS 8 mirror (clarkb 20210615) ** Draft matrix spec (corvus 20210615) *** https://review.opendev.org/796156 ** Scheduling project renames (clarkb 20210615) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting Agenda for June 8, 2021
by Clark Boylan 07 Jun '21

07 Jun '21

We will meet with this agenda on June 8, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Switch to OFTC now largely complete (clarkb 20210608) *** https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite *** https://review.opendev.org/c/openstack/ptgbot/+/793792 Update PTGbot for OFTC *** Is PGTbot python3 capable? We also need to register a nick for it. ** Gerrit Account cleanup (clarkb 20210608) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210608) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. ** Cleaning up PPC packages in our CentOS 8 mirror (clarkb 20210608) ** Scheduling project renames (clarkb 20210608) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting Agenda for June 1, 2021
by Clark Boylan 01 Jun '21

01 Jun '21

Hello everyone. We will meet at 19:00 UTC on June 1, 2021 (today) in #opendev-meeting on OFTC. Apologies for getting this agenda out late. Yesterday was a holiday and I ended up being distracted. Here is the agenda we will use today: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Switch to OFTC now largely complete (clarkb 20210601) *** https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite ** Gerrit Account cleanup (clarkb 20210601) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Refreshing non LE certs that expire in just under a month (clarkb 20210601) *** Wiki is the last remaining service needing an update. Clarkb on it. ** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210601) *** https://review.opendev.org/c/openstack/project-config/+/785769 ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210601) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot. ** Scheduling project renames (clarkb 20210601) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Reevaluating choice of IRC network for our service bots
by Jeremy Stanley 30 May '21

30 May '21

As most people reading this are likely aware, there has been a recent change in management for the Freenode IRC network. According to public statements[0], the former staff have all left and the network is in entirely new hands. Setting aside arguments about which parties are in the wrong, further stability of the service we provide is once again brought into question. We've seen evidence of (much) lower levels of abuse mitigation over the past few days, with a marked increase in registered accounts engaging in harassment over extended periods of time across our channels without getting banned from the network. Due to disagreements between Freenode and other large IRC networks and frequent related attacks against Freenode's servers, our communities have experienced many extended periods of instability there. In order to insure against a potential collapse of the network, the OpenDev Collaboratory (and the OpenStack Project Infrastructure before it) has for many years maintained a sort of evacuation plan: For the multitude of channels overseen by OpenDev, we also keep equivalent channel registrations on the Open and Free Technology Community (OFTC) IRC network[1], and have communicated with its operators on multiple occasions about their willingness and ability to host our communities there should the need arise. This week's events are, unfortunately, not an isolated incident. The systems administrators for OpenDev have encouraged a move to OFTC on multiple occasions, with the first formal proposal in March 2014[2]. In prior cases, the impact to the community for such a switch outweighed the perceived benefits, but as we've heard growing displeasure with Freenode from representatives of many of the projects we serve, it is time once more to revisit whether we should enact our longstanding evacuation plans. Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services. If projects were to move their channels to OFTC, the transition would not be seamless. In particular, differences in authentication (no SASL support, but you can authenticate your connection with a certificate[3] if you don't want to identify to the NickServ after connecting), permissions model (OFTC has coarse-grained RBAC designed to reduce channel mismanagement), and NickServ and ChanServ command syntax are among the challenges they're likely to face. The same IRC nicks may also not be available in some cases, though due to the generally smaller size of OFTC compared to Freenode this hopefully won't come up for too many users. On a positive note, we may be able to go back to not requiring nick registration just to join channels, easing onboarding for newcomers. If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter. [0] https://fuchsnet.ch/freenode-resign-letter.txt [1] https://www.oftc.net/ [2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html [3] https://www.oftc.net/NickServ/CertFP/ [4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html [6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021… -- Jeremy Stanley

6 14

Team Meeting Agenda for May 25, 2021
by Clark Boylan 24 May '21

24 May '21

We will meet May 25, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Refreshing non LE certs that expire in just under a month (clarkb 20210525) *** Shutdown/remove: ask.o.o **** https://review.opendev.org/c/opendev/system-config/+/792789 *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard **** https://review.opendev.org/c/opendev/system-config/+/792708 *** Meeting with foundation to discuss: openstackid and openstackid-dev **** They are interested in taking on the hosting but due to timing we probably need to provision a cert for it. Will look into LE for this as well. ** Potentially Migrating away from Freenode to OFTC (clarkb 20210525) *** Freenode is under new management *** Freenode policies are changing *** http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html ** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210525) ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210525) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot. ** Scheduling project renames (clarkb 20210525) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Updating configuration management for our listservs
by Clark Boylan 19 May '21

19 May '21

Hello everyone, Just a quick update that we are updating the configuration management for our listservs. This email serves as a quick sanity check that everything is continuing to function as expected. If you notice anything odd please let us know. Clark

1 0

Team meeting agenda for May 18, 2021
by Clark Boylan 17 May '21

17 May '21

We will meet with this agenda on May 18, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. *** Update our base job's nodeset **** https://review.opendev.org/789098 * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210518) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** https://review.opendev.org/c/opendev/system-config/+/789622 ** Refreshing non LE certs that expire in just under a month (clarkb 20210518) *** Shutdown/remove: ask.o.o *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard *** Meeting with foundation to discuss: openstackid and openstackid-dev ** Cleanup of too small swap setups on newer servers (clarkb 20210518) *** make_swap.sh was updated to accidentally limit swap size to 8MB when we wanted an 8GB limit ** Removing registration requirement from our IRC channels (clarkb 20210518) *** TheJulia asks if we think this is still necessary. *** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel *** https://review.opendev.org/c/openstack/project-config/+/791818 ** "Toggle CI" in Gerrit web UI (rosmaita 20210518) *** I thought this was on the [https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes 3.2 post-upgrade etherpad] but I can't find it. We used to have this button in the old web interface *** The Cinder team is looking for a solution to Change Log pollution from our 3rd party CI systems **** example: https://review.opendev.org/c/openstack/cinder/+/790796/ *** actual reviewer comments get lost in the Change Log *** the "Only comments" toggle hides all the "Added to cc:" lines, which is nice. It also hides the Zuul "build succeeded/failed" comments ... what do we need to do so this will work for other CI responses? **** The "Only Comments" toggle and the "Zuul Summary" tab are what you should be using. To filter with only comments the CI system needs to set an autogenerated tag on its comments. Zuul does this unconditionally if you set it up to report over http. For the Zuul Summary tab the CI system must have a Full Name in gerrit of "foo name CI" and the result format must match Zuul's. ** Scheduling project renames (clarkb 20210518) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting on May 11, 2021 Cancelled
by Clark Boylan 10 May '21

10 May '21

Hello everyone, Last week I mentioned that I won't be able to run the meeting tomorrow and others seemed to think skipping it this week would be fine. I have not heard any desire to have a meeting since, so I'll make it official and cancel the meeting now. We will see you next week on May 18, 2021 at 19:00UTC in #opendev-meeting. Clark

1 0

[devstack][infra] POST_FAILURE on export-devstack-journal : Export journal
by Martin Kopec 09 May '21

09 May '21

Hi, one of our jobs (python-tempestconf project) is frequently failing with POST_FAILURE [1] during the following task: export-devstack-journal : Export journal I'm bringing this to a broader audience as we're not sure where exactly the issue might be. Did you encounter a similar issue lately or in the past? [1] https://zuul.opendev.org/t/openstack/builds?job_name=python-tempestconf-tem… Thanks for any advice, -- Martin Kopec

4 10

Re: [service-announce] Switching default nodeset in base job to ubuntu-focal
by Sorin Sbarnea 05 May '21

05 May '21

I support the move as I am aware of several issues that are sorted (or simplified) by using a newer release. It worth mentioning that GHA has a nice way to announce pending upgrade of their “nodeset” upgrades: they include a visible warning on each executed job that use a node that is pending a change, something like: “This job is using ubuntu-latest which will soon be switched from x to y. Use specific labels if you want to avoid such a switch.” They do no mention the exact date of the switch but apparently these warnings to stay on for at least one month. On Tue, 4 May 2021 at 22:49, Jeremy Stanley <fungi(a)yuggoth.org> wrote: > The OpenDev Collaboratory is normally somewhat conservative about > when it changes its Zuul node "default" (that is, the nodeset in its > base job). It's been ubuntu-bionic since at least July of 2019 when > we first moved our base job into its own repository, and > ubuntu-focal nodes have been available in our deployment since > before Ubuntu's 20.04 LTS was officially released more than a year > ago. All that is to say, updating our default is overdue. > > On Tuesday, 2021-05-18 (two weeks from today) we'll approve > https://review.opendev.org/789098 to update the nodeset to > ubuntu-focal for any jobs which don't specify their own or inherit > from another job which does. We've already merged a change which > updates our base-test job to reflect this, so you can test either by > explicitly setting your jobs to use an ubuntu-focal nodeset or to > inherit from base-test (though we don't recommend merging changes > for the latter, just rely on speculative execution to identify > problems which you may encounter when the default changes). > > Please reply to the service-discuss mailing list with any questions > or concerns. You should also feel free to leave comments on the > change linked above. > -- > Jeremy Stanley > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce > -- -- /zbr

1 0

Team Meeting Agenda for May 4, 2021
by Clark Boylan 03 May '21

03 May '21

We will meet at 19:00 UTC in #opendev-meeting on May 4, 2021 with this agenda: == Agenda for next meeting == * Announcements ** Clark out during May 11 meeting time. Need a volunteer chair or can cancel that meeting. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. *** Update our base job's nodeset **** https://review.opendev.org/789098 *** Gerrit global config simplification **** https://review.opendev.org/789383 * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210504) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Zuul scheduler is last remaining server that needs an upgrade in the zuul cluster. ** openEuler patches (ianw 20210504) *** TC level, or being too officious ** InMotion Cloud reorganization (clarkb 20210504) *** Currently limited by number of IP addresses available *** We could deploy an executor in the cloud then set up test nodes without direct external access *** Any held nodes would need floating IPs attached to them to be accessible ** Removing registration requirement from our IRC channels (clarkb 20210504) *** TheJulia asks if we think this is still necessary. *** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel ** Switching artifact signing keys from RSA to ECC *** https://review.opendev.org/789062 * Open discussion

1 0

Team Meeting Agenda for April 27, 2021
by Clark Boylan 27 Apr '21

27 Apr '21

We will meet on April 27, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210427) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. **** https://etherpad.opendev.org/p/opendev-zookeeper-upgrade-2021 discussion of options for zk upgrades ** survey.openstack.org (clarkb 20210427) *** We're getting friendly reminders that this SSL cert is about to expire. Would be good to cleanup. ** Debian Bullseye Images (clarkb 20210427) *** Need some DIB updates to hack around Debian versioning and Ansible's factorizing of that info. ** Minor git-review release to support --no-thin (clarkb 202104027) * Open discussion

1 0

Debian bullsye image Ansible detection
by Ian Wienand 23 Apr '21

23 Apr '21

Hello, In short, Ansible reports "n/a" for ansible_distribution_release on our new bullseye nodes. This screws up our mirror setup. This has turned into quite an adventure. Currently, Debian is frozen to create the "bullseye" release. This means that "bullseye" is really an alias for "testing", that will turn into the release after the freeze period. So currently Debian bullseye reports itself in /etc/debian_version or /etc/os-release as "bullseye/sid". This sort of makes sense if you consider that you don't commit things to "testing" directly, they go into unstable ("sid") and then migrate after a period of stability. So you can't have "base-files" package in bullseye that hasn't gone through unstable/sid. You can read "bullseye/sid" as "we've chosen the name bullseye and packages going through unstable are destined for it". Now, you might see a problem in that "unstable" and "bullseye" (testing) now both report themselves in these version files as the same thing (because the unstable packages that provide them move into testing). "lsb_release -c" tries to be a bit smart about this, and looks at the output of "apt-cache policy" to try and see if you are actually pulling the .deb files from a bullseye repo or an unstable one. Interestingly, this relies on a "Label" being present in the mirror release files. Since we use reprepro to build our own mirrors, we do not have this (and why nobody else who doens't use our mirrors seems to notice this problem). A fix is proposed with https://review.opendev.org/c/opendev/system-config/+/787661 So "lsb_release -c" doesn't report anything, leaving Ansible in the dark as to what repo it uses. When "lsb_release -c" doesn't return anything helpful, Ansible tries to do it's own parsing of the release files. I started hacking on these, but the point raised in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845651 gave me pause. It is a fair point that you can not really know if you're on bullseye or sid by examining these files. N/A is probably actually the correct answer from Ansible's POV. Anyway, that is https://github.com/ianw/ansible/commit/847817a82ed86b5f39a4ccc3ffbff0e0cd63… Now, even more annoyingly, setting the label in our mirrors may not be sufficient for "lsb_release -c" to work on our images, because we have cleared out the apt repositories. You would need to run "apt-get update" before Ansible tries to run "lsb_release" to populate it's facts. Now the problem is that we're trying to use Ansible's fact about the distro name to setup apt to point to our mirrors -- so we can't apt-get update before we have that written out! Classic chicken and egg. The only other idea I have is to hack dib/early setup overwrite /etc/debian_version with "11.0" so that we look like the upcoming release has already been done. "lsb_release -c" will then report "bullsye". However, there is some possibility this will confuse other things, as this release technically hasn't been done. I've proposed that with https://review.opendev.org/c/openstack/diskimage-builder/+/787665 I'm open to suggestions! -i

3 4

Join OpenDev at the Project Teams Gathering
by Clark Boylan 13 Apr '21

13 Apr '21

The PTG is next week, and OpenDev is participating alongside the OpenStack TaCT SIG. We are going to try something a bit different this time around, which is to treat the time as office hours rather than time for our own projects. We will be meeting on April 22 from 14:00 - 16:00 UTC and 22:00 - 00:00 UTC in https://meetpad.opendev.org/apr2021-ptg-opendev. Join us if you would like to: * Start contributing to either OpenDev or the TaCT sig. * Debug a particular job problem. * Learn how to write and review Zuul jobs and related configs. * Learn about specific services or how they are deployed. * And anything else related to OpenDev and our project infrastructure. Feel free to add your topics and suggest preferred times for those topics here: https://etherpad.opendev.org/p/apr2021-ptg-opendev. This etherpad corresponds to the document that will be auto loaded in our meetpad room above. I will also be around next week and will try to keep a flexible schedule. Feel free to reach out if you would like us to join discussions as they happen. See you there, Clark

1 0

Team Meeting Agenda for April 13, 2021
by Clark Boylan 13 Apr '21

13 Apr '21

We will meet with this agenda on April 13, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack completing release April 14. Airship 2.0 doesn't seem to exist yet so will assume they are still working on it. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit upgrade to 3.2.8 **** https://review.opendev.org/c/opendev/system-config/+/784152 *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** ~224 accounts were cleaned up. Next batch of ~56 has been started. Will clean their external IDs after letting the retired users sit for a few days. ***** Email sent to two Third Party CI groups about correcting external id conflicts among their accounts. These accounts will not be retired (for the most part). *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210413) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. **** https://etherpad.opendev.org/p/opendev-zookeeper-upgrade-2021 discussion of options for zk upgrades ** planet.openstack.org (ianw 20210413) *** Strong preference from clarkb to retire it *** Superuser appears to be a major blog showing up there as well as a couple of others. Maybe we reach out to them and double check they don't want to help? (fungi and clarkb reached out to Superuser and they seem ok) ** survey.openstack.org (clarkb 20210413) *** Can we go ahead and clean this service up? I don't think it ever got much use (maybe one or two surveys total). ** docs-old volume cleanup (ianw 20210413) *** We were going to double check with Ajaeger if we can then proceed to cleanup if no one had a reason to keep it. ** PTG Planning (clarkb 20210413) *** Next PTG April 19-23 **** Thursday April 22 1400-1600UTC and 2200-0000UTC * Open discussion

1 0

Recent nodepool label changes
by James E. Blair 11 Apr '21

11 Apr '21

Hi, I recently spent some time trying to figure out why a job worked as expected during one run and then failed due to limited memory on the following run. It turns out that back in February this change was merged on an emergency basis, which caused us to start occasionally providing nodes with 32G of ram instead of the typical 8G: https://review.opendev.org/773710 Nodepool labels are designed to represent the combination of an image and set of resources. To the best of our ability, the images and resources they provide should be consistent across different cloud providers. That's why we use DIB to create consistent images and that's why we use "-expanded" labels to request nodes with additional memory. It's also the case that when we add new clouds, we generally try to benchmark performance and adjust flavors as needed. Unfortunately, providing such disparate resources under the same Nodepool labels makes it impossible for job authors to reliably design jobs. To be clear, it's fine to provide resources of varying size, we just need to use different Nodepool labels for them so that job authors get what they're asking for. The last time we were in this position, we updated our Nodepool images to add the mem= Linux kernel command line parameter in order to limit the total available RAM. I suspect that is still possible, but due to the explosion of images and flavors, doing so will be considerably more difficult this time. We now also have the ability to reboot nodes in jobs after they come online, but doing that would add additional run time for every job. I believe we need to address this. Despite the additional work, it seems like the "mem=" approach is our best bet; unless anyone has other ideas? -Jim

4 5

Next steps with new review server
by Ian Wienand 08 Apr '21

08 Apr '21

Hi, We have a large server provided by Vexxhost up and running in a staging capacity to replace the current server at review02.openstack.org. I have started to track some things at [1] There's a couple of things: 1) Production database Currently, we use a hosted db. Since NoteDB this only stores review seen flags. We've been told that other sites treat this data as ephemeral; they use a H2 db on disk and don't worry about backing up or restoring across upgrades. I have proposed storing this in a mariadb sibling container with [2]. We know how to admin, backup and restore that. That would be my preference, but I'm not terribly fussed. If I could request some reviews on that; I'll take +2's as a sign we should use a container, otherwise we can leave it with H2 it has now. 2) IPv6 issues We've seen a couple of cases that are looking increasingly like stray RA's are some how assigning extra addresses, similar to [1]. Our mirror in the same region has managed to acquire 50+ default routes somehow. It seems like inbound traffic keeps working (why we haven't seen issues with other production servers?). But I feel like it's a little bit troubling to have undiagnosed before we switch our major service to it. I'm running some tracing, trying to at least catch a stray RA while the server is quite, in the etherpad. But suggestions here are welcome. -i [1] https://etherpad.opendev.org/p/gerrit-upgrade-2021 [2] https://review.opendev.org/c/opendev/system-config/+/775961 [3] https://launchpad.net/bugs/1844712

3 4

Team Meeting Agenda for April 6, 2021
by Clark Boylan 06 Apr '21

06 Apr '21

We will meet with this agenda on April 6, 2021 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack producing final RCs this week. Airship also working on a release. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit upgrade to 3.2.8 **** https://review.opendev.org/c/opendev/system-config/+/784152 *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Cleaning external IDs for the last batch of retired users. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210406) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. **** https://etherpad.opendev.org/p/opendev-zookeeper-upgrade-2021 discussion of options for zk upgrades ** PTG Planning (ianw 20210406) *** Next PTG April 19-23 *** Clarkb filled out the survey and requested a few hours for us. Likely to be spent in more office hours type setup. **** Thursday April 22 1400-1600UTC and 2200-0000UTC ** docs-old volume cleanup (ianw 20210406) *** We were going to double check with Ajaeger if we can then proceed to cleanup if no one had a reason to keep it. ** planet.openstack.org (ianw 20210406) *** Strong preference from clarkb to retire it *** Superuser appears to be a major blog showing up there as well as a couple of others. Maybe we reach out to them and double check they don't want to help? (fungi and clarkb reached out to Superuser and they seem ok. ** tarballs ORD replication (ianw 20210406) *** This has been done. Other than long initial sync is this happy day to day? * Open discussion

1 0

Team Meeting Agenda for March 30, 2021
by Clark Boylan 29 Mar '21

29 Mar '21

We will meet with this agenda March 30, 2021 in #opendev at 19:00UTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210330) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. ** Deploy a new refstack.openstack.org server (ianw 20210330) *** https://review.opendev.org/c/opendev/system-config/+/781593 ** PTG Planning (ianw 20210330) *** Next PTG April 19-23 *** Clarkb filled out the survey and requested a few hours for us. Likely to be spent in more office hours type setup. **** Thursday April 22 1400-1600UTC and 2200-0000UTC ** docs-old volume cleanup (ianw 20210330) *** anyone think this is necessary? *** docs-dev? ** planet.openstack.org (ianw 20210330) *** there is no maintained aggregators any more *** stick current venus/planet in a python 2.7 container and forget about it *** retire it ** tarballs ORD replication (ianw 20210330) *** only things across regions are docs & airship *** tarballs is probably the only other thing worth having a out-of-region copy for *** https://etherpad.opendev.org/p/gjzssFmxw48Nn3_SBVo6 * Open discussion

1 0

Retirement of the Openstack/Opendev Asterisk service
by Ian Wienand 24 Mar '21

24 Mar '21

Hello, As part of ongoing service upgrades, we are proposing the retirement of the Asterisk-based SIP/voice conferencing tool [1]. The meetpad instance at https://meetpad.opendev.org/ has proven popular for recent events and provides better services for collaborative meetings. Please reach out if this service is important to you. If it is something you would like to maintain, it will require porting to our current Ansible and container-based infrastructure (there will be plenty of admins happy to help guide you). Otherwise we'll propose changes to remove the configuration soon. Thanks, -i [1] https://docs.opendev.org/opendev/system-config/latest/asterisk.html

1 0

Team Meeting Agenda for March 23, 2021
by Ian Wienand 22 Mar '21

22 Mar '21

We will meet on March 16, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210316) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers, executors and launchers done. Only remaining is scheduler? ** Deploy a new refstack.openstack.org server (ianw 20210323) *** https://review.opendev.org/c/opendev/system-config/+/781593 ** PTG Planning (ianw 20210323) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). * Open discussion

1 0

Team Meeting Agenda for March 16, 2021
by Clark Boylan 15 Mar '21

15 Mar '21

We will meet on March 16, 2021 at 19:00 UTC (note DST changes if you live in North America) in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out March 23. Ianw has volunteered to chair the meeting on the 23rd. ** DST change has happened in North America. EU and others follow in a few weeks. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210316) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers and executors done. Nodepool launchers in progress now. *** Ianw working on kerberos server updates. ** Deploy a new refstack.openstack.org server (kopecmartin 20210316) *** Ready for testing? ** PTG Planning (clarkb 20210316) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). ** review server upgrade (ianw 20210316) *** Initial spec https://review.opendev.org/c/opendev/infra-specs/+/780478 * Open discussion

1 0

Team Meeting Agenda for March 9, 2021
by Clark Boylan 09 Mar '21

09 Mar '21

We will meet March 9, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out March 23. Will need volunteer to run the meeting or we can skip it. ** DST change happens for those of us in North America this weekend. EU and others follow in a few weeks. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Identify accounts that cannot be used because they have no valid method of login (no username, and no openids) ***** Identify accounts that are unlikely to be used anymore based on activity and more forcefully retire those to fix these errors. (We can always undo specific updates to these accounts if necessary) ***** Work with remaining accounts to figure out how to best resolve the account conflicts. This may take some time. ***** https://review.opendev.org/c/opendev/system-config/+/777846 Collecting scripting efforts here ***** Inactive accounts have been cleaned up. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210309) *** Upgrading servers to Bionic then Focal next. *** New third db server for proper quorum. ** Bup and Borg Backups (clarkb 20210309) *** gitea sql db backup issues. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210309) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers and executors largely done. Next up Nodepool launchers. ** Deploy a new refstack.openstack.org server (kopecmartin 20210309) *** Ready for testing? ** Bridge disk space (clarkb 20210309) *** This is related to stevedore caching entrypoints *** We have upgraded stevedore and set config to prevent it from creating new cache files. ** PTG Planning (clarkb 20210309) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). * Open discussion

1 0

Can some projects configure auto addition of reviewers to changes?
by Sorin Sbarnea 05 Mar '21

05 Mar '21

As I observed that proposed changes can easily be ignored for months if the original submitter did not manually added anyone or missed to do the footwork of chasing cores on irc, I wonder if it would not be possible for *some* projects to enable a feature to auto-add reviewers. On the dark side of the world I seen the "codeowners <https://docs.github.com/en/github/creating-cloning-and-archiving-repositori…>" features being quite successful in assigning reviewers, as a simple file can define the pool of potential reviewers. Based on how the repository is configured it does randomly pick a limited number of candidates. Github mentions in their release notes from 2017 that their inspiration for that feature was the OWNERS file used by chromium team (gerrit users). I also seen that Wikimedia experimented something <https://phabricator.wikimedia.org/phame/live/1/post/139/gerrit_now_automati…> in that area, apparently there is a wiki page where people can mark them as available to be picked. The watched project feature in Gerrit is useless for this purpose as it does not randomly pick some of a pool of possible candidates. It may be practical to use if there are projects with only 2-3 core reviewers. There are two aspects I am looking to address, at least partially: A) Improve random external contributors first-contribution experience - right now, even if they propose a perfectly valid change, there is high chance it will not get noticed. As you can imagine this lowers the chance they would try again, or to become an active contributor. B) Spread the review burden on cores and avoid "punishing" those that are more active from being assaulted by review requests. The current secret source for getting reviews on a project where you do not know anyone involves looking at recent changes to spot people that did reviews or at least have core powers and try to contact those. Even if this works or not, it puts extra pressure on active people. That is why I would see an automatic round-robin approach more appropriate. As some may no longer be involved with the project, this could prove to be an opportunity to revise the core list, or to gently remove yourself from the list of cores. C) How to deter people from adding reviewers before CI jobs report green? I have no idea how to educate people not to add reviewers to changes that are not yet green. I know there are valid cases where this is needed, but likely is more like on less than one 1:5. As no hard limit would work, maybe there is a way to display some kind of message in the UI, advising user not to assign reviewers until CI reports green. The same issue applies even to automatic reviewer assignment, which should not happen right after a review is created. Ideally this should happen when it passed CI, but it could also be ok if it would happen when a review is moved from WIP to ready-for-review. I personally use a default where any change I create is marked as WIP and this default can also be changed at server level if we want. -- /zbr

3 2

Meeting Agenda for March 2, 2021
by Clark Boylan 02 Mar '21

02 Mar '21

We will meet on March 2, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Identify accounts that are inactive and can be more forcefully retired. Retire these to fix those errors. ***** Identify accounts that are unlikely to be used anymore based on activity and more forcefully retire those to fix these errors. (We can always undo specific updates to these accounts if necessary) ***** Work with remaining accounts to figure out how to best resolve the account conflicts. This may take some time. ***** https://review.opendev.org/c/opendev/system-config/+/777846 Collecting scripting efforts here *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210302) *** Upgrading servers to Bionic then Focal next. *** New third db server for proper quorum. ** Bup and Borg Backups (clarkb 20210302) *** gitea sql db backup issues. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210302) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers are done. Executors in progress. ** Deploy a new refstack.openstack.org server (kopecmartin 20210302) *** Ready for testing? ** Bridge disk space (clarkb 20210302) *** This appears at least partially related to ansible and python caching. Should we just clear those caches then profile them? * Open discussion

1 0

Critical Vulnerability Report
by Divya Singh 28 Feb '21

28 Feb '21

HEY SECURITY TEAM, I'm Security Researcher I have found a critical vulnerability at one of your domain that is cve-2019-15043 which can led to DDOS attack and can make system go down by grafana snapshot instance Vuln url: https://grafana.opendev.org/api/snapshots <https://grafana.quiknode.io/api/snapshots> POC: root@kali:/home/kali# curl -s XPOST https://grafana.opendev.org/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {}}' | json_pp { "deleteKey" : "6mYFALwQmmpImHeKS30XtFw8ogmoHaSm", "deleteUrl" : " http://localhost:3000/api/snapshots-delete/6mYFALwQmmpImHeKS30XtFw8ogmoHaSm ", "key" : "91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO", "url" : " http://localhost:3000/dashboard/snapshot/91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO" } for more reference - https://aaron-hoffmann.com/blog/cve-2019-15043/ Fix it to latest grafana instance Best Regards, Divya Singh - @Dgirlwhohacks

3 7

Meeting Agenda for February 23, 2021
by Clark Boylan 22 Feb '21

22 Feb '21

We will meet with this agenda on February 23, 2021, at 19:00UTC in #opendev-meeting. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Fungi suggests we simply identify the active accounts then retire the rest for simplicity and speed. Clarkb likes this idea. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210223) *** Upgrading servers to Bionic then Focal next. ** Bup and Borg Backups (clarkb 20210223) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210223) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. ** Deploy a new refstack.openstack.org server (kopecmartin 20210223) *** Ready for testing? ** Bridge disk space (clarkb 20210223) *** Our ansible logging is consuming a fair bit but user homedirs and /opt are other major consumers. * Open discussion

1 0

Meeting Agenda for February 16, 2021
by Clark Boylan 16 Feb '21

16 Feb '21

Hello, we will meet February 16, 2021 at 19:00UTC in #opendev-meeting on freenode with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/775287 **** https://review.opendev.org/c/opendev/system-config/+/773807 *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210216) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210216) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210216) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210216) *** Ready for testing? ** opendev.org not reachable via IPv6 from some ISPs (frickler 20210215) *** Caused by missing/inconsistent IRR records for the networks that vexxhost uses *** Pinged mnaser a couple of times but no progress so far *** Added here for increased visibility and tracking * Open discussion Sorry for the delay in sending this out. I had meant to get it out yesterday and had even prepped it to go out but then got distracted by some server updates and this was forgotten.

1 0

Meeting Agenda for February 9, 2021
by Clark Boylan 08 Feb '21

08 Feb '21

We will meet in #opendev-meeting at 19:00 UTC on February 9, 2021 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. * General topics ** OpenAFS cluster status (clarkb 20210209) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210209) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210209) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210209) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** ianw offered to help with the deployment side *** kopecmartin will help with testing the new instance ** Meetpad was not functional late last week. Seems fine now (clarkb 2020210209) * Open discussion

1 0

Meeting Agenda for February 2, 2020
by Clark Boylan 01 Feb '21

01 Feb '21

We will meet February 2, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html Only nomination appears to be from Clark. *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 81 accounts with preferred emails missing external ids have been fixed. **** We have 28 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210202) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210202) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210202) *** Enable Xenial -> Bionic/Focal system upgrades *** Clarkb to write up an etherpad that captures the rough TODO list. ** Deploy a new refstack.openstack.org server (kopecmartin 20210202) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** Help with deploying the server is needed *** kopecmartin will help with testing the new instance * Open discussion

1 0

Service Coordinator Nominations
by Clark Boylan 31 Jan '21

31 Jan '21

Hello everyone, It has been pointed out that in our governance document we stated we would elect a service coordinator every 6 months. I filled this role at the beginning of May 2020 which means we are overdue for a new election. The beginning of the year is always a bit slow so I figure we can give folks until January 31st at 23:59UTC to nominate themselves. Then if we need an election we can do that during the week from February 1st to February 8 at 23:59UTC. I've now been in this role (either as Service Coordinator or Infra PTL) a while. I think it would excellent to rotate in some new eyeballs and perspectives if possible. If you have interest but also have questions feel free to reach out. In order to avoid this slipping by in the future I'm going to go ahead and write down the dates for the next nomination period and election now: * Nominations July 12, 20201 - August 1, 2021 at 23:59 UTC * Election if necessary August 2 - August 9, 2021 at 23:59 UTC I think I got the math right. If I've got it wrong or you have other concerns feel free to bring them up on this thread and we can sort things out. I'm trying to balance giving people enough time to show interest at the beginning of the year while also not letting this languish for longer than it needs to. Then doing math for what 6 months from now will look like. Clark

1 1

Deploy a new refstack.openstack.org instance
by Martin Kopec 28 Jan '21

28 Jan '21

Hi, I'd like to ask for help with deploying a new refstack.openstack.org instance [1]. I also added this topic to the weekly meeting agenda [2]. I'll help with testing the new instance before we put it to the production. [1] https://review.opendev.org/c/opendev/system-config/+/705258 [2] https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Weekly_Project_In… Thank you for your help. Regards, -- Martin Kopec Software Quality Engineer Red Hat EMEA <https://www.redhat.com>

2 1

Fwd: [All][StoryBoard] Angular.js Alternatives
by Kendall Nelson 26 Jan '21

26 Jan '21

Hello Everyone! The StoryBoard team is looking at alternatives to Angular.js since its going end of life. After some research, we've boiled all the options down to two possibilities: Vue.js or React.js I am diving more deeply into researching those two options this week, but any opinions or feedback on your experiences with either of them would be helpful! Here is the etherpad with our research so far[3]. Feel free to add opinions there or in response to this thread! -Kendall Nelson (diablo_rojo) & The StoryBoard Team [1] https://vuejs.org/ [2] https://reactjs.org/ [3] https://etherpad.opendev.org/p/replace-angularjs-storyboard-research

5 5

Meeting agenda for January 26, 2021
by Clark Boylan 26 Jan '21

26 Jan '21

We will meet on January 26, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html less than a week remaining. *** Gerrit account and group inconsistencies **** ~1 group has a subgroup membership loop. **** We have ~109 accounts with preferred email addresses that don't have a matching external id ***** Some of these are inactive accounts that we can more properly retire, others are accounts that can be set to inactive due to lack of external ids, some have email addresses we might convert to preferred email address, and others will need proper digging. **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210126) *** https://review.opendev.org/c/opendev/system-config/+/771521 properly install new openafs on xenial openafs clients. *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210126) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210126) *** Enable Xenial -> Bionic/Focal system upgrades *** Clarkb to write up an etherpad that captures the rough TODO list. ** two-review rule impact on low-activity projects (zbr 20210126) *** projects like git-review rely on infra team and delay changes *** Properly setting expectations for what our small team is able to pay attention to *** Communicating the topics that are consuming our time day to day **** We update the priority efforts documentation and possibly link to a storyboard dashboard there that is populated with the current fires. * Open discussion

1 0

Requesting help with OpenStack configuration files
by Bessghaier, Narjes 19 Jan '21

19 Jan '21

Dear OpenStack team, I’m a Ph.D. student in software engineering at the ETS Montreal, University of Quebec working on the quality and configuration of web-based software systems. I’m particularly interested in analyzing configuration files from different OpenStack files. One of the main challenges I am currently facing is the proper identification of configuration files. I’m mostly confused between the python files used for production and the python files used for configuration. I am kindly requesting your precious help with the following questions: 1- How to distinguish between python files used for configuration and python files used for production? It will be very helpful if there are some configuration-based patterns (eg, textual patterns or expressions) that we can find in python files to help us distinguish between source code and configuration files? 2- Certain python files use the oslo_config to access and define configuration options. Could "all" these python files be considered as configuration files? For example, the following python file of the keystone project: keystone/keystone/conf/auth.py, is it considered a source code or a configuration file? 3- Why are there different source code and configuration repositories for OpenStack projects (eg, nova and puppet-nova)? For instance, does the OpenStack-nova service have some configuration files in its repository and have the puppet-nova as a separate configuration repository as well? Thank you very much in advance for your time and your help! Kind regards, Narjes Bessghaier Narjes Bessghaier Ph.D student in Software Engineering École de Technologie Supérieure (ETS)| University of Quebec Montreal, Canada narjes.bessghaier.1(a)ens.etsmtl.ca<mailto:yasmine.makroum.1@ens.etsmtl.ca>

2 1

Team Meeting Agenda for January 19, 2021
by Clark Boylan 18 Jan '21

18 Jan '21

We will meet January 19, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html *** Zuul results table rendering (ianw 20210105) **** https://review.opendev.org/q/topic:%22gerrit-admin-user%22+status:open *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea 1.13.1 now available **** https://review.opendev.org/c/opendev/system-config/+/769226 * General topics ** OpenAFS cluster status (clarkb 20210119) *** New packages built for CentOS *** Need to sort out openafs-client installation on Debian Buster *** Are we properly installing openafs-client on Xenial? *** https://review.opendev.org/c/opendev/system-config/+/771268 ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210119) *** Enable Xenial -> Bionic/Focal system upgrades ** Discuss infra-core (on behalf of OpenStack) (mnaser) ** two-review rule impact on low-activity projects (zbr 20210115) *** projects like git-review rely on infra team and delay changes *** risks for enabling exceptions? how could we expose these in gerrit? *** apparently that issue is affecting multiple projects, maybe we can think about a generic solution? ** Bup and Borg Backups (clarkb 20210105) * Open discussion

1 0

OpenAFS timestamp rollover issues, discussion and plan
by Ian Wienand 15 Jan '21

15 Jan '21

Hello, We were made aware of an issue relating to handling of a timestamp rollover affecting OpenAFS deployments [1] that in the worst case results in inability of the client and server to communicate. From the OpenAFS 1.8.7 release: It fixes a critical issue in the generation of Rx connection IDs that prevent Rx clients started after 14 Jan 2021 08:25:36 AM UTC from being able to successfully make connections. In addition to cache managers and client utilities, fileservers and database servers are also affected, since they initiate connections to (other) database servers during their normal operation. The issue occurs only at startup, while generating the initial connection ID, so cache managers or servers that were already running at the time in question will not be affected until they restart. The full extent of the issues for our particular circumstances is still somewhat unclear. We run a heterogeneous deployment with all clients under our control using 1.8.6 release, but all servers running 1.6 era packages from Ubuntu Xenial. My current understanding is that since we started our servers and clients before the problem time, we are currently unaffected. When we restart our servers, due to differences in the code with our 1.6 versions, the bug will manifest more as periodic/random-ish I/O failures rather than a complete inability to communicate. We have rebuilt our openafs packages with the required fixes and deployed these to all clients under our control. Thus if any client restarts for whatever reason, they will at least be using fixed code. Some clients we have restarted just to ensure sanity of the new version against the existing severs (so far seems good). All clients have been checked for deployment of these new packages. We do not expect issues, but will monitor this situation. This leaves us with the server side. We have been told there are no fixes for the 1.6 servers planned. We do not have a wonderful answer here, unfortunately. We have planned to move our AFS infrastructure off the Xenial hosts it runs on for some time. These servers are all deployed with legacy puppet, something none of us want to spend significant time hacking on. There is also the small matter that 1.6 to 1.8 upgrades are not terribly well documented, to say the least. While we don't intend on restarting the servers, from time to time the cloud provider needs to migrate things or has other issues that affect uptime; we can't predict this other than it will happen, eventually. It seems our best option here is to take some down-time of the AFS services and perform a manual, in-place upgrade of the existing servers to 1.8 code. We will freeze these from ongoing automated config managment (puppet). This will allow us to deploy and evaluate 1.8 servers and keep with the principle of "change one thing at a time". Once we are in a steady state, we should have enough redundancy that we can replace the servers one-at-a-time with no, or very little, downtime. This gives us time to write, test and review Ansible based deployment as usual. When we do switch in new Focal based servers, we have the advantage we are not also trying to change the AFS version too. In some ways, it works out well (something about lemons and lemonade). We usually try to have a rollback plan with any upgrades. Since 1.6 is not getting updates, and as far as I know we still do not have the technology to move reality backwards in the time continum before the problem rollover time (if I do figure out time travel, I will be sure to pre-respond to this message that it can be ignored) our revert plans seem limited. However, we will be no worse off than if the servers decided to reboot themselves now. In it's defence, this is not new code; 1.8.0 came out early 2018 and we have been using 1.8 clients since we started integrating arm64. The queues are very deep at the moment, so we'd obviously like to minimise downtime. I'd suggest that I could start looking at this around 2021-01-17 21:00UTC, following the plan layed out in [2], which interested parties should definitely review. This should hopefully be a quiet time, and give us a decent runway if we do hit issues. -i [1] https://lists.openafs.org/pipermail/openafs-info/2021-January/043013.html [2] https://etherpad.opendev.org/p/infra-openafs-1.8

3 3

Meeting Agenda for January 5, 2021
by Clark Boylan 04 Jan '21

04 Jan '21

Happy New Year and welcome back. We will have a meeting tomorrow, though depending on attendance it may be on the less formal side of things. I figure we'll try to have an agenda though. Here is our agenda for the meeting on January 5, 2021 at 19:00UTC in #opendev: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Post Gerrit Upgrade Updates **** Configuration tuning ***** Using strong refs for jgit caches ***** Batch user groups and threads **** Zuul results table rendering (ianw 20210105) **** WIP changes (ianw 20210105) ***** This needs work in Zuul to not treat these changes as mergeable. *** Gerrit 3.3.1 includes the fix for Zuul *** Gitea 1.13.1 now available * General topics ** OpenDev Project Update for the Foundation Annual Report (clarkb 20210105) ** Bup and Borg Backups (clarkb 20210105) ** Splitting puppet else into specific infra-prod jobs (clarkb 20201208) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

OpenDev git error: RPC failed; result=18, HTTP code = 200iB | 57.00 KiB/s
by Pengzhi 18 Dec '20

18 Dec '20

Hi OpenDev team, When trying to pull tripleo from opendev.org, git failed with Fatal Error 128. Can you please check and fix it? [ooo@VM-3O ~]$ git config http.postBuffer 24288000 [ooo@VM-3O ~]$ git clone https://opendev.org/openstack/tripleo-quickstart-extras/ Cloning into 'tripleo-quickstart-extras'... remote: Enumerating objects: 19996, done. remote: Counting objects: 100% (19996/19996), done. remote: Compressing objects: 100% (6835/6835), done. error: RPC failed; result=18, HTTP code = 200iB | 57.00 KiB/s     fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed The Error is reproducible with/without the following git configurations [ooo@VM-3O ~]$ git config --list core.compression=-1 http.postbuffer=24288000 Thanks!

3 4

powerkvm CI nova cloning
by Ian Wienand 11 Dec '20

11 Dec '20

Hello, We've identified that powerkvm CI seems to be in a loop attempting to clone nova. It would be helpful if we could get this to stop :) It's just a guess, but we found a similar problem in OpenDev where the executors were attempting to cache the repo, but their default timeout of 300 seconds wasn't long enough to complete the clone. They would kill their clone attempt and start over again immediately. In the Zuul executor logs you'll see something like 2020-12-10 02:02:08,578 ERROR zuul.ExecutorServer: [e: ...] [build:...] Got exception while updating repo gerrit/openstack/nova with a full traceback above. Our solution was to increase the git_timeout [1] to long enough for the clone to complete. The recent upgrade may be to blame. We're experimenting with different options for gc/repack, but cloning the repo from gerrit directly it close to 2gb. Another option is to convert this to cloning to https://opendev.org. Please reach out if there is anything we can do to help. Thanks, -i [1] https://review.opendev.org/c/opendev/system-config/+/766400

2 1

Re: Opendev review crawler
by Chouchen, Moataz 10 Dec '20

10 Dec '20

Dear Jeremy, Ian, Thank you for your kind support and for removing firewall rules that block my access. I will add more delay and use a User-Agent header with my contact details in my future requests as you recommended. I also agree with you that a dump of specific database tables would be great and can encourage other researchers to conduct more studies on code review and benefit from the available rich review data on Opendev projects. Kind regards, Moataz ________________________________ From: Jeremy Stanley Sent: Wednesday, November 4, 2020 12:12 PM To: service-discuss(a)lists.opendev.org Cc: Chouchen, Moataz; Ouni, Ali; Laurin, François Subject: Re: Opendev review crawler On 2020-11-04 00:02:33 +0000 (+0000), Chouchen, Moataz wrote: [...] > For this purpose of study, we need to collect the whole Opendev > data. Right now, we collected data from 2020 to 2015 and still > need the other years. As per requested we will add more waiting > time to my process to let the server catch up. If you also have > any other suggestions that will help, we will be happy to try it. [...] Just adding that delay/throttle will probably suffice, but also Ian's suggestion to set a custom user agent string with contact info on your API requests can help avoid confusion and make it easier for site administrators to reach you if there's a problem with your queries. I've gone ahead and removed the firewall rules we had temporarily blocking your IP addresses as well. If you have any questions, please don't hesitate to get in touch through our service-discuss mailing list. Good luck on the research too, I can't wait to read your findings! -- Jeremy Stanley

2 1

Meeting Agenda for December 8, 2020
by Clark Boylan 08 Dec '20

08 Dec '20

We will meet on December 8, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Need volunteer to run next week's meeting. Or we can cancel. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Post Gerrit Upgrade Updates **** Configuration tuning ***** Adding config to jgit.config ***** Using strong refs for jgit caches ***** Enabling git protocol v2 ***** Batch user groups and threads **** Zuul results table rendering (ianw 20201208) **** WIP changes (ianw 20201208) **** Zuul needs updates to properly support Gerrit 3.3.0 * General topics ** Bup and Borg Backups (clarkb 20201208) ** Docker rate limiting is being seen (at least in limestone) (clarkb 20201208) ** PTG followups (clarkb 20201208) ** meetpad was not useable for (some?) participants from China (frickler 20201208) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Splitting puppet else into specific infra-prod jobs (clarkb 20201208) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Re: [all][infra] CI test result table in the new gerrit review UI
by Ian Wienand 02 Dec '20

02 Dec '20

On Fri, Nov 27, 2020 at 02:11:21PM +0000, Tristan Cacqueray wrote: > From what I understand, we can either use java and the polymer template > system, or the javascript api to implement the zuul results table. This was discussed today in the infra meeting @ [1] There was a couple of broad conclusions. There's no dispute that it works, for now. The maintenace of this in perpituity is the major concern. Some points: There are already two Zuul plugins in the upstream repos zuul : https://gerrit.googlesource.com/plugins/zuul -- This is rather badly namespaced, and is focused on showing the status of Depends-On: changes a cyclic dependencies zuul-status : https://gerrit.googlesource.com/plugins/zuul-status/ -- This is also using a broad namespace, and appears to be related to showing the ongoing job status in the UI directly, but does not show final results. An example I found at [2] from an old wikimedia thread Upstream have shown to take some interest in these plugins when undertaking upgrades to polymer, etc. and thus integrating there is seen as the best practice. Unless we have a strong reason not to, we would like to consume any plugins from upstream for this reason. It would be in our best interest to work with these existing plugins to clarify what they do more directly and come up with some better namespacing to make it more obvious as potential number of plugins grows. The current proposed implementation does not look like any of the upstream plugins. Admittedly, [3] shows this is not a strong eco-system; the documentation is mostly TODO and examples are thin. Implementing this with Reason [4] adds one more significant hurdle; we have no track record of maintaing this javascript dialect and it is not used upstream in any way. It also doesn't use the same build and test frameworks; this may not be a blocker but again creates hurdles by being different to upstream. One more concrete comment is that I'm pretty sure walking the shadow DOM to update things is [5] not quite as intended and we should be using endpoints. In trying to find the best place to start, I've pulled out bits of the checks plugin, image-diff plugin and codeeditor plugin into the simplest thing I could get to a tab with a table in it @ [6]. You can see this for now @ [7]. I think we're going to get more potential eyes using polymer components and making things look like the existing plugins. It's a matter of taste but I think the blank canvas of a separate tab serves the purpose well. I think with the Bazel build bits, we can ultimately write an upstream Zuul job to build this for us in gerrit's Zuul (corvus might know if that's right?) Tristan -- maybe as step one, we could try integrating what you have into that framework? Maybe we rewrite the ReasonML ... but having it something is step 1. Overall I think this is the broad strokes of what we would look at sending upstream. If they think it's too specific, Zuul tenant seems a logical home. We would also like to expand the gate testing to better handle testing plugins. This will involve us automatically adding a repo, sample change, and Zuul user comments during the gate test. A logical extension of this would be to take samples using a headless browser for reporting as artifacts; held nodes can also be used for advanced debugging. This will give us better confidence as we keep our Gerrit up-to-date. I'd like to solict some feedback on the checks plugin/API, which Zuul added support for with [8]. My understanding is that this has never really consolidated upstream and is under redevelopment [9]. I don't think there's much there for us at this point; even still seeing as we are so Zuul centric we might be able to do things with a specific plugin this API will never do. -i [1] http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-12-01-19.01.l… [2] https://imgur.com/a/uBk2oxQ [3] https://gerrit-review.googlesource.com/Documentation/pg-plugin-dev.html [4] https://reasonml.github.io/ [5] https://github.com/softwarefactory-project/zuul-results-gerrit-plugin/blob/… [6] https://github.com/ianw/gerrit-zuul-summary-status [7] https://104.130.172.52/c/openstack/diskimage-builder/+/554002 [8] https://opendev.org/zuul/zuul/commit/e78e948284392477d385d493fc9ec194d54448… [9] https://www.gerritcodereview.com/design-docs/ci-reboot.html

2 1

Meeting Agenda for Decemer 1, 2020
by Clark Boylan 01 Dec '20

01 Dec '20

We will meet with this agenda on Decemer 1, 2020 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Post Gerrit Upgrade Updates **** Configuration tuning ***** Cache sizes ***** Autogc **** Zuul results table rendering **** Switch to Java 11 ***** Any concern about java version specific things in lucene or h2 caches? ***** Shoudl definitely test this on review-test * General topics ** Bup and Borg Backups (clarkb 20201201) ** Docker rate limiting is being seen (at least in limestone) (clarkb 20201201) ** PTG followups (clarkb 20201201) ** meetpad was not useable for (some?) participants from China (frickler 20201201) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Splitting puppet else into specific infra-prod jobs (clarkb 20201201) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Third Party CI updates to treat review.opendev.org better
by Clark Boylan 30 Nov '20

30 Nov '20

Hello everyone, Since the Gerrit 3.2 upgrade we've noticed occasional high load from Gerrit. In debugging we've discovered there are cache and other setting tunables that we could configure better for our setup. We are in the process of making these changes to our Gerrit configuration. We have also noticed that a specific Zuul behavior contributes to the load as well. We are asking that third party CI systems running Zuul apply this change [0] or add Gerrit http configuration [1] to your configuration. We recommend the http configuration because it is available as far back as Zuul 3.3.0 and adds more functionality to your setup. This is also an excellent time to update your Third Party CI information [2] (including contact info) as we've noticed that many of these wiki entries may be stale. If you are still running Zuul v2 can you reach out to us and we'll see what we can do to help. [0] https://review.opendev.org/c/zuul/zuul/+/764069 [1] https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%… [2] https://wiki.openstack.org/wiki/ThirdPartySystems Clark

1 0

Meeting Agenda for November 17, 2020
by Clark Boylan 16 Nov '20

16 Nov '20

We will meet Novemeber 17, 2020 at 19:00 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test is back up an running. Now with November 5th state. **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug update and the welcome message hooks rely on database access and will need to be updated or sunsetted **** Upgrade Notes ***** Gave more threads to notedb conversion and it went quicker. Not fast but was an improvement. ***** Need to land the non WIP changes at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep ***** The first WIP change in https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep is a number of squashed changes to simplify post upgrade config management sync up. ***** https://etherpad.opendev.org/p/opendev-gerrit-3.2-upgrade-plan the plan is here. A few items still need a little work. Please take a look. **** Unknowns ***** Storyboard integration * General topics ** Bup and Borg Backups (clarkb 20201117) ** PTG followups (clarkb 20201117) ** meetpad was not useable for (some?) participants from China (frickler 20201110) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Long term plans for running openstackid.org (clarkb 20201110) ** Splitting puppet else into specific infra-prod jobs (clarkb 20201110) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Meeting Agenda for November 10, 2020
by Clark Boylan 09 Nov '20

09 Nov '20

We will meet with this agenda Novemer 10, 2020 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test is back up an running. Now with November 5th state. **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug update and the welcome message hooks rely on database access and will need to be updated or sunsetted **** Upgrade Notes ***** Gave more threads to notedb conversion and it went quicker. Not fast but was an improvement. ***** Don't think we will do the surrogate Gerrit as it increases complexity of testing and implementation. We only need to do this once. ***** Gerrit 3.2 seems to want email addresses set on accounts in order to modify them (eg add an ssh pub key). This affects our admin accounts and the project creator. ***** How do we safely land the stack at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep since Zuul will try to run ansible in sequence for each change? **** Unknowns ***** Storyboard integration *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG followups (clarkb 20201110) ** meetpad was not useable for (some?) participants from China (frickler 20201110) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Bup and Borg Backups (clarkb 20201110) *** Borg backup all the things https://review.opendev.org/#/c/761855/ ** Long term plans for running openstackid.org (clarkb 20201110) ** Splitting puppet else into specific infra-prod jobs (clarkb 20201110) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Re: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Antoine Musso 05 Nov '20

05 Nov '20

Hello, Wikimedia has upgraded from 2.16 to 3.2 in June 2020 conducted by Christian Aistleitner. He wrote a nice report on the upstream mailling list: https://groups.google.com/g/repo-discuss/c/G5wucKJg9Ag On 27/10/2020 22:16, Clark Boylan wrote: > The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. > > The upgrade has two major portions. First we will incrementally move from our current version 2.13 to 2.16. Each point release requires a database migration and git indexing operations that take the majority of the time. From Christian report, you should be able to skip the git indexing between each minor upgrades and only do a full reindexing once you are upgraded to 3.2. The git indexing had an issue which is that the changes to index were split by repository. In the worse case scenario, if you have a thousand of small repositories and one very large one, the later was only processed by a single thread. Christian added code to chunk by changes regardless of the repository, that has dramatically speed up the indexing. If I am not mistaken, the commit is 20784548c3fb and it has been released in 2.16.22, 3.0.12, 3.1.8 and 3.2.3. Given you should be able to skip reindexing between upgrades. Make sure to use 3.2.3 to benefit from the faster indexing. > The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend. This stores reviews together with code in the git trees, rather than in a separate database. There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps. This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing. We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2. 2.16 has received improvements for the NoteDB migration. Again make sure to use the latest patch release (2.16.23 at this time). <snip> > Some important things to know: > * Gerrit's web UI will be changing. We don't have a lot of control over this. As a result we'll lose some existing CI result rendering niceness that we have on 2.13 (in particular the summary table and CI results toggle will go away). We hope that once we've upgraded we can investigate solving this through Gerrit plugins. summary table ------------- The new Gerrit web UI is enterely JavaScript driven (using https://www.polymer-project.org/ ) and relies on the REST API to fetch data. For the summary table, Wikimedia has a light version of the one you have which got borrowed from somewhere else. See "gr-test-result-table-module" at: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/hea… It is hopefully not too complicated to implement. CI results toggle ----------------- That is now built-in Gerrit, the UI will have a "Only comments" toggle which filters out any message with a tag prefixed by "autogenerated:". Thus in Wikimedia good old Zuul 2.5 we have simply tagged with "autogenerated:ci" by using: success: gerrit: verified: 2 tag: autogenerated:ci Result: https://phabricator.wikimedia.org/T48148#6294913 (there are a few more details on that task). Another one is that the SUCCESS/FAILURE message can no more be highlighted via rewriting the message and using CSS. The way text is parsed has completely changed and its now impossible to do it via the commentlinks Gerrit settings. As a result on Zuul 2.5 we have set job_name_in_report = false and lack the nice formatting and green/red coloring. I am not sure whether that still applies to Zuul 3.x though. Wikimedia task: https://phabricator.wikimedia.org/T256575 That being said, it is definitely possible to write a Gerrit Javascript plugin that would parse the message and prettify the results comment on the client side. <snip> > * Q&A * > If this upgrade isn't perfect why are we doing it anyway? > We've come to the realization that if we don't make imperfect progress we'll never make any progress. We have decided that the benefits outweigh the known drawbacks and we'll do our best to work on those issues after the upgrade. I would add a couple user facing improvements: The new UI alone is definitely worth the upgrade. It is way more pleasant than the old one. Albeit it needs some days to adjust, you will surely never look back :] <snip> The cherry on the cake is that with the upgrade comes support for Git protocol version 2. In short, when enabled, it makes fetches from big repository an order of magnitude faster. I wrote a quick blog post about git protocol v2 at: https://phabricator.wikimedia.org/J199 And Google announcement with lot more details: https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-… I wish you the very best for the maintenance. Thank you for making it happen! -- Antoine "hashar" Musso

3 2

Re: Opendev review crawler
by Chouchen, Moataz 04 Nov '20

04 Nov '20

Hi Jeremy, Thank you for your quick answer and for your support we will be more than happy to share our research with you once published! For this purpose of study, we need to collect the whole Opendev data. Right now, we collected data from 2020 to 2015 and still need the other years. As per requested we will add more waiting time to my process to let the server catch up. If you also have any other suggestions that will help, we will be happy to try it. We will be happy to receive an answer from you Kind regards, Moataz ________________________________ From: Jeremy Stanley Sent: Tuesday, November 3, 2020 10:42 AM To: service-discuss(a)lists.opendev.org Cc: Chouchen, Moataz; Ouni, Ali; Laurin, François Subject: Re: Opendev review crawler On 2020-11-03 17:43:53 +0000 (+0000), Chouchen, Moataz wrote: > I am Moataz Chouchen a Phd Student in Ets Montréal under the > supervision of Professor Ali Ouni. I am sending this email to you > to explain my requests of the Opendev data to respond to your > request regarding Opendev code review data crawling. Thanks so much for getting back to us! I'm sorry we ended up blocking access to your various systems, but we were at a loss for how to otherwise get in contact with whoever was running them since the queries were being performed anonymously (at least until we finally saw similar API requests originating from your university). > In fact, my Phd subject consists of the identification, the > localization, and the understanding of Modern Code Review (MCR) > problems (also called antipatterns). For this purpose, we need to > crawl code review data from different projects (including Opendev) > since this data will help us to study the process of MCR in these > projects and their associated anti-patterns. Specifically, we need > the whole data of Opendev to perform data analysis methods on them > (including code review metrics distributions, social graph > analysis etc) and study them in more depth. For this reason, we > created a script to crawl data from Opendev and use them in our > empirical study on MCR antipatterns in Opendev. This sounds really interesting, and I'm sure this mailing list would also love to receive a link to your research once you've completed and published it. I'm thrilled you have an interest in looking into these topics, especially with regards to the usage patterns and trends for a service we're operating. > I would like to thank you for your interest to understand the > issues behind our requests and your intention to solve the issue > for both sides. I am looking forward to collaborating with you and > I would like to know if there are any rules that I should follow > when requesting data from your servers and I will be more than > happy If you can help me crawling the data and provide an access > for me. I am also at your disposal for further clarifications > regarding the requests. I hope I hear from you soon. Mainly what I think we need to know is whether you've already collected the data you need from our Gerrit API, or how much you still need to query. The server wants to cache requests in memory for the sake of efficiency, but as you're aware we've got quite a lot of data. When queries for older data consume available memory on the server it begins to spend a lot more time trying to aggressively garbage collect cache contents, and that degrades its ability to serve requests for you and for other users. If you still have a lot left to query, you might try to introduce a bit of a pause between each batch of paginated results to give the server some time to catch up freeing memory for caching other requests. Also if obtaining the data in a different form would be useful to you, we might be able to look into performing a dump of specific database tables so you don't have to slowly trickle it out of the API. Please do reply to the service-discuss mailing list at your earliest convenience and let us know how we can better help you with your research. Thanks again! -- Jeremy Stanley

2 1

opendev/gear reviews
by Antoine Musso 04 Nov '20

04 Nov '20

Hello, The opendev/gear is a pure python implementation of the Gearman protocol, written for Zuul 2.x. The project has a few pending changes that could use a final approval, one is required to unbreak the CI jobs. I am interested in helping on the maintenance, though I don't know which process I can start to eventually be blessed with W+1 rights on this repository. I will be more than happy to help on that front. Meanwhile, here is a summary of the changes that could certainly be approved immediately: "use python3 as context for build-python-release" https://review.opendev.org/#/c/742165/ Without it, the job fails for all proposed change. The same fix has been made to Zuul via https://review.opendev.org/#/c/742761/ "Bump crypto requirement to accommodate security standards" https://review.opendev.org/#/c/742117/ Raise the RSA security certificate bits from 1024 to 2048 and sign the key with sha256 instead of sha1. Without it, it seems pyOpenSSL complains the key is too small and the hash system too weak. "wakeConnections: Randomize connections before scanning them" https://review.opendev.org/#/c/747119/ That is an issue we had for a while at Wikimedia, the server wakes connection in the order they get registered. The connection that registered first ends up executing way more work than the last registered one. By shuffling the list of connections before waking them up, that helps spreading the load. "Modify connection timeout process" https://review.opendev.org/#/c/531059/ Resolves a potential deadlock, a connection timeout would not release the connection lock. "Move handleDisconnect into BaseClientServer" https://review.opendev.org/#/c/714709/ Really a noop, it is just moving a method to the parent class which is the sole use of it. "Added Docker image builds" https://review.opendev.org/#/c/688446/ Generates a Docker image to run the gear daemon. cheers, -- Antoine "hashar" Musso

1 0

Opendev review crawler
by Chouchen, Moataz 04 Nov '20

04 Nov '20

Dear Opendev administrators, I am Moataz Chouchen a Phd Student in Ets Montréal under the supervision of Professor Ali Ouni. I am sending this email to you to explain my requests of the Opendev data to respond to your request regarding Opendev code review data crawling. In fact, my Phd subject consists of the identification, the localization, and the understanding of Modern Code Review (MCR) problems (also called antipatterns). For this purpose, we need to crawl code review data from different projects (including Opendev) since this data will help us to study the process of MCR in these projects and their associated anti-patterns. Specifically, we need the whole data of Opendev to perform data analysis methods on them (including code review metrics distributions, social graph analysis etc) and study them in more depth. For this reason, we created a script to crawl data from Opendev and use them in our empirical study on MCR antipatterns in Opendev. I would like to thank you for your interest to understand the issues behind our requests and your intention to solve the issue for both sides. I am looking forward to collaborating with you and I would like to know if there are any rules that I should follow when requesting data from your servers and I will be more than happy If you can help me crawling the data and provide an access for me. I am also at your disposal for further clarifications regarding the requests. I hope I hear from you soon. Thank you Please stay safe, Moataz

3 2

Team meeting Agenda for November 3, 2020
by Clark Boylan 02 Nov '20

02 Nov '20

We will meet November 3, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html ** Note the Daylight Savings Time switch has happened in many parts of the world. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** Need to rerun upgrade testing on review-test and revive it for usage testing **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug/spec update hooks and the welcome message hook rely on database access and will need to be updated or sunsetted **** Upgrade Process ***** Backup then upgrade from 2.13 to 2.16. This is our fallback midpoint checkpoint ***** Backup again then migrate to notedb on 2.16 ***** Upgrade to 3.2 ***** Upgrade to 2.16 along with backups should be doable in a day. Then notedb migration can happen overnight with 3.2 upgrade happening on day two. ***** Looking into using a surrogate Gerrit running on a very performance vexxhost flavor to do the upgrade/data migrations which may speed things up. Needs testing. **** Unknowns ***** Storyboard integration *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG followups (clarkb 20201103) ** meetpad was not useable for (some?) participants from China (frickler 20201030) *** neutron went back to using Zoom for their last sessions because of this *** can we (maybe via help from the foundation involving their local members) work on improving this? *** would be sad to see open tools not being able to be used because of this ** Bup and Borg Backups (clarkb 20200929) *** Ethercalc to be the first borg backed up service ** Long term plans for running openstackid.org (clarkb 20201103) ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Re: [service-announce] October 20 Gerrit Outage Update
by Mohammed Naser 30 Oct '20

30 Oct '20

Hi everyone, I'm happy to see that things are back in order, however, I *hate* to be that person, but I think there are still some hard questions that we need to answer together transparently. I am especially concerned because of how this affects the workflow of developers overall but also the security measures we have in place, should something like Zuul be targeted instead. I've added some questions in-line below Thanks, Mohammed On Tue, Oct 20, 2020 at 8:33 PM Ian Wienand <iwienand(a)redhat.com> wrote: > > As of this mail, Gerrit access has been restored. Please read on for > important information, especially around change verification. > > Background > ----------- > > On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval > to a change that they were not expected to have access to. At 02:06 > UTC an alert was raised via IRC. Administrators found the account had > added themselves to a core group and made the +W vote. The account > was disabled, and removed from the groups it had added itself to by > 02:55 UTC. Administrators began to analyse the situation and Gerrit > was taken offline at 04:02 UTC to preserve state and allow for > analysis. > > From this time, administrators were working on log collection and > analysis, along with restoring backups for comparison purposes. > > By around 08:45 UTC it was clear that the privilege escalation had > been achieved by gaining control of a Launchpad SSO account with > Gerrit administrator privileges. By this time, we had ruled out > software vulnerabilities. Logs showed the first unauthorized access > of the administrator account in Gerrit on 2020-10-06. Communication > with Launchpad admins agrees with this analysis. We saw one session > opened as the administrator user to StoryBoard on this same day, but > logs show no data was modified or hidden stories viewed. So, just to be clear, someone who had root access to our Gerrit installation had their account compromised which resulted in this (and not something that occurred as a by-product of some other service -- say storyboard -- leaking some sort of information?) I see two issues in this at the moment: - There is no need for us to have anyone with admin powers to Gerrit at all times, we've done enough automation to sustain us and a manual 'circuit breaker' of adding a user *IF* necessary should be put in place. - If the above is not possible, anyone who is part of this group should have 2FA enabled inside Launchpad's SSO. I would very much prefer the first option rather than the second one. If it was an individual's account that was accessed and not a system account, have we audited that there are not other things that might have been accessed such as resources relating to Zuul, other systems and potentially rotating/auditing all our infrastructure? > Analysis has been performed on the Gerrit database and git trees from > October 1st, pre-dating any known unauthorized access. > > Access was restored at around 2020-10-21 00:00 UTC > > Outcomes > ----------- > > The following has been verified: > > The administrator account used has been disabled and credentials > updated > > We have verified that all group and user addition/removals since > Oct 1 are valid. The only invalid additions were made by the > compromised administrator account to add a single user account to > the Administrators group; and then that account added itself to > another known group. > > The account given administrator privilege has been removed from > the groups it added itself to and is disabled. > > There is no evidence of any unauthorized access via methods other > than Gerrit HTTP and Gerrit SSH access. > > No commits have been pushed to git trees bypassing code review. > Every git tree has been compared to the Oct 1 version and all > commits have been correctly inserted via Gerrit changes. I saw this artifact, I have no idea if it was put into consideration, but, food for thought: https://review.opendev.org/#/c/758881/ > The version of Gerrit we use stores HTTP API passwords in > plain-text. We know that a limited number of passwords were > gathered via the HTTP API and it is possible passwords were > gathered via the database. We thus have assumed that all HTTP API > passwords have been disclosed. This password needs to be > explicitly enabled by users, and many users do not have it > enabled. > > Remediation > ----------- > > This leaves us with the following remediation actions: > > Users should double-check their Launchpad recent activity at > https://login.launchpad.net/activity for any suspicious logins. If > found, please notify the OpenDev admins in Freenode #opendev and > Launchpad admins in #launchpad immediately. > > All HTTP API passwords have been cleared. If you push changes via > HTTPS (instead of typical SSH), are a gertty user, or run a CI > system or something else that communicates with the Gerrit HTTP > API, you will need to regenerate a password. > > Any SSH keys added to accounts since 2020-10-01 have been removed. > This affects only a limited number of accounts. This is done in > an abundance of caution, and we do not believe any accounts had > unauthorized SSH keys added > > We should audit all changes for projects since 2020-10-01. > > We have no evidence that any account had its ssh keys compromised, > thus we can rule out any unauthorized changes being uploaded via SSH. > However we can not conclusively rule out that compromised HTTP API > passwords were used to push a change through Gerrit. For example, a > change could be uploaded that looks like it came from a user, or the > API key of a core team member may have been used to approve a change > without authorization. > > Given our extensive analysis we consider it exceedingly unlikely that > this vector was used. We have had no notifications of users seeing > unexpected changes either uploaded by them, or approved by them in > projects they work on. This said, we believe it is important to > inform the community of this very unlikely, but still possible, > vulnerability of the source code. > > To this end, we have prepared a list of all changes from the known > affected period which should be audited for correctness. These are > available at > > https://static.opendev.org/project/opendev.org/gerrit-diffs/ > > Team members should browse these changes and make sure they were > correctly approved in Gerrit. If any change looks suspicious you > should notify OpenDev administrators in Freenode #opendev immediately. > > Further actions > ---------------- > > We are planning the following for the short term future: > > The Opendev administrators will be looking at alternative models > for Gerrit admin account management. > > We are already well into planning and testing a coming upgrade to > a version of Gerrit which does not store plain-text API keys. > > Longer term, we've written a spec for replacing Launchpad SSO as > our authentication provider. > > We thank you for your patience during this trying time, and we look > forward to returning to supporting the community doing what it does > best -- working together to create great things. > Thank you for this. I'd also like to raise the question of moving forward, how to be able to track these things. We had a user that had full root access to our Gerrit installation for ~2 weeks without our knowledge entirely, only uncovered when they did something (that, in the grand scheme of things, was relatively trivial, compared to what could have happened). What can we do to set up the necessary infrastructure to ensure that these things are monitored. OpenDev is considered to be critical infrastructure for this entire community and there's not much that an outsider can do other than the 'keyholders' for the resources. We've historically refused to have any monitoring and now things like this have slipped up, I'm just worried that we have a big looming thing coming up ahead of us that will catch us off guard and we'll be completely unprepared for it... > > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -- Mohammed Naser VEXXHOST, Inc.

4 7

Re: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Radosław Piliszek 28 Oct '20

28 Oct '20

On Tue, Oct 27, 2020 at 10:17 PM Clark Boylan <cboylan(a)sapwetik.org> wrote: > The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. <3 Finally! <snip> > * Gerrit 3.2 requires git 2.2.0 or newer to use the Change ID commit hook. This may be a problem for RHEL/CentOS 7 users. I thought it depends on git-review itself rather than a particular Gerrit version? <snip> > How can I help? > Once the upgrade is complete you'll want to confirm the basic functionality you rely on is there. We know there will be differences or missing features. Patience as we figure out how to address those on a new Gerrit installation is much appreciated. If you're interested in hacking on Java and Javascript we'd love help with the plugins necessary to address the known problems. You should be able to build this out locally without any special access. Please let us know if you are interested and we can help you bootstrap. Count me in if there are things to be done, might be able to spare some time for Gerrit. :-) We might want to port the live Zuul status reporter as well. -yoctozepto

2 2

Cancelling the team meeting today
by Clark Boylan 27 Oct '20

27 Oct '20

Hello everyone, We'll cancel the team meeting today since many of us are distracted by the PTG. We'll see you next week! Clark

1 0

Re: Some of your OpenDev Gerrit SSH keys have been deleted
by App Support 23 Oct '20

23 Oct '20

Hi Jeremy, Would the deletion affect the login to https://review.opendev.org/? <https://review.opendev.org/?> Somehow I got an error when trying to login. I am new to OpenStack and just started contributing to documentations. Thank you, Qi Zhang > On Oct 22, 2020, at 2:36 PM, Jeremy Stanley <fungi(a)yuggoth.org> wrote: > > In response to a security breach for the Gerrit Code Review service > at https://review.opendev.org/ I have deleted one or more SSH public > keys uploaded to your account between October 1 and October 19, > 2020. If you added any SSH public keys during that time, you will > need to reupload them in your user preferences before they can be > used to authenticate to the service. For more information on the > breach, see this announcement: > > http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html > > If you have any questions, feel free to reach out to OpenDev's > systems administrators on the service-discuss(a)lists.opendev.org > mailing list or in the #opendev channel on the Freenode IRC network. > -- > Jeremy Stanley, OpenDev Systems Administrator

2 2

Re: [service-announce] October 20 Gerrit Outage
by Jonathan Bryce 20 Oct '20

20 Oct '20

Thanks Clark and the rest of the OpenDev infra crew for literally working around the clock on this issue! Appreciate the effort to verify everything. Also wanted to share that the updates are being posted here if people want to see the history for completeness: https://review.opendev.org/maintenance.html Jonathan On October 20, 2020 12:50:32 "Clark Boylan" <cboylan(a)sapwetik.org> wrote: > Hello everyone, > > By now most of you have probably noticed that we took Gerrit offline > recently. The reason for that is we believe an admin account in Gerrit was > compromised allowing an attacker to escalate privileges within Gerrit. > > Around 02:00 UTC October 20 suspicious review activity was noticed, and we > were made aware of it shortly afterwards. The involved account was disabled > and removed from privileged Gerrit groups. After further investigation we > decided that we needed to stop the service, this happened at about 04:00 UTC. > > After the service was stopped we shifted focus to identifying the source of > the issue as well as investigating impact. We believe this originated on > October 6th with at least two compromised Ubuntu One accounts. One of which > was a Gerrit admin account. These accounts, like the one that initially > tipped us off, have been dealt with at this point. > > In order to evaluate impact we are using backups from October 1 to find > configuration, database, and git repo changes that have been made. We have > identified 97 accounts that updated ssh keys after that point in time. > These ssh keys are being removed as we can't be sure the changes were valid > changes made by the user. If you are one of these users you will need to > add your key(s) back in. We will also attempt to reach out to the affected > users directly by email soon. We will be checking openid urls and group > membership changes as well. We will determine what actions make sense for > these items once we have evaluated the impact to them. > > All Gerrit HTTP API tokens will be deleted. You will need to generate new > ones if you are an API user. Sorry, gertty fans. > > On the git repo side of things there are a few things that we will need to > check. Using our October 1 state we will generate lists of commits that > have landed since then for each branch on each repo. We will verify that > the latest commit can reach the last known good commit in the git DAG. For > non merge commits we will also correlate these to Gerrit changes. We will > then ask that you help us by verifying the commits on your projects are as > reviewed and not malicious. We will also need to check git tags which > should all be signed and can be verified that way. > > This is a good reminder to check activity on your online accounts and > identities for anything unexpected. > > We understand that an inaccessible Gerrit is not fun. We are trying to go > as quickly as we can while also not sacrificing caution and care. > > Clark > > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

1 0

Cancelling the team meeting October 20, 2020
by Clark Boylan 20 Oct '20

20 Oct '20

Hello everyone, I've been swamped with summit things. Initially I thought that we could do a quieter meeting during the summit, but I've been too distracted to even look at an agenda. For that reason I think we should cancel the meeting this week. We've got the PTG next week. Please register and check the schedule if you plan to attend. We'll see you there as well as at next week's meeting (I hope). Clark

1 0

Meeting Agenda for October 13, 2020
by Clark Boylan 12 Oct '20

12 Oct '20

We will meet in #opendev-meeting at 19:00 UTC on October 13, 2020 with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release October 14 ** Summit next week. PTG the week after. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test.opendev.org is an upgraded snapshot of production from October 1. Please check it out **** Basic functionality seems to be working ***** logging in, git review -s, git review to push, commenting on changes, ICLA signing, replication, change searching, and so on. **** jeepyb bug/spec update hooks and the welcome message hook rely on database access and will need to be updated or sunsetted **** Upgrade Process ***** Backup then upgrade from 2.13 to 2.16. This is our fallback midpoint checkpoint ***** Backup again then migrate to notedb on 2.16 ***** Upgrade to 3.2 ***** Upgrade to 2.16 along with backups should be doable in a day. Then notedb migration can happen overnight with 3.2 upgrade happening on day two. **** Unknowns ***** Storyboard integration **** Can we start talking about scheduling the outage and upgrade? *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG PLanning (clarkb 20200929) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Bup and Borg Backups (clarkb 20200929) *** Ethercalc to be the first borg backed up service ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Testing the results of Gerrit upgrades on review-test.opendev.org
by Clark Boylan 09 Oct '20

09 Oct '20

Hello everyone, Recently fungi and I managed to make review-test.opendev.org a copy of production Gerrit (review.opendev.org) as of October 1. We have since upgraded that installation to Gerrit 2.16. The 2.16 version is a potential upgrade target for our production Gerrit as we have to stop there and do a required notedb migration step. What isn't yet clear is if we'll expose the 2.16 installation or continue on to upgrade to a 3.x version. In order to figure that out we'll be doing the notedb migration and the 3.x upgrades in the near future. Once we've gone through the process we should have a good idea of what makes sense as far as upgrades go for us. In any case the server is up for testing now if people want to take a look at 2.16 and offer feedback. I'll followup on this thread when we get upgraded to 3.x and ask for feedback on that version too. Head to https://review-test.opendev.org and check it out. There are some minor known issues. The biggest that people will probably notice is that our hacky javascript CI results table does not work in the old gerrit UI or the new polygerrit UI. The commentlinks for zuul results are also not working on polygerrit UI. We're still investigating, but we're going to be careful that we don't worry about every single issue like that pre upgrade otherwise we'll never get the upgrade done. Instead we'll gather feedback and prioritize then sort out what needs fixing now and what we can sort out later. Let us know what you think, Clark

1 1

Team Meeting Agenda for October 6, 2020
by Clark Boylan 06 Oct '20

06 Oct '20

We will meet at 19:00 UTC in #opendev-meeting on October 6, 2020 with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release next week. ** Rax hosted db outages around 03:00-05:00 UTC Friday including those for review and grafana * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Where is review-test is now a mimic of production Gerrit as of October 1 **** Testing of the upgrade process to begin shortly. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG PLanning (clarkb 20200929) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Rehoming tarballs (ianw 20200929) *** large selection of things in tarballs.openstack.org/openstack/... that should be under tenant dirs opendev/ x/ zuul/ etc. *** https://review.opendev.org/#/c/754257/ -- script to generate script that generates a script to cleanup *** http://paste.openstack.org/show/798368/ -- resulting moves *** symlink old directories? apache redirects (files gone on afs)? just move but notifiy lists? ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Bup and Borg Backups (clarkb 20200929) *** https://review.opendev.org/741366 is ready to land when we are ready. ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Team Meeting Agenda for September 29, 2020
by Clark Boylan 29 Sep '20

29 Sep '20

We will meet on September 29, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. **** We should consider spinning up a new host or modifying the existing host to match productions cinder volume usage for git repos. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG PLanning (clarkb 20200929) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Rehoming tarballs (ianw 20200929) *** large selection of things in tarballs.openstack.org/openstack/... that should be under tenant dirs opendev/ x/ zuul/ etc. *** https://review.opendev.org/#/c/754257/ -- script to generate script that generates a script to cleanup *** http://paste.openstack.org/show/798368/ -- resulting moves *** symlink old directories? apache redirects (files gone on afs)? just move but notifiy lists? ** zuul tarball publishing (ianw 20200929) *** https://review.opendev.org/#/c/754245/ -- current publishing extracts javascript tarball into root *** do we need separate javascript publishing with containers *** zuul only publishing tag tarballs to pypi, intention? ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Bup and Borg Backups (clarkb 20200929) *** https://review.opendev.org/741366 is ready to land when we are ready. ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Team Meeting Agenda for September 22, 2020
by Clark Boylan 21 Sep '20

21 Sep '20

We will meet at 19:00 UTC in #opendev-meeting on September 22, 2020 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Keeping Gitea project descriptions up to date. **** Some users have noticed we don't update descriptions when they change. Process is currently manual. Can we make that better? *** Gitea 1.12.4 has released. Anyone interested in pushing up a change to test and upgrade us? *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. **** We should consider spinning up a new host or modifying the existing host to match productions cinder volume usage for git repos. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** Splitting puppet else into specific infra-prod jobs (clarkb 20200922) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Bup and Borg Backups (clarkb 20200922) *** https://review.opendev.org/741366 is ready to land when we are ready. ** PTG PLanning (clarkb 20200922) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200922) *** Wiki updates * Open discussion

1 0

When we will see Gerrit 3.x on opendev.org?
by Sorin Sbarnea 16 Sep '20

16 Sep '20

I am wondering when we will be able to get our gerrit upgraded to 3.x? I am mostly interested about the new UI changes as the old one is a bit of a nightmare to use on split screens, even after hacking its css. How far are we from having a d-day for this? Thanks, /sorin

2 1

Team Meeting Agenda for September 15, 2020
by Clark Boylan 15 Sep '20

15 Sep '20

We will meet in #opendev-meeting at 19:00 UTC September 15, 2020 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. **** We should consider spinning up a new host or modifying the existing host to match productions cinder volume usage for git repos. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** Recurring bogus IPv6 addresses on mirror01.ca-ymq-1.vexxhost.opendev.org (frickler 20200915) *** We configured the interface statically with netplan. Any further issues? ** Zuul-web performance issues (clarkb 20200915) *** We weren't caching as expected. *** zuul web js had a bug which refreshed the status.json often *** Do we need to run more zuul-web processes? ** Bup and Borg Backups (clarkb 20200915) *** https://review.opendev.org/741366 is ready to land when we are ready. ** PTG PLanning (clarkb 20200915) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200908) *** Wiki updates * Open discussion

1 0

review.opendev.org SSH key change?
by Marcin Cieslak 14 Sep '20

14 Sep '20

I have clned the python-jenkins repo few days ago and today I am getting this: > git remote update Fetching upstream @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:RXNl/GKyDaKiIQ93BoDvrNSKUPFvA1PNeAO9QiirYZU. Please contact your system administrator. Update the SSHFP RR in DNS with the new host key to get rid of this message. > git remote -v upstream ssh://saperski@review.opendev.org:29418/jjb/python-jenkins (fetch) upstream ssh://saperski@review.opendev.org:29418/jjb/python-jenkins (push) My cached entries are: [review.opendev.org]:29418 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfsIj/jqpI+2CFdjCL6kOiqdORWvxQ2sQbCzSzzmLXic8yVhCCbwarkvEpfUOHG4eyB0vqVZfMffxf0Yy3qjURrsroBCiuJ8GdiAcGdfYwHNfBI0cR6kydBZL537YDasIk0Z3ILzhwf7474LmkVzS7V2tMTb4ZiBS/jUeiHsVp88FZhIBkyhlb/awAGcUxT5U4QBXCAmerYXeB47FPuz9JFOVyF08LzH9JRe9tfXtqaCNhlSdRe/2pPRvn2EIhn5uHWwATACG9MBdrK8xv8LqPOik2w1JkgLWyBj11vDd5I3IjrmREGw8dqImqp0r6MD8rxqADlc1elfDIXYsy+TVH review.opendev.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBjjFqbkqLfVKn5vJnKh/LfGGo0gXp/vWlXRs0H91E0X8ce4r8DHLB8PMScHrX/n7c29/DY3FQqSaYsY6o13dFA= DNS sshfp records: review.opendev.org. 193 IN CNAME review01.opendev.org. review01.opendev.org. 276 IN SSHFP 1 1 4A38DD28E379EA443F8A722D8330EF41754F0D5E review01.opendev.org. 276 IN SSHFP 1 2 4234515B3D59F8BF7E7095BD881F39DB21B97A15511C4B10B7FD0240 25AD93FC review01.opendev.org. 276 IN SSHFP 3 1 382C41E6FFC60CF1939B292FA62879B1918145D4 review01.opendev.org. 276 IN SSHFP 3 2 52A81E8DD662F92D903199DBC5068280D33D21D3A8E5BD023FAADD47 99AC1DDF review01.opendev.org. 276 IN SSHFP 4 1 DE5FAA47C38E616ECB2CCC4C30C7E3F788C0927A review01.opendev.org. 276 IN SSHFP 4 2 4BE301BEEC8DCC06C1084BEF1DB1D136686022B7026F678D958E548E 4B7D2FC7 Is everything ok? Marcin

5 7

Farewell Party for Monty
by corvus＠inaugust.com 10 Sep '20

10 Sep '20

Hi, Monty is starting a new gig and won't be spending as much time with us. Since we all haven't seen each other in a while, let's have one more beer[1] together and say farewell. Join us for a virtual going-away party on meetpad tomorrow (Friday) at 21:00 UTC at this URL: https://meetpad.opendev.org/farewell-mordred Stop by and chat for old time's sake. -Jim [1] Bring your own beer.

1 0

Moving on
by Monty Taylor 10 Sep '20

10 Sep '20

Hi everybody, After 10 years of OpenStack, the time has come for me to move on to the next challenge. Actually, the time came a few weeks ago, but writing farewells has always been something I’m particularly bad at. My last day at Red Hat was actually July 31 and I’m now working in a non-OpenStack job. I’m at a loss for words as to what more to say. I’ve never done anything for 10 years before, and I’ll be very surprised if I do anything else for 10 years again. While I’m excited about the new things on my plate, I’ll obviously miss everyone. As I am no longer being paid by an OpenStack employer, I will not be doing any OpenStack things as part of my day job. I’m not sure how much spare time I’ll have to be able to contribute. I’m going to hold off on resigning core memberships pending a better understanding of that. I think it’s safe to assume I won’t be able to continue on as SDK PTL though. I wish everyone all the best, and I hope life conspires to keep us all connected. Thank you to everyone for an amazing 10 years. Monty

11 10

Team Meeting Agenda for September 8, 2020
by Clark Boylan 08 Sep '20

08 Sep '20

Sorry for the late meeting agenda email. We will meet today, September 8, 2020, in #opendev-meeting at 19:00UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt **** nb03.opendev.org launched to be docker run arm64 nodepool-builder. Needs to be configured. *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. **** We should consider spinning up a new host or modifying the existing host to match productions cinder volume usage for git repos. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** Recurring bogus IPv6 addresses on mirror01.ca-ymq-1.vexxhost.opendev.org (frickler 20200907) *** Likely some glitch in Neutron that allows RAs to pass from other instances which usually should be blocked? *** Pending investigation by mnaser/vexxhost *** Possible mitigation: Configure IPv6 statically and disallow RAs to take effect altogether ** Bup and Borg Backups (clarkb 20200908) *** https://review.opendev.org/741366 is ready to land when we are ready. ** PTG Planning (clarkb 20200908) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200908) *** Wiki updates * Open discussion

1 0

How to use Tox in the flock layer for adding new dashboard plugin for starlingx dashboard
by Kumar, Sharath 08 Sep '20

08 Sep '20

Hi Team, I would like to check with the community regarding use of Tox framework in the flock layer. It is available under distro layer for building changes on openstack_dashboard but missing in the flok layer. Any help how to setup and use tox and its dependencies under flock layer to make changes on starlingx dashboard? Regards, Sharath

1 0

Team Meeting Agenda for September 1, 2020
by Clark Boylan 01 Sep '20

01 Sep '20

We will meet tomorrow, September 1, at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. **** We should consider spinning up a new host or modifying the existing host to match productions cinder volume usage for git repos. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. *** Non master repo HEAD support for new projects has landed to gitea, gerritlib, and jeepyb. I think we can offer this to new projects now. **** Keep in mind that this will likely be a learning experience and we should communicate that to our "volunteers" * General topics ** Setuptools 50 release (clarkb 20200901) *** Changes installation path of packages on Debian and Ubuntu *** Seems to have a number of bugs with python3 < 3.8 ** Bup and Borg Backups (clarkb 20200901) *** https://review.opendev.org/741366 is ready to land when we are ready. ** Making ask.openstack.org read only (clarkb 20200901) *** https://review.opendev.org/#/c/746497/ *** Any reason to not land this now? Feedback has been onboard so far. ** PTG PLanning (clarkb 20200901) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200901) *** Wiki updates * Open discussion

1 0

Meeting Agenda for August 25, 2020
by Clark Boylan 24 Aug '20

24 Aug '20

We will meet August 25, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** https://review.opendev.org/#/c/746335/ add missing files to config management *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Bup and Borg Backups (clarkb 20200825) *** https://review.opendev.org/741366 is ready to land when we are ready. ** github 3rd party ci (clarkb 20200825) *** Update on progress ** Making ask.openstack.org read only (clarkb 20200818) *** https://review.opendev.org/#/c/746497/ ** PTG PLanning (clarkb 20200825) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates * Open discussion

1 0

elastic-reckeck maintenance takeover
by Sorin Sbarnea 18 Aug '20

18 Aug '20

Hi! I would like to request taking over the maintenance of elastic-recheck project. My team (triple-ci), really valued the tool and has a strong interest not only in keeping it alive but also improving it. While the team considered writing our own tool or forking it and changing it for our own use, adding missing features like multiple gerrit servers or multiple issue trackers, I advocated that we should instead become active maintainers and avoid the not-invented-here approach. I already did some work towards making the tool an easy to deploy standalone container. Starting new instances for testing or production should soon be as easy as a one-line command. I am more than happy to act as a liason between openstack-infra and triplo-ci and assure that no changes are made that could negatively impact openstack main deployment. I propose adding tripleo-ci-core gerrit group as core to the project but if there are concerns that the group is too wide, I can also provide a list of people that I trust to follow the expected review process. Best regards, Sorin Sbarnea Red Hat TripleO-CI

5 6

Meeting agenda for August 18, 2020
by Clark Boylan 18 Aug '20

18 Aug '20

We will meet tomorrow, August 18, 2020 in #opendev-meeting at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot containerized. Needs https://review.opendev.org/746181 as final followup. **** https://review.opendev.org/#/c/745595/ fixes for our Gerrit images **** https://review.opendev.org/#/c/746335/ add missing files to config management *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Bup and Borg Backups (clarkb 20200818) *** https://review.opendev.org/741366 is ready to land when we are ready. ** github 3rd party ci (clarkb 20200818) *** Update on progress *** https://review.opendev.org/#/q/topic:opendev-3pci ** Making ask.openstack.org read only (clarkb 20200818) *** https://review.opendev.org/#/c/746497/ ** PTG PLanning (clarkb 20200818) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates * Open discussion

1 0

Meeting Agenda for August 11, 2020
by Clark Boylan 11 Aug '20

11 Aug '20

We will meet August 11, 2020 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot in progress at https://review.opendev.org/#/c/745573/1 and parents **** Our bazel built Gerrit images have some broken plugins ***** javamelody is suffering https://groups.google.com/forum/#!topic/repo-discuss/ls6JV2q6b_g ***** Gerrit 2.16 image cannot find codemirror-editor html files which should be part of the codemirror-editor plugin ***** https://review.opendev.org/745595 pushed to trigger builds and logs so that we can investigate. *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates ** Bup and Borg Backups (clarkb 20200811) *** https://review.opendev.org/741366 is ready to land when we are ready. ** github 3rd party ci (clarkb 20200811) *** Update on progress *** https://review.opendev.org/#/q/topic:opendev-3pci * Open discussion

1 0

Retiring the katacontainers tenant
by Thierry Carrez 04 Aug '20

04 Aug '20

Hi! As you may know, the Kata Containers project has no intent to add jobs to its Zuul tenant to drive its GitHub-based development. We originally set up two jobs (a DCO check and a WIP check) there as an experiment to drive more adoption. Since this did not succeed in convincing them to use more Opendev tooling, those jobs were migrated to GitHub actions a couple of weeks ago, and as a result Zuul no longer runs jobs for Kata GitHub repositories. I'm wondering what's the proper way to retire the katacontainers tenant and zuul job repository (kata-containers/zuul-config). I suspect removing the tenant is as simple as clearing out the tenant definition in project-config:zuul/main.yaml. However there might be a catch-22 there for the retirement of kata-containers/zuul-config. I was thinking about these steps: 1- Push a closing commit to kata-containers/zuul-config (that will remove the job definition for zuul-config itself, but IIRC config projects are not self-testing, so this should work ?) 2- Rename kata-containers/zuul-config to x/kata-zuul-config to clear out the namespace from https://opendev.org/explore/organizations (or should it just be left in place?) 3- Remove the katacontainers tenant definition from project-config:zuul/main.yaml (or should that be run before (2)?) Thanks in advance for your guidance. -- Thierry Carrez (ttx)

2 2

Meeting Agenda for August 4, 2020
by Clark Boylan 04 Aug '20

04 Aug '20

We will meet on August 4, 2020 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Third and final Virtual OpenDev event next week: August 10-11 * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates ** Bup and Borg Backups (clarkb 20200804) *** review request https://review.opendev.org/741366 ** github 3rd party ci (clarkb 20200804) *** Update on progress *** https://review.opendev.org/#/q/topic:opendev-3pci * Open discussion

1 0

Meeting Agenda for July 28, 2020
by Clark Boylan 28 Jul '20

28 Jul '20

We will meet tomorrow, July 28, in #opendev-meeting at 19:00UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine **** Went about 5 days without normal CD jobs running due to Zuul's local code execution update. *** Zuul and Nodepool running on containers with exception of nb03 **** nb03 needs an arm64 docker image. Nodepool disabled attempting to build arm64 images to get kazoo fix release out. **** Need to start docker after afs on executors: https://review.opendev.org/743316 ** OpenDev *** Progress on Governance changes *** Gerrit /p/ mirror deprecation. http://lists.opendev.org/pipermail/service-announce/2020-July/000007.html * General topics ** Trusty Upgrade Progress (clarkb 20200728) *** Wiki updates ** Bup and Borg Backups (clarkb 20200728) *** review request https://review.opendev.org/741366 ** github 3rd party ci (ianw 20200728) *** immediate need is arm64 for pyca/cryptography *** https://github.com/pyca/cryptography/issues/5339 *** http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-07-22.log… internal discussion *** should we? -- in summary conclusion seems to be yes *** how? --- separate tenant to both have isolation and have a config project (API uploads) *** https://review.opendev.org/#/q/topic:opendev-3pci ** Gerrit project CI rework from Google (corvus 20200727) *** Google is abandoning work on checks plugin in favor of new approach *** corvus spoke with Googlers to share our use cases/experiences/requirements * Open discussion

1 0

Meeting agenda for July 21, 2020
by Clark Boylan 21 Jul '20

21 Jul '20

We will meet with this agenda in #opendev-meeting at 19:00 UTC July 21,2020: == Agenda for next meeting == * Announcements ** OpenDev virtual event #2 happening July 20-22 * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine *** Zuul and Nodepool running on containers with exception of nb03 **** nb03 needs an arm64 docker image. Currently building wheels is slow on arm64 and image jobs are timing out. Working out options in https://review.opendev.org/741973 ** OpenDev *** Progress on Governance changes **** Advisory Board welcomed. http://lists.opendev.org/pipermail/service-discuss/2020-July/000057.html *** Gerrit /p/ mirror deprecation. http://lists.opendev.org/pipermail/service-announce/2020-July/000007.html * General topics ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates ** Bup and Borg Backups (clarkb 20200721) *** Status update? ** Project Renames (clarkb 20200721) * Open discussion

1 0

Thank you OpenDev Advisory Board Volunteers
by Clark Boylan 20 Jul '20

20 Jul '20

Hello everyone, Thanks again for volunteering. I've cc'd everyone that volunteered. I think next steps for you will be to sign up to OpenDev's mailing lists if you haven't already: http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce and http://lists.opendev.org/cgi-bin/mailman/listinfo/service-discuss. You may also want to pay attention to https://review.opendev.org/#/q/project:opendev/infra-specs+status:open not necessarily to review every change but to be aware of the big planned updates. service-announce will be low traffic and used to communicate important announcements broadly like http://lists.opendev.org/pipermail/service-announce/2020-July/000007.html. Ideally we'll be getting your input on relevant changes before they end up on service-announce but good to sign up anyway. service-discuss is where the bulk of our email communication will happen. I also expect this is where we'll bring things up with the Advisory Board and vice versa. To help kick things off and properly record initial membership I've sent this to service-discuss as well. Feel free to bring up any other bookkeeping or organizational thoughts in this thread and we'll take it from there. >From a service standpoint some of the things we've been working on recently are Gitea upgrades, containerizing our Zuul and Nodepool deployment, planning a Gerrit upgrade, working on a spec for centralized authentication, and more. Clark

1 0

Meeting Agenda for July 14, 2020
by Clark Boylan 14 Jul '20

14 Jul '20

We will meet with this agenda at 19:00UTC in #opendev-meeting on July 14, 2020: == Agenda for next meeting == * Announcements ** OpenDev virtual event #2 happening July 20-22 * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine *** ze01 running on container again. gear is vendored in the role that uses it now. *** Debugging mismatched multi arch container manifests. ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. Bumped with July 2, 2020 deadline. *** Gitea updates **** Upgrade to v1.12.2 https://review.opendev.org/#/c/740716/ **** Fix for repo listing pagination has merged upstream but hasn't been backported. We can probably just carry our workaround as belts and suspenders. * General topics ** Trusty Upgrade Progress (clarkb 20200714) *** Wiki updates ** Dealing with bup volume rotation and index size (clarkb 20200714) *** Fungi is rotating volumes backing our bup backups *** Does this allow us to clean up and reduce the size of bup indexes? ** Retiring openstack-infra ML July 15 (fungi 20200714) *** http://lists.openstack.org/pipermail/openstack-infra/2020-July/006632.html ** China telecom blocks (ianw/fungi 20200714) *** http://lists.opendev.org/pipermail/service-discuss/2020-July/000053.html ** Project Renames (clarkb 20200714) * Open discussion

1 0

Meeting Agenda for July 7, 2020
by Ian Wienand 07 Jul '20

07 Jul '20

We will meet at 19:00 UTC in #opendev-meeting on June 30, 2020 with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out July 7 * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine *** Zuul executors running on containers; complete? *** Multi-arch container update? ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. Bumped with July 2, 2020 deadline. *** Gitea api troubles (clarkb 20200707) **** upstream fix accepted; track local update? **** Workaround: https://review.opendev.org/#/c/738109/ **** Upstream fix: https://github.com/go-gitea/gitea/pull/12057 * General topics ** Trusty Upgrade Progress (clarkb 20200707) *** Wiki updates ** Time to retire the openstack-infra ML yet? (fungi 20200707) *** http://lists.openstack.org/pipermail/openstack-infra/2020-July/006632.html ** China telecom blocks (ianw/fungi 20200707) *** http://lists.opendev.org/pipermail/service-discuss/2020-July/000053.html *** do we need to continue with this? ** Project Renames (clarkb 20200707) * Open discussion

1 0

Re: [service-announce] Temporarily blocking AS4837 (China Unicom) from Git
by Mohammed Naser 05 Jul '20

05 Jul '20

On Tue, Jun 30, 2020 at 2:30 PM Jeremy Stanley <fungi(a)yuggoth.org> wrote: > > Due to a flood of connections from random addresses within AS4837 > (China Unicom), we have temporarily blocked access to the > opendev.org Git service from all prefixes they're announcing in BGP. > This is not a decision we took lightly, but whatever distributed > system these requests are coming from is not limited to any easily > classified network set. We're also seeing similar activity from > AS4134 (ChinaNet), and so may have to add filtering for their > prefixes too in the near term. Have those blocks been removed or are they still in place? > Hopefully we can lift this block once the problematic traffic > subsides, or as soon as we implement better alternative mitigations. > -- > Jeremy Stanley > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -- Mohammed Naser VEXXHOST, Inc.

2 1

Relinquishing the libra project on PyPI
by Jeremy Stanley 05 Jul '20

05 Jul '20

David, just to confirm our IRC discussion, you'd like us to delete the https://pypi.org/project/libra/ project from PyPI so someone else can take over the name? I've also Cc'd the E-mail addresses listed as "authors" for you and Andrew in the PyPI entry for completeness. Anyway, if you can reply, I'm happy to assist. -- Jeremy Stanley

2 2

Meeting Agenda for June 30, 2020
by Clark Boylan 30 Jun '20

30 Jun '20

We will meet at 19:00 UTC in #opendev-meeting on June 30, 2020 with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out July 7. Will need someone else to run the meeting. * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine *** Zuul executors running on containers ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. Bumped with July 2, 2020 deadline. *** Gitea api troubles. **** Workaround: https://review.opendev.org/#/c/738109/ **** Upstream fix: https://github.com/go-gitea/gitea/pull/12057 * General topics ** DNS cleanup (ianw 20200630) *** https://etherpad.opendev.org/p/rax-dns-openstack-org First pass of record removals has been done. Could use a second pass. *** On the agenda as a reminder for clarkb to do this second cleanup pass. ** Trusty Upgrade Progress (clarkb 20200630) *** Wiki updates ** Time to retire the openstack-infra ML yet? (fungi 20200630) ** Grafana on containers (clarkb 20200630) *** https://review.opendev.org/#/q/status:open+topic:grafana-container * Open discussion

1 0

Forming the OpenDev Advisory Board
by Clark Boylan 30 Jun '20

30 Jun '20

Hello everyone, I've BCC'd potentially interested parties to ensure they get this email, but didn't CC to avoid putting too much pressure on them if I've gauged potential interest poorly. I'm also sure that this list isn't complete. Please share it with others that may be interested. One of the governance bodies that we've described is the OpenDev Advisory Board [0]. The purpose of this group is to work with the OpenDev admins to ensure that project and resource provider needs are being met. Through this group OpenDev can communicate important updates, manage load on cloud resources over time, and plan for future needs. I'm also hopeful that it can serve as a conduit for our collaborating constituents to find time to help administer OpenDev as well. We would like to welcome both projects consuming OpenDev resources and OpenDev resource providers to join the Advisory Board. The OpenDev admins don't want to prescribe how various groups decide who represents them as what works for OpenStack may not work for Zuul or Vexxhost. I don't expect this will be a large time sink, more that we'll have better communication channels when necessary. Once a participating group has decided who their representation should be, it would be great to have that representative reply to this thread publicly and introduce themselves. This way we'll build a roster over time. For Advisory Board process, I'd like to avoid making it heavyweight and suggest using this mailing list (service-discuss(a)lists.opendev.org) with a subject tag of [AdvisoryBoard]. That said, I think the Advisory Board can decide if something else would work better for them once we've started to collect a roster here. [0] https://docs.opendev.org/opendev/system-config/latest/project.html#governan… Clark

7 7

OpenStack from repo Ubuntu 20.04 LTS
by emre brookes 26 Jun '20

26 Jun '20

Not sure if this is the correct place to post this. I installed the OpenStack clients via $ pip2 install python-openstackclient on a new Ubuntu 20.04 LTS system. $ python2 --version Python 2.7.18rc1 (strange the default repos would use a release candidate version) after installing the openstack clients using pip2 and attempting to run the OpenStack command line client "openstack" I received an "ImportError: No module named queue" Following https://github.com/bit4woo/teemo/issues/1 I had to change "import queue" to "from multiprocessing import Queue" in multiple places. e.g. http://codesearch.openstack.org/?q=import%20queue&i=nope&files=&repos= which got me past the error. HTH, -e.

3 2

Meeting Agenda for June 23, 2020
by Clark Boylan 23 Jun '20

23 Jun '20

We will meet in #opendev-meeting at 19:00 UTC June 23, 2020 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. Bumped with July 2, 2020 deadline. * General topics ** Etherpad Upgrade to 1.8.4 or 1.8.5 (fungi/clarkb 20200623) *** https://review.opendev.org/#/c/729029/ *** Now thinking it may be best to do this upgrade after the OpenDev event June 29 - July 2, 2020 ** DNS cleanup (ianw 20200623) *** https://etherpad.opendev.org/p/rax-dns-openstack-org First pass of record removals has been done. Could use a second pass. ** Getting more stuff off of python2 (clarkb 20200623) *** https://etherpad.opendev.org/p/opendev-tools-still-running-python2 *** I'm sure my audit is incomplete. Please add notes if you are aware of other python2 services/tools that need updating. ** Trusty Upgrade Progress (clarkb 20200623) *** Wiki updates * Open discussion

1 0

Meeting Agenda for June 16, 2020
by Clark Boylan 15 Jun '20

15 Jun '20

We will meet in #opendev-meeting at 19:00 UTC Tuesday June 16, 2020 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. Clarkb will send reminders to those that haven't responded as suggested during the PTG. * General topics ** pip-and-virtualenv next steps (ianw 20200616) *** This has merged. Keep an eye out for job failures that need fixes. *** Can we clean up the extra content in the nodepool config and mark the spec as implemented now? ** Zookeeper TLS (corvus 20200616) *** https://review.opendev.org/720302 is ready and will require a full zuul restart ** Etherpad Upgrade to 1.8.4 or 1.8.5 (fungi/clarkb 20200616) *** https://review.opendev.org/#/c/729029/ *** We found a bug in Etherpad's UI in 1.8.4. Previous change works around it if we wish to deploy with a workaround or we can wait for 1.8.5 which is also fixed. ** DNS cleanup (ianw 20200616) *** https://review.opendev.org/728739 : Add tool to export Rackspace DNS domains to bind format *** https://etherpad.opendev.org/p/rax-dns-openstack-org Review output of tool. ** Getting more stuff off of python2 (clarkb 20200616) *** https://etherpad.opendev.org/p/opendev-tools-still-running-python2 *** Many cases where we run python3 capable tools under python2 and they simply need a deployment update *** Meetbot and Logstash tooling not yet python3 capable. *** I'm sure my audit is incomplete. Please add notes if you are aware of other python2 services/tools that need updating. ** Trusty Upgrade Progress (clarkb 20200609) *** Wiki updates * Open discussion

1 0

[donate compute resources] Support for MIPS architecture
by zhaoxiaolin 11 Jun '20

11 Jun '20

Hi, We hope that OpenStack can support for mips architecture, As we are not already cloud operators, we prefer to donate mips based compute resources. I hope you can give us some more suggestions. Thanks

2 1

[donate compute resources] Support for MIPS architecture
by 赵晓琳 11 Jun '20

11 Jun '20

Hi, We hope that OpenStack can support for mips architecture, As we are not already cloud operators, we prefer to donate mips based compute resources. We hope you can give us some more suggestions. Thanks

1 0

June 2020 PTG Recap
by Clark Boylan 08 Jun '20

08 Jun '20

Hello Everyone, Last week we both helped host the PTG (with the meetpad service) and attended the event as participants. Now that the event is over I'll try to recap things for us. On the meetpad side of things we scaled up the Jitsi Video Bridge service from a single instance to five. Based on Cacti data I think this scale up worked well. Unfortunately, it seems that Jitsi Meet is also limited by client side capabilities in addition to the server itself. In particular, having many people in a room with video enabled impacts the performance of that conference in clients. It seems that using Chrome/Chromium, setting your local settings to low bandwidth mode, and simply reducing the number of people using video helps. Overall, I think the service did well considering its recency, and the teams that used it were happy. In general that seemed to be groups with 20 or fewer participants. Since then we've scaled the Jitsi Video Bridge service down to two instances. This should give us reasonable capacity for non PTG periods of time, while consuming a reasonable set of resources. We may also want to look into running the Jibri service which would enable us to stream a conference. This way only active participants would need to fit into that ~20 person limit and lurkers can view via a live stream. Our time at the PTG as participants also went well. We had three 2 hour blocks across a range of time zones which enabled us to talk about a variety of subjects. I'll try to call out notable items in this email, but you can find all of the notes we kept at https://etherpad.opendev.org/p/opendev-virtual-ptg-june-2020 We are starting to dig into what is required to upgrade Gerrit now that it is running on containers. We expect we'll upgrade from 2.13 to 2.16 and then sit there for a while. One reason for this is 2.16 is the last version with the existing UI. 3.0 and beyond drop the existing UI in favor of PolyGerrit UI. Since 2.16 has both UIS this gives us the chance to have people transition over to the new UI. Once we are ready to go to 3.0 that should be a trivial upgrade as the only change was removal of the current existing UI. We do still need to figure out if one large single outage is our best option or if we should upgrade to 2.14, perform online migrations, then upgrade to 2.15, perform online migrations, and finally repeat with an online upgrade to 2.16. To determine which option is best for us we plan to spin up a copy of the production server and try these different options to see how reliable they are and how much time is required to perform each one. To enable more updates of configuration management from Puppet to Ansible, Monty plans to set up Ansible driven testing of our existing puppetry with testinfra. Using that, we can then replace the Puppet with Ansible and be confident things continue to work with our testinfra tests. Hopefully, this makes it easier for more people to get involved in the configuration management updates. Another place where we have "legacy" code that needs updating is porting any existing python2 utilities to python3. I'll try to get a list of those put up so that people can volunteer to do the conversions. The conversions themselves should be straightforward but not everyone knows what needs porting. We hope the list addresses this. Using the data collected in the previous two items we may identify services and tools that should simply be retired rather than ported. One example could be pbx.openstack.org in favor of meetpad.opendev.org. Once we've got lists of work to be done we can evaluate if any of them need to be deleted and can make individual arguments for each one. Over time we've become more and more aware that having a diverse set of identity options for login would make OpenDev more appealing to a broader set of users. One of the big speed bumps to making that change has been enacting a transition that doesn't break existing users. In particular the existing identity service in use, Ubuntu One, requires OpenID v1 and none of the modern identity brokers support this protocols. Jeremy has started to put a spec together on how we might do this, https://review.opendev.org/#/c/731838/, and during the PTG Kristi Nikolla made an excellent suggestion for using SimplySAMLphp to front OpenID v1 for identify brokers that don't speak OpenID v1. With that we now have a plan to move forward on making this a reality. The topic of statistics, metrics, and logging came up again. We asked a few difficult questions that likely need further discussion in separate threads: * How do we make it easier for people to build grafana dashboards? * Should we replace Cacti with something like collectd + graphite/grafana? * Should we consider shutting down the ELK stack since the main user (OpenStack) of this service is not really using it much anymore? The future of Bup for backups was discussed briefly. Bup is still very python2 specific though I think they've recently started to work on a python3 port. I mentioned that I use borg backup for home backups happily, and like Bup, borg claims to support append only backups. Whether or not we want to switch to another tool and what that tool should be if so will require more investigation. Input is welcome. Finally, we discussed generating a bit more involvement in OpenDev from our users. On the Advisory Board side of things I'll be sending out some gentle reminders in the near future to see if we can get anyone else to sign up. Then in a couple weeks move forward with those who have volunteered already. The thought is that starting small and leading by example will drive involvement. We also discussed advertising tools, like the docker image build tooling that OpenDev uses, towards audiences that might be inclined to use them, like Airship. Involvement isn't limited to the Advisory Board or Zuul Jobs. If any of the topics you've just read sound interesting to you and you'd like to get involved please reach out. We're more than happy to incorporate others in what we are doing. Thank you for reading this very long email, Clark

1 0

Meeting Agenda for June 9, 2020
by Clark Boylan 08 Jun '20

08 Jun '20

We will meet at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine **** Split of base and service playbooks is underway. ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. Clarkb will send reminders to those that haven't responded as suggested during the PTG. * General topics ** Project Renames (clarkb 20200609) *** We've got a single project rename queued and now that OpenStack release is done cost of doing renames is lower *** Aiming for June 12, 2020 currently. ** pip-and-virtualenv next steps (ianw 20200609) *** remove from more popular distros *** notify and merge? ** DNS cleanup (ianw 20200609) *** Clearing out old mirrors and lots of old stuff seems to be there *** https://review.opendev.org/728739 : Add tool to export Rackspace DNS domains to bind format *** Should I post the results of the above to a etherpad for group audit (is equivalent of high latency zone xtfr) *** run as backup? ** PTG Recap (clarkb 20200609) *** Meetpad seemed to work well up to 20 conference participants. Client issues with cpu consumption became a problem beyond that. *** Provide your event feedback at https://etherpad.opendev.org/p/June2020-PTG-Feedback *** Decompress with online group activity of some sort? ** Trusty Upgrade Progress (clarkb 20200609) *** Wiki updates * Open discussion

1 0

Retiring several opendev/puppet modules
by Andreas Jaeger 05 Jun '20

05 Jun '20

We've used a lot puppet modules for the OpenDev infrastructure and migrate to ansible, thus we can drop a couple of modules and retire them formally! For details, see https://review.opendev.org/#/c/720901/ and https://review.opendev.org/733850 Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

1 0

Meeting Agenda for June 2, 2020
by Clark Boylan 01 Jun '20

01 Jun '20

This week is PTG week. For that reason many of us are likely busy and distracted. In the past I would have cancelled tomorrow's meeting as we'd have to deal with timezones and local PTG schedules but this week we have a Virtual PTG. However, schedules are still a potential problem. Last week I noted we could meet if enough of us were around and we'd play it by ear. Considering all that, I figure we can have a bit more of a free form agenda if we end up having the meeting tomorrow. I should be around; if you are able to be there as well then we may have enough to talk about. If not, that's fine and have a good PTG. Clark

1 0

Meetpad name
by Amy Marrich 01 Jun '20

01 Jun '20

He all, I named the room for the D&I session - PTGDiversityAndInclusion <https://meetpad.opendev.org/PTGDiversityAndInclusion>/. Not clear from fungi's email whether the meetpad and etherpad need to be renamed or just the linked etherpad. Thanks, Amy (spotz)

2 1

Team meeting agenda for May 26, 2020
by Clark Boylan 26 May '20

26 May '20

We will meet today, May 26, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** PTG happening next week. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine **** Relying on base playbook (and others like letsencrypt playbooks) to complete successfully on all hosts before modifying subsets of hosts can make it difficult to land updates at times. ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. We have some volunteers already! * General topics ** pip-and-virtualenv next steps *** remove from more popular distros (ianw 20200526) *** notify and merge? ** DNS cleanup (ianw 20200526) *** Clearing out old mirrors and lots of old stuff seems to be there *** https://review.opendev.org/728739 : Add tool to export Rackspace DNS domains to bind format *** Should I post the results of the above to a etherpad for group audit (is equivalent of high latency zone xtfr) *** run as backup? ** HTTPS on in region mirrors (clarkb 20200526) *** https://review.opendev.org/730861 Test ssl with mirrors via base-test *** https://review.opendev.org/730862 Use ssl with mirrors in production if base-test is happy ** Scaling Jitsi Meet for Meetpad (clarkb 20200526) *** Ready to launch a jvb01.opendev.org server and test. ** Project Renames (clarkb 20200526) *** We've got a single project rename queued and now that OpenStack release is done cost of doing renames is lower *** Aiming for June 12, 2020 currently. ** Virtual PTG Attendance (clarkb 20200526) *** https://virtualptgjune2020.eventbrite.com Register if you plan to attend. This helps with planning details. *** https://etherpad.opendev.org/p/opendev-virtual-ptg-june-2020 PTG planning *** Do we want to hold this meeting next week? ** Trusty Upgrade Progress (clarkb 20200526) *** Wiki updates * Open discussion

1 0

Team meeting agenda for May 19, 2020
by Clark Boylan 18 May '20

18 May '20

We will be meeting May 19, 2020 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** clarkb out Friday and likely Monday * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine ** OpenDev *** Progress on Governance changes **** https://etherpad.opendev.org/p/XRyf4UliAKI9nRGstsP4 Email draft for building advisory board. * General topics ** pip-and-virtualenv next steps *** remove from more popular distros (ianw 20200519) *** notify and merge? ** DNS cleanup (ianw 20200519) *** Clearing out old mirrors and lots of old stuff seems to be there *** https://review.opendev.org/728739 : Add tool to export Rackspace DNS domains to bind format *** Should I post the results of the above to a etherpad for group audit (is equivalent of high latency zone xtfr) *** run as backup? ** HTTPS on in region mirrors (clarkb 20200519) *** https://review.opendev.org/#/c/728986/ *** Older debuntu don't come with apt support for ssl/tls but bionic/buster and newer do. *** Can probably switch many users of this via base jobs and they'll never know? ** Scaling Jitsi Meet for Meetpad (clarkb 20200519) *** Last friday we had ~22 concurrent users on a meetpad conference. The tool seemed to hold up. *** the Jitsi Video Bridge (jvb) component is the greatest consume of resources (CPU primarily) *** https://review.opendev.org/#/c/729008/ Starting to poke at running more jvbs. ** Project Renames (clarkb 20200519) *** We've got a single project rename queued and now that OpenStack release is done cost of doing renames is lower *** Do we maybe need to wait for PTG to avoid any impact there as well? *** Maybe we should make sure there are no renames that will show up the day after we do this one too? ** Virtual PTG Attendance (clarkb 20200519) *** https://virtualptgjune2020.eventbrite.com Register if you plan to attend. This helps with planning details. *** https://etherpad.opendev.org/p/opendev-virtual-ptg-june-2020 PTG planning ** Trusty Upgrade Progress (clarkb 20200519) *** Wiki updates * Open discussion

1 0

Starlingx/API-Ref review
by Liu, Tao 13 May '20

13 May '20

Hi OpenDev support team, The review below stuck at Build failed (promote pipeline). Can you please check the status and let me know how to fix this issue? https://review.opendev.org/#/c/720589/ Thanks Tao Liu, Member of Technical Staff, Engineering,, Wind River direct 613.963.1413 fax: 613.492.7870 skype: tao_at_home 350 Terry Fox Drive, Suite 200, Ottawa, Ontario, K2K 2W5

3 4

Team Meeting Agenda for May 12, 2020
by Clark Boylan 12 May '20

12 May '20

We will meet with this agenda May 12, 2020 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine **** Improving testing of services pre deployment to ensure we have minimal checks of some sort beyond "ansible ran" **** Managing container process users/uids **** Zuul configs reorged in system-config. Check if you need to rebase. ** OpenDev *** Progress on Governance changes * General topics ** Virtual PTG Attendance (clarkb 20200512) *** https://virtualptgjune2020.eventbrite.com Register if you plan to attend. This helps with planning details. *** https://etherpad.opendev.org/p/opendev-virtual-ptg-june-2020 PTG planning ** Trusty Upgrade Progress (clarkb 20200512) *** Wiki updates * Open discussion

1 0

June Virtual PTG Planning
by Clark Boylan 08 May '20

08 May '20

Hello everyone, I've started to put together some thoughts on what we might discuss during the PTG on this etherpad: https://etherpad.opendev.org/p/opendev-virtual-ptg-june-2020 Feel free to add your thoughts and ideas as well. Once the schedule is published I'll be sure to update the etherpad with details on when and where we are meeting. As we get closer to the event I'll do my best to try and schedule certain topics when interested individuals will be available to discuss them. The time zone problem may make this difficult, but we'll do our best. Clark

1 0

Team Meeting Agenda for May 5, 2020
by Clark Boylan 04 May '20

04 May '20

We will meet May 5, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Meeting now in #opendev-meeting. Same time and day of week. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine **** Deploying Nodepool with containers. **** Reorganizing system-config zuul configs. ** OpenDev *** Progress on Governance changes **** New (old) Service Coordinator * General topics ** Virtual PTG Attendance (clarkb 20200505) *** https://virtualptgjune2020.eventbrite.com Register if you plan to attend. This helps with planning details. ** Trusty Upgrade Progress (clarkb 20200505) *** Wiki updates ** Reviewers plugin for Gerrit (fungi 20200505) *** https://review.opendev.org/#/q/topic:gerrit-plugins+is:open * Open discussion

1 0

Volunteering/Self Nomination for OpenDev Service Coordinator
by Clark Boylan 30 Apr '20

30 Apr '20

As mentioned in my governance mail thread [0] I'm happy to volunteer for OpenDev Service Coordinator. I figured I should send a more explicit nomination email to make it official. As mentioned in that thread my approach to the position has been to try and treat it a bit like project management. We have a lot of coals in the fire currently with deploying more services with ansible and docker images, Gerrit upgrades pending, rebranding of existing services and so on. I'll aim to keep on top of that and do my best to help people pushing those efforts make progress. Basically, more of the same of what I have been doing. That said, it is not too late for other volunteers (though April is quickly ending). I continue to think having a rotation is a good thing. I'm sure others would approach the position differently and that diversity is often very useful. New ideas are helpful. If you don't think you can volunteer, but do have thoughts and ideas, feel free to share them. We don't need new leadership to share good ideas. [0] http://lists.opendev.org/pipermail/service-discuss/2020-April/000001.html Thank you for your consideration, Clark

1 0

Starlingx/docs reviews with no Zuul verification
by Camp, MaryX 30 Apr '20

30 Apr '20

Hi OpenDev support team, There are 2 reviews in the project:starlingx/docs with no Zuul verification. Can you please check the status and let me know how to fix this issue? https://review.opendev.org/#/c/721296/ https://review.opendev.org/#/c/723583/ thanks, Mary Camp PTIGlobal Technical Writer | maryx.camp(a)intel.com<mailto:maryx.camp@intel.com>

3 3

Team Meeting Agenda for April 28, 2020
by Clark Boylan 28 Apr '20

28 Apr '20

We will meet in #opendev-meeting April 28, 2020 at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements ** Meeting now in #opendev-meeting. Same time and day of week. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates *** Zuul as CD engine **** Zuul now ansible dockered and driven by... Zuul **** Improvements to ensuring we run playbook applications against changes in sequence ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-April/000001.html * General topics ** Virtual PTG Attendance (clarkb 20200428) *** Requested: **** Monday 1300-1500 UTC **** Monday 2300-0100 UTC **** Wednesday 0400-0600 UTC ** Trusty Upgrade Progress (clarkb 20200428) *** Wiki updates * Open discussion

1 0

Team Meeting Agenda for April 21, 2020
by Clark Boylan 20 Apr '20

20 Apr '20

We will meet at 19:00 UTC April 21, 2020 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Meeting now in #opendev-meeting. Same time and day of week. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** docker-compose upgraded on all hosts via pip **** Gerrit restarted to replicate to /opt/git and stop replicating changes to github *** Zuul as CD engine **** Now using system-config and project-config from zuul when running ansible playbooks. Note if you need to manually use different checkouts you want to /home/zuul/DISABLE-ANSIBLE to keep zuul from overwriting your checkout details. ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-April/000001.html * General topics ** Virtual PTG Attendance (clarkb 20200421) *** We need to decide if we want to have virtual PTG sessions and for how much time. *** Proposal: **** Monday 1300-1500 UTC **** Monday 2300-0100 UTC **** Wednesday 0400-0600 UTC ** Trusty Upgrade Progress (clarkb 20200421) *** Wiki updates ** Etherpad dockerization, db move, and upgrade. (clarkb 20200421) *** Double check that apache tuning has made service happier. *** Delete etherpad-dev and rely on end to end testing in Zuul instead? * Open discussion

1 0

Infrastructure for StarlingX Sanity
by Zvonar, Bill 20 Apr '20

20 Apr '20

Hi There, The StarlingX community is very interested in making our sanity executable in a fully open virtual environment, to be executed by our test sub-team, or anyone that wants to. Does OpenDev have infrastructure available that we could leverage for this? There are probably many questions to be asked & answered about this - I guess as a starting point, is there someone in particular we can talk to about this? Thanks, Bill...

2 2

Extracting bug metrics from Launchpad
by Jones, Bruce E 17 Apr '20

17 Apr '20

I'm being asked by management to provide bug count metrics for the StarlingX project. Our LP instance is at https://bugs.launchpad.net/starlingx. They are looking for things like "average age of closed bugs", "# bugs closed per week" and so on. There doesn't seem to be a way to gather this kind of data from Launchpad. Some of the items needed like "bug creation date" aren't visible in the UI and may not exist in the database. I can't search using date ranges in the advanced search. If I could export all of the bug info into a spreadsheet I could just pound it out, but I can't see how to do that. Even better would be to extend our project's bitergia page https://starlingx.biterg.io/ to show some of these metrics. But I'm not sure how to do that or if it's even possible. Thoughts and suggestions would be greatly appreciated. Thank you in advance! brucej

2 2

Making Progress with OpenDev Governance
by Clark Boylan 14 Apr '20

14 Apr '20

Hello everyone, It is time to talk governance and start to formalize membership in the structures defined in our documentation [0]. For the Advisory Board my plan is to reach out to specific individuals in our various constituent groups that are likely to be willing to coordinate their representation. For the PTL position consider this a call for volunteers. If you are interested, please indicate that interest via email to this list. I figure we can give it until the end of April to solicit volunteers and if we have more than one can do a condorcet election. Normally I wouldn't suggest so much time, but I think with the transition in communication platforms as well as global pandemic a bit of extra time is warranted. I'm happy to continue [1] to fill this role, but I'm also happy for someone else to take over. Having a rotation seems like a good thing overall, and its starting to feel like we're at a point in the OpenDev transition that we can make changes without much disruption. The way I've been treating the role is its a bit like being a project manager, particularly as we've got quite a few irons in the fire. However, there are probably other ways to approach it, too. Managing core team membership, spec reviews, and incorporating the Advisory Board into decision making will be the specific duties. [0] https://docs.openstack.org/infra/system-config/project.html#governance Note I have changes up to move this to docs.opendev.org as well as retheme with Alabaster. [1] If you consider the OpenStack Infra PTL position to roughly map onto this new role. Let me know what you think, Clark

2 1

Team Meeting Agenda for April 14, 2020
by Clark Boylan 14 Apr '20

14 Apr '20

We will meet in #opendev-meeting at 19:00 on April 14, 2020 with this agenda: == Agenda for next meeting == * Announcements ** Meeting now in #opendev-meeting. Same time and day of week. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Now running in a container. What cleanup is necessary? **** Need to upgrade docker-compose to use stop_grace_period config. *** Zuul as CD engine **** All playbooks are zuul driven now **** Work in progress to run with zuul provided system-config and project-config. This splits Zuul ansible and human run ansible on Bridge. ** OpenDev *** Progress on Governance changes **** http://lists.opendev.org/pipermail/service-discuss/2020-April/000001.html * General topics ** Trusty Upgrade Progress (clarkb 20200414) *** Wiki updates ** Etherpad dockerization, db move, and upgrade. (clarkb 20200414) * Open discussion

1 0

[service-discuss] Team Meeting Agenda for April 7, 2020
by Clark Boylan 07 Apr '20

07 Apr '20

We will meet in #opendev-meeting at 19:00UTC April 7, 2020 with this agenda: == Agenda for next meeting == * Announcements ** Meeting now in #opendev-meeting. Same time and day of week. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates *** splitting manage-projects out into zuul driven playbook *** Zuul as CD engine ** OpenDev *** Progress on Governance changes **** We should probably elect the project lead? **** Need to start formalizing the Advisory Board membership * General topics ** Trusty Upgrade Progress (clarkb 20200407) *** Wiki updates * Open discussion Note this is a new location for the meeting but not a new time. See you there, Clark

1 0