- service-discuss - lists.openstack.org

Infra Team Meeting Agenda for September 21, 2021
by Clark Boylan 21 Sep '21

21 Sep '21

We will meet on September 21, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210921) *** lists.openstack.org is running focal now **** Kernel packages have been pinned. If you update the kernel you need to decompress it. **** Fungi is working on a mailman3 spec. We can decide the best route for replacing the server with that context. ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Stack at https://review.opendev.org/c/opendev/system-config/+/809510 migrates services to the new asset locations. ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 ** Gerrit replication "leaks" (clarkb 20210921) *** We've added a 15 minute network timeout to the gerrit config. Need to restart the server to pick that up. ** InMotion cloud has been scaled up (clarkb 20210921) *** We need to work on tuning the over subscription ratios. * Open discussion

1 0

Mailing lists offline 2021-09-12 for server upgrade
by Jeremy Stanley 14 Sep '21

14 Sep '21

On Sunday, September 12, the OpenDev sysadmins will be performing operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Expect extended outages for message delivery and access to archives between 15:00 UTC and 21:00 UTC, though messages sent to the mailing lists at these sites should queue normally and be delivered automatically after maintenance activities have concluded. If you have any questions, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley

1 2

Team Meeting Agenda for September 14, 2021
by Clark Boylan 14 Sep '21

14 Sep '21

We will meet September 14, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210914) *** lists.openstack.org is running focal now **** Number of manual edits made to services captured in https://review.opendev.org/c/opendev/system-config/+/808570 **** Ran into issues with Xen being able to boot lz4 compressed kernels **** We decompressed the kernel but that still failed **** We installed grub-xen then set up a chainload situation to a xen specific grub preloader via /boot/grub/menu.lst **** This chainload situation may not need a compressed kernel? If it doesn't this server may be viable long term and we can document things. **** We should consider replacing the server to get away from Xen PV weirdness and to maybe use a floating IP which can be moved between upgraded instances in the future. ** Improving OpenDev's CD throughput (clarkb 20210914) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210914) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210914) *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. ** Working with InMotion Friday to increase IP addr allocation and scale up (clarkb 20210914) *** Will try to debug instance launch failures there prior to Friday. ** Scheduling Gerrit project renames (clarkb 20210914) *** We've got a few of these showing up now. We should look at a calendar and pencil in some plans. * Open discussion

1 0

Team meeting agenda for September 7, 2021
by Clark Boylan 07 Sep '21

07 Sep '21

We will meet with this agenda September 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210907) *** lists.openstack.org upgrade scheduled for September 12 beginning 15:00UTC ** Improving OpenDev's CD throughput (clarkb 20210907) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210907) *** 33 conflicts remain. Need to investigate and email their users about planned next steps. ** OpenDev logo hosting (clarkb 20210907) *** https://review.opendev.org/c/opendev/system-config/+/805932/ ready to switch to using the assets container for gitea *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. * Open discussion

1 0

Team Meeting Agenda for August 31, 2021
by Clark Boylan 31 Aug '21

31 Aug '21

Sorry, this is going out a little late this week. We will meet on August 31, 2021 (today) at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Feature Freeze has begun. * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210831) *** lists.kc.io has been upgraded to Focal. Seems to be happy. *** Do resource consumption numbers look different compared to pre upgrade state? *** Time to schedule a lists.o.o upgrade? ** Improving OpenDev's CD throughput (clarkb 20210831) *** Quick check on how things look after our quick and easy updates. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210831) *** Need to delete external ids for the recent batch of acount retirements. ** OpenDev logo hosting (clarkb 20210831) *** https://etherpad.opendev.org/p/logo-locations *** Are we happy with an approach yet? * Open discussion

1 0

Infra Team Meeting Agenda for August 24, 2021
by Clark Boylan 24 Aug '21

24 Aug '21

We will meet on August 24, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Zuul is now on our matrix homeserver at #zuul:opendev.org ** OpenStack Feature Freeze starts next week * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210824) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o *** Will coordinate with Kata to do the upgrade this week hopefully. ** Improving OpenDev's CD throughput (clarkb 20210824) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/805487 Be more specific about when inventory changes force jobs to run. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210824) *** Need to delete external ids for the recent batch of acount retirements. ** Gitea 1.15.0 upgrade (clarkb 20210824) *** This release exists now *** How do we want to handle the opendev logo image hosting for the gerrit theme? * Open discussion

1 0

Team Meeting Agenda for August 17, 2021
by Clark Boylan 17 Aug '21

17 Aug '21

We will meet with this agenda on August 17, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210817) *** Clark was the only nomination. Takes the seat by default. ** Matrix eavesdrop and gerritbot bots (clarkb 20210817) *** Zuul's next steps ** Gitea01 backups to one of two servers are unhappy (clarkb 20210817) *** Reboots didn't help *** On Friday we had a set of servers that backup at the same time with similar issues but to the other backup server. Doesn't seem to be a persistent issue. ** Mailman Ansible and Server Upgrades (clarkb 20210817) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o ** Improving OpenDev's CD throughput (clarkb 20210817) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/804795 Run cloud launcher daily not hourly. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. * Open discussion

1 0

Service Coordinator Nominations and Election
by Clark Boylan 11 Aug '21

11 Aug '21

Hello everyone, It has been about six months since the previous election for the OpenDev Service Coordinator spot. When we did the last round I said we could do about 2 weeks of nomination period then follow that with a week long election if necessary. Considering this the official open call for nominations which will stay open until 23:59 UTC on August 10, 2021. If we need an election that will run from 00:01 UTC August 11, 2021 to 23:59 UTC August 17, 2021 UTC. If anyone else is remotely interested please feel free to step up. I've been doing this for a bit now and would be happy to have someone else take on the role. Let me know if you have questions about what is required. I am also more than happy to help in any transition that happens. Looking ahead to the next election I think we'll run those with nominations January 25, 2022 - February 8, 2022. The election would run February 9, 2022 - February 15, 2022. Clark

3 3

Team meeting agenda for August 10, 2021
by Clark Boylan 10 Aug '21

10 Aug '21

We will meet August 10, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210810) *** Nominations end 23:59 UTC on August 10, 2021. *** That is about 4 hours after the meeting ends. ** review upgrade (ianw 20210810) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Project Renames (clarkb 20210810) *** Need to update docs ** Matrix eavesdrop and gerritbot bots (clarkb 20210810) *** Debugging matrix gerritbot connection problems to review02. *** What are the next steps? ** Gitea01 backups to one of two servers are unhappy (clarkb 20210810) *** Reboots didn't help *** Now lists has the same problem? ** Gitea 1.15.0 prep *** Updating our Gitea management to only use the REST API https://review.opendev.org/c/opendev/system-config/+/803367/ ** Mailman Ansible and Server Upgrades (clarkb 20210810) *** Should we use lists.kc.io as the snapshot image and new test booted node? *** Need to pick this work up again. * Open discussion

1 0

Team Meeting Agenda for August 3, 2021
by Clark Boylan 03 Aug '21

03 Aug '21

Hello, We will meet with this agenda on August 3, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210803) *** Nominations end 23:59 UTC on August 10, 2021. ** review upgrade (ianw 20210803) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Gerrit User Cleanups (clarkb 20210803) *** Next batch of cleanups proposed in files on review02 *** Hoping this is the second to last batch with last batch getting direct cleanups that we can push. ** Project Renames (clarkb 20210803) *** Rename went well *** Need to update docs *** Need to manage ssh known_hosts for review02 users https://review.opendev.org/c/opendev/system-config/+/802922 *** Further testing improvements https://review.opendev.org/c/opendev/system-config/+/803266 ** Matrix eavesdrop and gerritbot bots (clarkb 20210803) *** Deployments are in progress ** Gitea01 backups to one of two servers are unhappy (clarkb 20210727) *** Reboots didn't help ** Gitea 1.15.0 prep *** Upstream no longer compatible with nodejs 16 to build the front end https://review.opendev.org/c/opendev/system-config/+/803265 *** Working through changes to gitea in https://review.opendev.org/c/opendev/system-config/+/803231 ** Mailman Ansible and Server Upgrades (clarkb 20210803) *** Need to fix newlist command invocation for the prod servers https://review.opendev.org/c/opendev/system-config/+/803263 *** Should we use lists.kc.io as the snapshot image and new test booted node? *** Need to pick this work up again. * Open discussion

1 0