- service-discuss - lists.openstack.org

Team Meeting Agenda for April 22, 2025
by Clark Boylan 22 Apr '25

22 Apr '25

We will meet with this agenda on April 22, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. *** Differences discovered in handling of private_ip vars for ipv4 and ipv6. Nodepool would set the private_ip to the public_ip if there is no private_ip. NIZ does not do that currently. *** One idea is to leave the NIZ behavior in place for private_ip but then add a new var that can interpret the value for jobs. *** We have volunteers to port image builds from nodepool to niz. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Switching Gerrit to run on Review03 (clarkb 20250401) *** https://etherpad.opendev.org/p/i_vt63v18c3RKX2VyCs3 Notes on the migration plan *** Servers switched on April 21, 2025 *** There is a bit of followup to do when we are satisfied we won't roll back: **** https://review.opendev.org/c/opendev/system-config/+/947758 Drop review02 from inventory **** https://review.opendev.org/c/opendev/system-config/+/947759 Drop ignored docker compose version specifier **** https://review.opendev.org/c/opendev/system-config/+/882900 Migrate gerrit images to quay *** The last change for the quay move may be exposing a problem with our assumptions around testing with quay that we should double check with lodgeit and hound. *** Now it is time to start thinking about the Gerrit 3.11 upgrade. **** https://www.gerritcodereview.com/3.11.html ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. ** Moving hound image to quay.io (clarkb 20250415) *** Now that codesearch is a Noble node running with podman we can host the hound image on quay and still have speculative testing. *** https://review.opendev.org/c/opendev/system-config/+/947010 *** See note about in the Gerrit topic. We should be careful with this update to ensure the testing is doing what we expect. ** Renewing wiki's cert (clarkb 20250422) *** Wiki's cert expires soon. We will need to manually provision a new annual cert and apply it to the server. * Open discussion

1 0

Image availability in OpenDev
by James E. Blair 19 Apr '25

19 Apr '25

Hi, The OpenDev community needs volunteers to help with the maintenance of the following images, otherwise we may need to remove them from Zuul: * CentOS Stream (x86 and Arm) * Rocky Linux (x86 and Arm) * Ubuntu (Arm only) * Debian (Arm only) The preceding images are all active and in use; the following images do not currently build and without further action are already slated to be removed: * openEuler * Gentoo The Zuul project is nearing the end of an effort to update the way that it launches nodes when running tests. This promises to bring a number of benefits to OpenDev users, including better responsiveness, availability, scaling, and more end-user control over the configuration. In short, we are removing Nodepool as a standalone project and incorporating all of its functionality into Zuul itself. OpenDev currently builds the base images we use for test nodes using nodepool-builder, which is a standalone daemon running on a small number of dedicated hosts which runs diskimage-builder and uploads the results to the clouds we use. This approach ensures a consistent image across the many clouds that we employ, plus effective caching of content used in jobs. The configuration for these image builds is, as is standard for every part of OpenDev, public and changes to it are able to be proposed by any member of our community. Yet there is limited testing of those changes, and new image builds can only happen after configuration changes are merged and deployed. With the new system, image builds will happen in Zuul jobs, which makes it far easier to observe the builds and examine the results. Most importantly, any changes to the image configuration are entirely self-testing. Which leads us to the request for volunteers. We have already copied over and adapted the x86 Debian and Ubuntu images to the new system. But we need someone with knowledge of Arm, CentOS, and Rocky images to port the old configuration to the new system. It's not a lot of work, but from time to time these images do need updates (for new OS releases, for example), so it's important that we have a community of users who are able to assist in that maintenance. If no volunteers are willing to help maintain this community resource, we may need to remove support for these images. Starting around May 15, we would consider these images best-effort: we may not run them in all providers as we move more of our workload to zuul-launcher. Jobs that use them may start to fail with NODE_ERROR. Later, once we finish the move to zuul-launcher (tentatively July 1), any request for nodes without images would return NODE_ERROR. Along with myself, others in the OpenDev community are willing to assist anyone who is interested in working on this. It is an excellent opportunity to get involved in helping to support the community: it requires no previous knowledge of OpenDev systems or any special access. If you are willing to port the nodepool-builder configuration for these images to the new Zuul system, please let us know (by either responding to this thread on service-discuss, or contacting us in OFTC in #opendev). Here are some relevant links to get you started: The current nodepool-builder configuration: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nod… The new zuul-launcher configuration: https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/image-b… Thanks in advance, -Jim (on behalf of the OpenDev project)

3 2

Team Meeting Agenda for April 15, 2025
by Clark Boylan 15 Apr '25

15 Apr '25

Welcome back from our week "off" (not really as the PTG was demanding out time instead). We will have our weekly meeting this week on April 15, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. *** VHD support has been added to image builds. Node leak cleanups are getting added. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Switching Gerrit to run on Review03 (clarkb 20250401) *** A new BFV server of the same size as the old server has been booted in vexxhost ca-ymq-1 *** https://etherpad.opendev.org/p/i_vt63v18c3RKX2VyCs3 Notes on the migration plan *** We have been working through the process of spinning up and testing the new server *** https://review.opendev.org/c/opendev/system-config/+/947044 Prep work to ensure port 29418 ssh host keys match on old and new servers *** April 21, 2025 at 1600 UTC is the planned outage time to migrate. *** DNS updates are slightly tricky with Gerrit being the server that is updated ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. ** Moving hound image to quay.io (clarkb 20250415) *** Now that codesearch is a Noble node running with podman we can host the hound image on quay and still have speculative testing. *** https://review.opendev.org/c/opendev/system-config/+/947010 *** We can do similar with Gerrit after running on the new host and not rolling back * Open discussion

1 0

Team Meeting Agenda for April 1, 2025
by Clark Boylan 01 Apr '25

01 Apr '25

We will meet with this agenda on April 1, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack releases Epoxy 2025.1 Wednesday. Please be aware of this as you make changes ** The PTG is being hosted on Meetpad April 7-11. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Booting a new Gerrit server (clarkb 20250401) *** Its about time to start preparing the new Gerrit server. We have a decent amount of experience with Noble now and it seems to work in several clouds *** Clarkb thinks we should stick to the same flavor size wherever we host. This likely means Vexxhost is still the best candidate location despite the IPv6 issues *** If we boot in Vexxhost we should use the equivalent v3 flavor so that we don't have to boot from volume (this simplifies server rescue should we need it). *** Maybe aim for swapping the new server into production the week after the PTG? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. * Open discussion

1 0

Team Meeting Agenda for March 25, 2025
by Clark Boylan 25 Mar '25

25 Mar '25

We will meet with this agenda on March 25, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Dropping uWSGI (clarkb 20250318) *** frickler provided a workaround for building uwsgi containers (we dropped arm64 container builds since we don't use uwsgi on arm) *** clarkb would still like to switch lodgeit to granian but that is a lower priority now. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** clarkb replaced all nodepool builders and the osuosl mirror *** Currently working through updating the nodepool launchers *** Please grab a host or three and work on updating them if you can. This is an activity where we can't boil the ocean so every little bit of help is important. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Upgrading to Gitea 1.23.6 (clarkb 20250325) *** New release of gitea is available. We should update *** https://review.opendev.org/c/opendev/system-config/+/945414 ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. * Open discussion

1 0

Zuul errors
by ronald.stone＠windriver.com 18 Mar '25

18 Mar '25

Apologies if this is the wrong group. Zuul appears to be broken. See https://zuul.opendev.org/t/openstack/build/179c40f7e382474cb9c676b812caa6fb

2 1

Team Meeting Agenda for March 18, 2025
by Clark Boylan 18 Mar '25

18 Mar '25

We will meet on March 18, 2025 at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. *** OVH said March 17 (today) was not good for them and they would be in touch to reschedule later. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/c/opendev/system-config/+/944799 drop python 3.10 image builds *** https://review.opendev.org/c/opendev/system-config/+/944789 rebuild python 3.11 and 3.12 base images *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Dropping uWSGI (clarkb 20250318) *** uWSGI has never built reliable and we are hitting a new segfault during compilation *** uWSGI is also on life support and not getting new development just bugfixes here and there *** Thinking about switching to Granian which is an ASGI/RSGI/WSGI server written in Rust that publishes wheels for aarch64 and x86_64 *** In theory this means we should be able to get away with one server regardless of the protocol that web services we choose to run use. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint *** nb05 and nb06 are in the process of getting deployed on Noble with podman and docker compose. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Packaging updates for bindep (fungi 20250318) *** https://review.opendev.org/938570 Drop requirements.txt *** https://review.opendev.org/940711 Drop auxiliary requirements files *** It would be nice to get consensus on whether or not to include these when replicating the new packaging methods to our other tools. * Open discussion

1 0

Team Meeting Agenda for March 11, 2025
by Clark Boylan 11 Mar '25

11 Mar '25

We will meet with this agenda on March 11, 2025 at 19:00 UTC (note that North America and maybe others have switched to DST) in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. *** Plan is to target March 17 for OVH flavor surgery ** Running infra-prod Jobs in Parallel on Bridge (clarkb 20250304) *** https://review.opendev.org/c/opendev/system-config/+/943488 Actually run infra-prod jobs in parallel *** The refactor to infra-prod jobs has gone in but with a semaphore limit of 1 which continues to serialize things *** There were some small issues. **** bootstrap-bridge needed extra required-projects to setup repos like project-config in periodic pipelines **** project-config and dns zone repos needed to be setup with bootstrap-bridge as the parent job to ensure repos are configured and locks are respected *** Seems like things work serially at this point. Time to proceed with parallel execution? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for March 4, 2025
by Clark Boylan 04 Mar '25

04 Mar '25

We will meet with this agenda on March 4, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Redeploying raxflex resources (clarkb 20250225) *** We have access to a new raxflex region. To align with the old region and to correct network MTUs we need to redeploy the old region on the new project/tenant. *** https://review.opendev.org/q/hashtag:flex-dfw3 ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. ** Running infra-prod Jobs in Parallel on Bridge (clarkb 20250304) *** https://review.opendev.org/c/opendev/system-config/+/942439 *** This change refactors how we run infra-prod-* jobs on bridge to enable running them in parallel *** The current iteration will continue to run them serially with an additional semaphore which can be expanded from a single lock to 2 or 5 etc when we're happy with it *** Need to pick a good day when we are able to observe the behavior of the system in order to catch issues as soon as possible. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Docker Hub rate limits (clarkb 20250304) *** Docker hub announced they would be changing rate limits from 100 pulls per 6 hours to 10 pulls per hour on March 1, 2025 *** This could be good for us if the shorter reset period works well with our zuul jobs as they tend to run for some time but not 6 hours *** Or it could be bad for us. *** Early data doesn't show any indicators for it being better or worse yet. *** Also keep in mind ipv6 addresses are rate limited per /64. We might consider forcing docker hub access through ipv4 via /etc/hosts entries. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Help w/ storyboard issue
by Khalil, Ghada 26 Feb '25

26 Feb '25

Hi all, I hope someone here can point me to the right contact to help with this issue. I had previously created a storyboard: https://storyboard.openstack.org/#!/story/2011332 -- titled: Platform Application Maintenance - ptp-notification It was accessible. We were able to create tasks and link them to code reviews previously: - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/938075 - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/940343 However, for some reason, we now get an API error saying the story is not found. 404: GET /api/v1/stories/2011332: Story 2011332 not found Thanks, Ghada Khalil StarlingX Release Prime

2 1