June 2025 - service-discuss - lists.openstack.org

Team Meeting Agenda for July 1, 2025
by Clark Boylan 01 Jul '25

01 Jul '25

We will meet with this agenda on July 1, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Much of the job configs in OpenDev have converted to using labels provided by niz/zuul-launcher as of June 30. *** Be on the lookout for unexpected behaviors *** At some point we'll need to plan for nodepool cleanup. *** Image builds are plagued by gnutls connection issues here and there. One appraoch is to have git fetches retry in image builds. **** https://review.opendev.org/c/openstack/diskimage-builder/+/721581 ** Gerrit shutdown problems (clarkb 20250527) *** This was almost certainly related to our h2 compaction time increase change. *** We reverted that change and when restarting to pick up the revert I manually ran kill -HUP against Gerrit. It took ages again. I ran lsof and strace and we could see that Gerrit was reading, writing, and syncing to one of the h2 cache files. *** I expect SIGINT to work via docker compose and podman at this point without issue as h2 compaction should be quick. *** We'll just continue to delete the large h2 cache files during restarts. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** clarkb has replaced the zookeeper cluster with a new triple (zk01-03) *** corvus replaced all zuul servers (reused existing names) *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** Clarkb to write a spec explaining the motivation for a move to matrix for opendev comms. *** Then idea is to do a bit of a trial and see if it does help users who primarily rely on matrix for IRC as well as new users. ** Pre PTG Planning (clarkb 20250624) *** OpenStack release is October 1 *** OpenInfra Summit is October 17-19 *** PTG is October 27-31 *** Would 2-3 days the week of October 6-10 work for us to have some focused OpenDev Pre PTG time? * Open discussion

1 0

Request to Clear Previous Commits and Start Fresh
by 田玉桐 24 Jun '25

24 Jun '25

I am tianyutong, responsible for the project heterogeneous-distributed-training-framework.(cfn/heterogeneous-distributed-training-framework: Heterogeneous distributed training framework researches the training method and framework over heterogeneous hardware. - heterogeneous-distributed-training-framework - OpenDev: Free Software Needs Free Tools) I attempted to overwrite the previous commits on the following branches: master, release_1.0.0, and core_0.8.0. However, due to merge conflicts, the commit history became corrupted, and the format is now incorrect. I need assistance in clearing all the previous commits and starting fresh. I would like to push new commits without keeping any of the previous history. Could you please guide me on how to remove the old commits and push the new changes? Best regards, tianyutong

2 1

Re:Re:Re: Authorization and Branch Creation in OpenDev Repo
by 田玉桐 24 Jun '25

24 Jun '25

Dear OpenDev Community, I am Tianyutong from China Mobile, responsible for the project cfn/heterogeneous-distributed-training-framework. I encountered a permission issue while trying to create a new branch, and I need your assistance to resolve it. Issue Description When attempting to create a branch named "release_1.0.0" via the OpenDev web interface (following the official documentation), I received the following error: An error occurred Error 403 (Forbidden): not permitted: create on refs/heads/release_1.0.0 Endpoint: /projects/*/branches/* The error screenshot is attached for reference. Request Could you please grant our team permission to create new branches in cfn/heterogeneous-distributed-training-framework? We need this privilege to manage version branches for our project. Thank you for your support! Best regards, Tianyutongtianyutong(a)sd.chinamobile.com ----邮件原文---- 发件人："赵奇慧" <zhaoqihui(a)chinamobile.com> 收件人：fungi <fungi(a)yuggoth.org> 抄　送: "黄蕾" <huangleiyjy(a)chinamobile.com>,tianyutong <tianyutong(a)sd.chinamobile.com>,chenzhengwei <chenzhengwei(a)sd.chinamobile.com>,qihuiz93 <qihuiz93(a)163.com> 发送时间：2025-06-12 09:32:52 主题：Re:Re: Authorization and Branch Creation in OpenDev Repo Hi Jeremy, Thank you so much for your kind help and advice. We'll try with the link. Best, Qihui --------------------------------------- 赵奇慧中国移动通信研究院网络所手机：13810659120 邮箱：zhaoqihui(a)chinamobile.com ----邮件原文---- 发件人：fungi 收件人："赵奇慧" 抄　送: "黄蕾" ,tianyutong ,chenzhengwei ,qihuiz93 发送时间：2025-06-11 20:50:29 主题：Re: Authorization and Branch Creation in OpenDev Repo On 2025-06-11 17:18:48 +0800 (+0800), 赵奇慧 wrote: > This is Qihui Zhao from China Mobile and OIF CFN WG. We write this > email to ask about how to get management and branch creation > authorization on repo > cfn/heterogeneous-distributed-training-framework. [...] I believe the documentation you're looking for is here: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#branches In the future, I recommend asking questions on the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network, since there are many people in both places who could answer and I'm often not available due to other obligations. Hope that helps! -- Jeremy Stanley Subject：Re: Authorization and Branch Creation in OpenDev Repo On 2025-06-11 17:18:48 +0800 (+0800), 赵奇慧 wrote: > This is Qihui Zhao from China Mobile and OIF CFN WG. We write this > email to ask about how to get management and branch creation > authorization on repo > cfn/heterogeneous-distributed-training-framework. [...] I believe the documentation you're looking for is here: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#branches In the future, I recommend asking questions on the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network, since there are many people in both places who could answer and I'm often not available due to other obligations. Hope that helps! -- Jeremy Stanley

3 4

Team Meeting Agenda for June 24, 2025
by Clark Boylan 24 Jun '25

24 Jun '25

We will meet on June 24, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Auto hold functionality is rolling out *** Are there any other required features before all OpenDev tenants switch to use zuul launcher managed resources? ** Gerrit shutdown problems (clarkb 20250527) *** After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL *** https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out *** Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default) *** Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. *** clarkb is in the process of rolling out a new mirror-update with new zookeepers to follow *** corvus is replacing zuul servers over the next week or two *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** Clarkb to write a spec explaining the motivation for a move to matrix for opendev comms. *** Then idea is to do a bit of a trial and see if it does help users who primarily rely on matrix for IRC as well as new users. ** Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) *** https://review.opendev.org/c/openstack/diskimage-builder/+/949942 DIB functional testing without Nodepool **** This will allow us to have greater control of the devstack environment allowing us to configure cpu features that CentOS 10 Stream requires *** https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream ** projects.yaml normalization *** a (possibly intentional) regression in ruamel.yaml has started adding spaces to the end of wrapped strings, akin to format=flowed E-mail wrapping **** https://sourceforge.net/p/ruamel-yaml/tickets/546/ *** this is impacting our normalization job output **** https://review.opendev.org/952006 ** Pre PTG Planning (clarkb 20250624) *** OpenStack release is October 1 *** OpenInfra Summit is October 17-19 *** PTG is October 27-31 *** Would 2-3 days the week of October 6-10 work for us to have some focused OpenDev Pre PTG time? * Open discussion

1 0

Team Meeting Agenda for June 17, 2025
by Jeremy Stanley 16 Jun '25

16 Jun '25

We will meet with this agenda on June 17, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) ** Gerrit shutdown problems (clarkb 20250527) *** After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL *** https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out *** Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default) *** Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://github.com/matrix-org/matrix-appservice-irc/issues/1851 *** This means that matrix bridge users don't see messages from newer IRC nicks/users *** Options: **** Declare bankruptcy: Matrix needs a Matrix client and IRC needs an IRC client **** Pay EMS for a bridge. This is apparently billed per active IRC bridge user and could be a large increase on current EMS hosting cost **** Host our own bridge. https://postmarketos.org/blog/2025/03/31/matrix-bridge-migration/ has useful related info **** Embrace Matrix and move there. This represents a compromise between those who want IRC and those who want something like Slack/Discord. But for both sides the important functionality/needs seems to be preserved. (IRC users continue to get to use open source tooling with federated servers. Slack/Discord users get a system that runs over HTTPS using clients with modern bells and whistles). ** Enabling hashtags globally (corvus 20250523) *** Agreed to apply this the day of the meeting (June 3, 2025). *** Announced at https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) *** https://review.opendev.org/c/opendev/glean/+/941672 Glean NetworkManager Keyfile support *** https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream *** There have been a few struggles **** CentOS 10 Stream requires x86-64-v3 which clouds other than rax classic can provide (or about only 40% of our capacity). **** CentOS 10 Stream upstream images label the / partition as /boot with a partition uuid. This seems like an upstream bug. centos-minimal element mitigates this. ** projects.yaml normalization *** a (possibly intentional) regression in ruamel.yaml has started adding spaces to the end of wrapped strings, akin to format=flowed E-mail wrapping **** https://sourceforge.net/p/ruamel-yaml/tickets/546/ *** this is impacting our normalization job output **** https://review.opendev.org/952006 * Open discussion -- Jeremy Stanley

1 0

Team Meeting Agenda for June 10, 2025
by Jeremy Stanley 10 Jun '25

10 Jun '25

We will meet with this agenda on June 10, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) ** Gerrit shutdown problems (clarkb 20250527) *** After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL *** https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out *** Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default) *** Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://github.com/matrix-org/matrix-appservice-irc/issues/1851 *** This means that matrix bridge users don't see messages from newer IRC nicks/users *** Options: **** Declare bankruptcy: Matrix needs a Matrix client and IRC needs an IRC client **** Pay EMS for a bridge. This is apparently billed per active IRC bridge user and could be a large increase on current EMS hosting cost **** Host our own bridge. https://postmarketos.org/blog/2025/03/31/matrix-bridge-migration/ has useful related info **** Embrace Matrix and move there. This represents a compromise between those who want IRC and those who want something like Slack/Discord. But for both sides the important functionality/needs seems to be preserved. (IRC users continue to get to use open source tooling with federated servers. Slack/Discord users get a system that runs over HTTPS using clients with modern bells and whistles). ** Enabling hashtags globally (corvus 20250523) *** Agreed to apply this the day of the meeting (June 3, 2025). *** Announced at https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) *** https://review.opendev.org/c/opendev/glean/+/941672 Glean NetworkManager Keyfile support *** https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream *** There have been a few struggles **** CentOS 10 Stream requires x86-64-v3 which clouds other than rax classic can provide (or about only 40% of our capacity). **** CentOS 10 Stream upstream images label the / partition as /boot with a partition uuid. This seems like an upstream bug. centos-minimal element mitigates this. * Open discussion -- Jeremy Stanley

1 0

Team Meeting Agenda for June 3, 2025
by Clark Boylan 03 Jun '25

03 Jun '25

We will meet with this agenda on June 3, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) ** Gerrit shutdown problems (clarkb 20250527) *** After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL *** https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out *** Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default) *** Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://github.com/matrix-org/matrix-appservice-irc/issues/1851 *** This means that matrix bridge users don't see messages from newer IRC nicks/users *** Options: **** Declare bankruptcy: Matrix needs a Matrix client and IRC needs an IRC client **** Pay EMS for a bridge. This is apparently billed per active IRC bridge user and could be a large increase on current EMS hosting cost **** Host our own bridge. https://postmarketos.org/blog/2025/03/31/matrix-bridge-migration/ has useful related info **** Embrace Matrix and move there. This represents a compromise between those who want IRC and those who want something like Slack/Discord. But for both sides the important functionality/needs seems to be preserved. (IRC users continue to get to use open source tooling with federated servers. Slack/Discord users get a system that runs over HTTPS using clients with modern bells and whistles). ** Enabling hashtags globally (corvus 20250523) *** Agreed to apply this the day of the meeting (June 3, 2025). *** Announced at https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) *** https://review.opendev.org/c/opendev/glean/+/941672 Glean NetworkManager Keyfile support *** https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream *** There have been a few struggles **** CentOS 10 Stream requires x86-64-v3 which clouds other than rax classic can provide (or about only 40% of our capacity). **** CentOS 10 Stream upstream images label the / partition as /boot with a partition uuid. This seems like an upstream bug. centos-minimal element mitigates this. * Open discussion

1 0