[all][tc][ptl][qa] Migrating upstream CI/CD jobs to Ubuntu Noble (24.04)
Hello Everyone, As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. This is the etherpad which has the migration plan and will be used for the tracking: - https://etherpad.opendev.org/p/migrate-to-noble I have prepared the change(WIP) for base jobs migration on Noble (openstack tox jobs, devstack, and tempest) which will migrate most of the projects jobs. Tempest change[3] include dependency on openstack tox, devstack base jobs changes so testing project gate using the the tempest change will test the complete set. Refer to the etherpad for the complete migration plan/steps. * Deadline: Nov 29th 2024 (R18) is the deadline to test/fix the projects jobs and to merge the base jobs running on Noble (openstck tox, devstack, and tempest). [1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156 -gmann
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote ---
Hello Everyone,
As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2].
This is the etherpad which has the migration plan and will be used for the tracking: - https://etherpad.opendev.org/p/migrate-to-noble
I have prepared the change(WIP) for base jobs migration on Noble (openstack tox jobs, devstack, and tempest) which will migrate most of the projects jobs. Tempest change[3] include dependency on openstack tox, devstack base jobs changes so testing project gate using the the tempest change will test the complete set. Refer to the etherpad for the complete migration plan/steps.
* Deadline: Nov 29th 2024 (R18) is the deadline to test/fix the projects jobs and to merge the base jobs running on Noble (openstck tox, devstack, and tempest).
As you might know, I have been testing the projects on Noble and fixing a few of them. You can find the testing changes in the etherpad[1]. It seems we need more work this time as there are many failures to fix. I will be trying my best to start fixing a few of them (which I can), but many failures need project expertise. Please check your project testing change and fix the issue before the deadline. The deadline is Nov 29th to merge the base devstack/tox base jobs and I do not want to extend it. Below are the known bugs/fixes, which you can refer to if the same failure happens in other repos. In case you encounter more issues, feel free to add those to etherpad[1]. Current Known Bugs/Failures: ------------------------------------ * Ubuntu Noble: Failed to build installable wheels for Pillow: FIXED ** https://bugs.launchpad.net/barbican/+bug/2088355 * Ubuntu Noble: flake8 AttributeError: 'EntryPoints' object has no attribute 'get': FIXED ** https://bugs.launchpad.net/barbican/+bug/2088356 * Ubuntu Noble: cannot install non-Debian-packaged Python package, system wide ** https://bugs.launchpad.net/aodh/+bug/2088358 * Ubuntu Noble: Failed building wheel for python-pcre ** https://bugs.launchpad.net/cinder/+bug/2088359 * Ubuntu Noble: Could not import extension openstackdocstheme (exception: No module named 'distutils') ** https://bugs.launchpad.net/openstacksdk/+bug/2088360 NOTE about doc job: ------------------------ Doc jobs are facing multiple issues (2088355, 2088359, and 2088360) and need fixes across multiple places/toolings. If you want to test your doc job on Noble, use https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935459/ as depends-on [1] https://etherpad.opendev.org/p/migrate-to-noble -gmann
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156
-gmann
---- On Sat, 16 Nov 2024 21:08:52 -0800 Ghanshyam Mann wrote ---
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote ---
Hello Everyone,
As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2].
This is the etherpad which has the migration plan and will be used for the tracking: - https://etherpad.opendev.org/p/migrate-to-noble
I have prepared the change(WIP) for base jobs migration on Noble (openstack tox jobs, devstack, and tempest) which will migrate most of the projects jobs. Tempest change[3] include dependency on openstack tox, devstack base jobs changes so testing project gate using the the tempest change will test the complete set. Refer to the etherpad for the complete migration plan/steps.
* Deadline: Nov 29th 2024 (R18) is the deadline to test/fix the projects jobs and to merge the base jobs running on Noble (openstck tox, devstack, and tempest).
Current Status: ============ Projects Green on Noble: 13 projects (mentioned below). ----------------------------- They need changes/fixes to review/merge, but with those, their gate is green on Noble. * Barbican (Key Manager service) * Blazar (Resource reservation service) * Cyborg (Accelerator Life Cycle Management) * Glance (Image service) * Ironic (Bare Metal service) * Keystone (Identity service) * Kolla (Containerised deployment of OpenStack) * Mistral (Workflow service) * Nova (Compute service) * Openstackansible (Ansible playbooks and roles for deployment) * Openstacksdk (Multi-cloud Python SDK and CLI for End Users) * Storlets (Compute inside Object Storage service) * Sunbeam Link: * https://etherpad.opendev.org/p/migrate-to-noble#L37 Projects Failing on Noble: 26 ---------------------------------- There are still ~26 projects testing are failing. I am checking and fixing the ones which are in my expertise but I would request the project teams to start looking into those. Link: https://etherpad.opendev.org/p/migrate-to-noble#L81 IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken. -gmann
[1] https://etherpad.opendev.org/p/migrate-to-noble
-gmann
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156
-gmann
---- On Mon, 18 Nov 2024 21:17:35 -0800 Ghanshyam Mann wrote ---
---- On Sat, 16 Nov 2024 21:08:52 -0800 Ghanshyam Mann wrote ---
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote ---
Hello Everyone,
As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. . . .
Current Status: ============ * Projects Green on Noble: 19 (mentioned in etherpad). * Projects Failing on Noble: 20 The below projects are failing, and testing change can be found in etherpad: * Adjutant (Operations Processes automation) * Cloudkitty (Rating service) * Freezer (Backup, Restore, and Disaster Recovery service) * Heat (Orchestration service) * Horizon (Dashboard) * Magnum (Container Infrastructure Management service) * Masakari (Instances High Availability service) * Monasca (Monitoring) * Neutron (Networking service) * Octavia (Load-balancer service) * Oslo (Common libraries) * Requirements (Common dependency management) * Skyline (Dashboard) * Swift (Object Storage service) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Venus (Log management service) * Vitrage (RCA (Root Cause Analysis) service) * Zaqar (Message service) * Zun (Containers service) Link: https://etherpad.opendev.org/p/migrate-to-noble IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken. -gmann
-gmann
[1] https://etherpad.opendev.org/p/migrate-to-noble
-gmann
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156
-gmann
I looked into the failure in heat, and it seems the issue is narrowed down to the "RuntimeError: do not call blocking functions from the mainloop" exception raised from eventlet. However I'm running out of ideas of any wrong usage causing it. Quick search explains this may be mostly caused by incomplete monkey_patch-ing but we do it at the first step of each command. I see sys is imported early in the console script generated by setuptools but idk if that can be a cause and also is fix-able. One suspicious thing is that we use threadgroup to run stack update stacks but the implementations are all imported from oslo.services which depend on eventlet (and not any other threading/concurrency mechanism). So I'm wondering if I anyone has better idea about the current issue. My rough view is that the issue is caused by not heat itself but something underlying but I can be wrong as I'm still struggling to understand the full details inside eventlet after some years. Thank you, Takashi On 11/23/24 11:34 AM, Ghanshyam Mann wrote:
---- On Mon, 18 Nov 2024 21:17:35 -0800 Ghanshyam Mann wrote ---
---- On Sat, 16 Nov 2024 21:08:52 -0800 Ghanshyam Mann wrote ---
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote ---
Hello Everyone,
As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. . . .
Current Status: ============
* Projects Green on Noble: 19 (mentioned in etherpad). * Projects Failing on Noble: 20
The below projects are failing, and testing change can be found in etherpad:
* Adjutant (Operations Processes automation) * Cloudkitty (Rating service) * Freezer (Backup, Restore, and Disaster Recovery service) * Heat (Orchestration service) * Horizon (Dashboard) * Magnum (Container Infrastructure Management service) * Masakari (Instances High Availability service) * Monasca (Monitoring) * Neutron (Networking service) * Octavia (Load-balancer service) * Oslo (Common libraries) * Requirements (Common dependency management) * Skyline (Dashboard) * Swift (Object Storage service) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Venus (Log management service) * Vitrage (RCA (Root Cause Analysis) service) * Zaqar (Message service) * Zun (Containers service)
Link: https://etherpad.opendev.org/p/migrate-to-noble
IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken.
-gmann
-gmann
[1] https://etherpad.opendev.org/p/migrate-to-noble
-gmann
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156
-gmann
---- On Fri, 22 Nov 2024 18:34:08 -0800 Ghanshyam Mann wrote ---
---- On Mon, 18 Nov 2024 21:17:35 -0800 Ghanshyam Mann wrote ---
---- On Sat, 16 Nov 2024 21:08:52 -0800 Ghanshyam Mann wrote ---
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote ---
Hello Everyone,
As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. . . .
Updating the current status as a few projects are fixed (Fix is up for review). NOTE: Deadline for this migration is approaching: 3 days away (Nov 29th) Current Status: ============ Projects Green on Noble: 27 (mentioned in etherpad). Projects Failing on Noble: 14 (listed below) * Heat (Orchestration service) * Horizon (Dashboard) * Magnum (Container Infrastructure Management service) * Masakari (Instances High Availability service) * Monasca (Monitoring) * Neutron (Networking service) * Octavia (Load-balancer service) * Oslo (Common libraries) * Requirements (Common dependency management) * Skyline (Dashboard) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Vitrage (RCA (Root Cause Analysis) service) * Zaqar (Message service) Link: https://etherpad.opendev.org/p/migrate-to-noble IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken. -gmann
-gmann
-gmann
[1] https://etherpad.opendev.org/p/migrate-to-noble
-gmann
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156
-gmann
---- On Tue, 26 Nov 2024 20:43:35 -0800 Ghanshyam Mann wrote ---
---- On Fri, 22 Nov 2024 18:34:08 -0800 Ghanshyam Mann wrote ---
---- On Mon, 18 Nov 2024 21:17:35 -0800 Ghanshyam Mann wrote ---
---- On Sat, 16 Nov 2024 21:08:52 -0800 Ghanshyam Mann wrote ---
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote ---
Hello Everyone,
As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. .
Updating the current status as a few projects are fixed (Fix is up for review). NOTE: Deadline for this migration is approaching: 1 days away (Nov 29th) Current Status: ============ Projects Green on Noble: 32 (+5 from previous status. mentioned in etherpad). Projects Failing on Noble: 10 (listed below) * Heat (Orchestration service) * Horizon (Dashboard) * Octavia (Load-balancer service) * Oslo (Common libraries) * QA : devstack-plugin-ceph * Skyline (Dashboard) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Vitrage (RCA (Root Cause Analysis) service) * Zaqar (Message service) Link: https://etherpad.opendev.org/p/migrate-to-noble IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken. -gmann
Updating the current status as a few projects are fixed (Fix is up for review).
NOTE: Deadline for this migration is approaching: 3 days away (Nov 29th)
Current Status: ============
Projects Green on Noble: 27 (mentioned in etherpad).
Projects Failing on Noble: 14 (listed below) * Heat (Orchestration service) * Horizon (Dashboard) * Magnum (Container Infrastructure Management service) * Masakari (Instances High Availability service) * Monasca (Monitoring) * Neutron (Networking service) * Octavia (Load-balancer service) * Oslo (Common libraries) * Requirements (Common dependency management) * Skyline (Dashboard) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Vitrage (RCA (Root Cause Analysis) service) * Zaqar (Message service)
Link: https://etherpad.opendev.org/p/migrate-to-noble
IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken.
-gmann
-gmann
-gmann
[1] https://etherpad.opendev.org/p/migrate-to-noble
-gmann
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156
-gmann
---- On Thu, 28 Nov 2024 21:35:21 -0800 Ghanshyam Mann wrote ---
---- On Tue, 26 Nov 2024 20:43:35 -0800 Ghanshyam Mann wrote ---
---- On Fri, 22 Nov 2024 18:34:08 -0800 Ghanshyam Mann wrote ---
---- On Mon, 18 Nov 2024 21:17:35 -0800 Ghanshyam Mann wrote ---
---- On Sat, 16 Nov 2024 21:08:52 -0800 Ghanshyam Mann wrote ---
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote ---
Hello Everyone,
As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. .
Updating the current status as a few projects are fixed (Fix is up for review).
NOTE: Deadline for this migration is approaching: 1 days away (Nov 29th)
Current Status: ============ Base jobs migration to Noble is completed, and they are running on Noble now (except doc job) We still have below 9 projects that need fixes to be green on the Noble. Projects Failing on Noble: 9 (listed below) * Heat (Orchestration service) * Horizon (Dashboard) * Octavia (Load-balancer service) * Oslo (Common libraries) * QA : devstack-plugin-ceph * Skyline (Dashboard) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Zaqar (Message service) Link: https://etherpad.opendev.org/p/migrate-to-noble - https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... -gmann
IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken.
-gmann
Updating the current status as a few projects are fixed (Fix is up for review).
NOTE: Deadline for this migration is approaching: 3 days away (Nov 29th)
Current Status: ============
Projects Green on Noble: 27 (mentioned in etherpad).
Projects Failing on Noble: 14 (listed below) * Heat (Orchestration service) * Horizon (Dashboard) * Magnum (Container Infrastructure Management service) * Masakari (Instances High Availability service) * Monasca (Monitoring) * Neutron (Networking service) * Octavia (Load-balancer service) * Oslo (Common libraries) * Requirements (Common dependency management) * Skyline (Dashboard) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Vitrage (RCA (Root Cause Analysis) service) * Zaqar (Message service)
Link: https://etherpad.opendev.org/p/migrate-to-noble
IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken.
-gmann
-gmann
-gmann
[1] https://etherpad.opendev.org/p/migrate-to-noble
-gmann
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... [3] https://review.opendev.org/c/openstack/tempest/+/932156
-gmann
---- On Sat, 30 Nov 2024 13:24:53 -0800 Ghanshyam Mann wrote ---
---- On Thu, 28 Nov 2024 21:35:21 -0800 Ghanshyam Mann wrote ---
---- On Tue, 26 Nov 2024 20:43:35 -0800 Ghanshyam Mann wrote ---
---- On Fri, 22 Nov 2024 18:34:08 -0800 Ghanshyam Mann wrote ---
---- On Mon, 18 Nov 2024 21:17:35 -0800 Ghanshyam Mann wrote ---
---- On Sat, 16 Nov 2024 21:08:52 -0800 Ghanshyam Mann wrote ---
---- On Mon, 21 Oct 2024 07:03:11 -0700 Ghanshyam Mann wrote --- > Hello Everyone, > > As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better > tracking and speeding up this work, it is the community-wide goal for this cycle[2]. .
Updating the current status as a few projects are fixed (Fix is up for review).
NOTE: Deadline for this migration is approaching: 1 days away (Nov 29th)
Current Status: ============ Base jobs migration to Noble is completed, and they are running on Noble now (except doc job), 38 projects are good on Noble. Below are two tasks pending for this migration to complete: 1. Doc job to be migrated to Noble ----------------------------------------- * https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935459 * Below are bugs that need to be fixed: ** https://bugs.launchpad.net/pbr/+bug/2088360 ** https://bugs.launchpad.net/cinder/+bug/2088359 2. The below 3 projects need fixes ----------------------------------------- * Heat (Orchestration service) ** heat team is checking the failure * QA : devstack-plugin-container ** devstack-plugin-container-k8s (and Tacker k8s jobs) failing to install CRI-O on Noble *** https://bugs.launchpad.net/tacker/+bug/2090859 ** hongbin is checking the failure. * Skyline (Dashboard) ** The existing gate is broken which fix is up + pep8 job green on noble but team seems not active on review or debugging failure Link: https://etherpad.opendev.org/p/migrate-to-noble -gmann
Current Status: ============
Base jobs migration to Noble is completed, and they are running on Noble now (except doc job)
We still have below 9 projects that need fixes to be green on the Noble.
Projects Failing on Noble: 9 (listed below) * Heat (Orchestration service) * Horizon (Dashboard) * Octavia (Load-balancer service) * Oslo (Common libraries) * QA : devstack-plugin-ceph * Skyline (Dashboard) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Zaqar (Message service)
Link: https://etherpad.opendev.org/p/migrate-to-noble
- https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
-gmann
IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken.
-gmann
Updating the current status as a few projects are fixed (Fix is up for review).
NOTE: Deadline for this migration is approaching: 3 days away (Nov 29th)
Current Status: ============
Projects Green on Noble: 27 (mentioned in etherpad).
Projects Failing on Noble: 14 (listed below) * Heat (Orchestration service) * Horizon (Dashboard) * Magnum (Container Infrastructure Management service) * Masakari (Instances High Availability service) * Monasca (Monitoring) * Neutron (Networking service) * Octavia (Load-balancer service) * Oslo (Common libraries) * Requirements (Common dependency management) * Skyline (Dashboard) * Tacker (NFV Orchestration service) * Telemetry (Telemetry service) * Vitrage (RCA (Root Cause Analysis) service) * Zaqar (Message service)
Link: https://etherpad.opendev.org/p/migrate-to-noble
IMP: Deadline: Nov 29th 2024 (R18) to migrate the devstack/tempest/tox-based jobs to Noble and after that project gate who have not fixed the failure will be broken.
-gmann
-gmann
-gmann
[1] https://etherpad.opendev.org/p/migrate-to-noble
-gmann
> > [1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html > [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubuntu... > [3] https://review.opendev.org/c/openstack/tempest/+/932156 > > -gmann > > >
participants (2)
-
Ghanshyam Mann
-
Takashi Kajinami