[tripleo][ci] socializing upstream job removal ( master upgrade, scenario010 )
Greetings, Expect to see several of these emails as we propose additional items to reduce the upstream resource footprint of TripleO. The intent of the email is to broadcast our general intent regarding specific ci jobs with some details regarding the how and when things will happen. Expect jobs to be removed in a staged pragmatic manner. Historical Context [1] Please feel free to respond to this thread with your opinions. Summary: First stage of TripleO's job reduction will be to remove most TripleO upgrade jobs on master and remove all scenario010 octavia jobs from upstream. *Upgrade jobs in master:* All the upgrade jobs in master are non-voting based on policy and to give the upgrade developers time to compensate for new developments and features. The feedback provided by CI can still occur in our periodic jobs in RDO's software factory zuul. Upgrade jobs will remain running in periodic but removed from upstream. Specifically: ( master ) tripleo-ci-centos-8-standalone-upgrade tripleo-ci-centos-8-undercloud-upgrade tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades I'll note there is interest in keeping the undercloud-upgrade, however we are able to remove all the upgrade jobs for a branch in the upstream we can also remove a content-provider job. I would encourage folks to consider undercloud-upgrade for periodic only. *Scenario010 - Octavia:* Scenario010 octavia has been non-voting and not passing at a high enough rate [2] to justify the use of upstream resources. I would propose we only run these jobs in the periodic component and integration lines in RDO softwarefactory. Specifically: ( all branches ) tripleo-ci-centos-8-scenario010-ovn-provider-standalone tripleo-ci-centos-8-scenario010-standalone Please review and comment. The CI team will start taking action on these two items in one week. Thank you! [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020235.... [2] http://dashboard-ci.tripleo.org/d/iEDLIiOMz/non-voting-jobs?orgId=1&from=now-30d&to=now
On 9 Feb 2021, at 16:29, Wesley Hayutin <whayutin@redhat.com<mailto:whayutin@redhat.com>> wrote: Upgrade jobs in master: All the upgrade jobs in master are non-voting based on policy and to give the upgrade developers time to compensate for new developments and features. The feedback provided by CI can still occur in our periodic jobs in RDO's software factory zuul. Upgrade jobs will remain running in periodic but removed from upstream. Specifically: ( master ) tripleo-ci-centos-8-standalone-upgrade tripleo-ci-centos-8-undercloud-upgrade tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades I'll note there is interest in keeping the undercloud-upgrade, however we are able to remove all the upgrade jobs for a branch in the upstream we can also remove a content-provider job. I would encourage folks to consider undercloud-upgrade for periodic only. +1 I think this makes sense. These jobs are long running and consume quite a few VM over that lengthy period of time.
On Tue, Feb 9, 2021 at 6:46 PM Jesse Pretorius <jesse@odyssey4.me> wrote:
On 9 Feb 2021, at 16:29, Wesley Hayutin <whayutin@redhat.com> wrote:
*Upgrade jobs in master:* All the upgrade jobs in master are non-voting based on policy and to give the upgrade developers time to compensate for new developments and features. The feedback provided by CI can still occur in our periodic jobs in RDO's software factory zuul. Upgrade jobs will remain running in periodic but removed from upstream.
Specifically: ( master ) tripleo-ci-centos-8-standalone-upgrade tripleo-ci-centos-8-undercloud-upgrade tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades
I'll note there is interest in keeping the undercloud-upgrade, however we are able to remove all the upgrade jobs for a branch in the upstream we can also remove a content-provider job. I would encourage folks to consider undercloud-upgrade for periodic only.
+1 I think this makes sense. These jobs are long running and consume quite a few VM over that lengthy period of time.
just to be clear however, note that this means that there will be no stable/wallaby version of these upgrades jobs, or at least it will be a lot harder to add them if we aren't running these jobs against master. In the 'normal' case, the non-voting master jobs become the voting stable/latest once we have created the new latest branch. If we remove master we may have a challenge to add voting stable/wallaby versions when the time comes. Based on our recent discussions, we don't *want* stable/wallaby versions since we want to eventually remove all the upstream upgrade jobs once d/stream replacements have been put in place. The point then is that ideally we need to be in a position to do that (i.e. replacement jobs in place) before stable/wallaby is branched, otherwise we may have a tough time adding stable/wallaby versions of the (removed) master upgrade jobs. regards, marios
On 2021-02-09 19:03:58 +0200 (+0200), Marios Andreou wrote: [...]
just to be clear however, note that this means that there will be no stable/wallaby version of these upgrades jobs, or at least it will be a lot harder to add them if we aren't running these jobs against master. In the 'normal' case, the non-voting master jobs become the voting stable/latest once we have created the new latest branch. If we remove master we may have a challenge to add voting stable/wallaby versions when the time comes.
Based on our recent discussions, we don't *want* stable/wallaby versions since we want to eventually remove all the upstream upgrade jobs once d/stream replacements have been put in place.
The point then is that ideally we need to be in a position to do that (i.e. replacement jobs in place) before stable/wallaby is branched, otherwise we may have a tough time adding stable/wallaby versions of the (removed) master upgrade jobs.
If for some reason you aren't able to get that far in Wallaby, running this once a day in the periodic pipeline in OpenDev's Zuul would still be *far* less resource-intensive than the current state of running multiple times for most proposed changes, and so could serve as a reasonable compromise. -- Jeremy Stanley
On Tue, Feb 9, 2021 at 7:19 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2021-02-09 19:03:58 +0200 (+0200), Marios Andreou wrote: [...]
just to be clear however, note that this means that there will be no stable/wallaby version of these upgrades jobs, or at least it will be a lot harder to add them if we aren't running these jobs against master. In the 'normal' case, the non-voting master jobs become the voting stable/latest once we have created the new latest branch. If we remove master we may have a challenge to add voting stable/wallaby versions when the time comes.
Based on our recent discussions, we don't *want* stable/wallaby versions since we want to eventually remove all the upstream upgrade jobs once d/stream replacements have been put in place.
The point then is that ideally we need to be in a position to do that (i.e. replacement jobs in place) before stable/wallaby is branched, otherwise we may have a tough time adding stable/wallaby versions of the (removed) master upgrade jobs.
If for some reason you aren't able to get that far in Wallaby, running this once a day in the periodic pipeline in OpenDev's Zuul would still be *far* less resource-intensive than the current state of running multiple times for most proposed changes, and so could serve as a reasonable compromise.
ack thanks and yeah we are definitely keeping our 3rd party periodics on these but good point we could also consider the upstream periodic i almost forgot we have one of those defined too ;) https://opendev.org/openstack/tripleo-ci/src/commit/8b23749692c03f2acfb67218... regards, marios
-- Jeremy Stanley
Hi, Jesse Pretorius <jesse@odyssey4.me> writes:
On 9 Feb 2021, at 16:29, Wesley Hayutin <whayutin@redhat.com> wrote:
Upgrade jobs in master: All the upgrade jobs in master are non-voting based on policy and to give the upgrade developers time to compensate for new developments and features. The feedback provided by CI can still occur in our periodic jobs in RDO's software factory zuul. Upgrade jobs will remain running in periodic but removed from upstream.
Specifically: ( master ) tripleo-ci-centos-8-standalone-upgrade tripleo-ci-centos-8-undercloud-upgrade tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades
I'll note there is interest in keeping the undercloud-upgrade, however we are able to remove all the upgrade jobs for a branch in the upstream we can also remove a content-provider job. I would encourage folks to consider undercloud-upgrade for periodic only.
+1 I think this makes sense. These jobs are long running and consume quite a few VM over that lengthy period of time.
My personal feeling about undercloud-upgrade is that I haven't used it in a while, so this would be a +1 for removing this one as well (beside all previous discussion) as it would remove an extra content provider (cost/gain ...). Jesse is that also ok for you ? Thanks, -- Sofer Athlan-Guyot chem on #irc@rhos-upgrades DFG:Upgrades Squad:Update
On Tue, Feb 9, 2021 at 5:32 PM Wesley Hayutin <whayutin@redhat.com> wrote:
*Scenario010 - Octavia:* Scenario010 octavia has been non-voting and not passing at a high enough rate [2] to justify the use of upstream resources. I would propose we only run these jobs in the periodic component and integration lines in RDO softwarefactory.
Specifically: ( all branches ) tripleo-ci-centos-8-scenario010-ovn-provider-standalone tripleo-ci-centos-8-scenario010-standalone
We are still working on fixing scenario010, there were several huge changes in octavia-tempest-plugin that broke scenario010 but two weeks ago we fixed tripleo-ci-centos-8-scenario010-standalone by reducing the number of tests we are running (only smoke tests and perhaps more tests in the future) and by adding some new options for non-devstack environments in octavia-tempest-plugin. Then tripleo-ci-centos-8-scenario010-ovn-provider-standalone failed for another reason and the test class has been added in tempest-skiplist (even for non-ovn-provider jobs), and now all scenario010 jobs are in FAILURE because tempest doesn't run any test ( https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z... ). We still want to have those scenario010 jobs, we first need to remove the skiplist for the non-ovn-provider job, then we will figure out why the ovn-provider job fails.
Please review and comment. The CI team will start taking action on these two items in one week.
Thank you!
[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020235.... [2] http://dashboard-ci.tripleo.org/d/iEDLIiOMz/non-voting-jobs?orgId=1&from=now-30d&to=now
participants (6)
-
Gregory Thiemonge
-
Jeremy Stanley
-
Jesse Pretorius
-
Marios Andreou
-
Sofer Athlan-Guyot
-
Wesley Hayutin