[tc][ptl][quality-assurance][infra] Switching default node type to ubuntu-bionic
We've committed to testing[*] on Ubuntu 18.04 LTS (Bionic Beaver) by the Stein release, but it's nearly upon us and we still have quite a few jobs, particularly Python 2.7 unit tests for most projects, running on older Ubuntu 16.04 LTS (Xenial Xerus) images instead. I propose that next Wednesday, March 13, we update the default node type in our Zuul job configurations from ubuntu-xenial to ubuntu-bionic for master branches of our projects. Note that most projects are already running Python 3.6 unit testing which happens on ubuntu-bionic images and DevStack/Tempest testing is already happening on ubuntu-bionic as well, so the switch for any job definitions not calling an explicit node type should be a non-event. Nevertheless, I wanted to start a brief discussion on the list as we are coming close to RC target week and it would be good to have the cut-over happen before then as some projects may not see any additional changes in Stein after their first RC tag. [*] https://governance.openstack.org/tc/reference/runtimes/stein.html -- Jeremy Stanley
Jeremy Stanley <fungi@yuggoth.org> writes:
We've committed to testing[*] on Ubuntu 18.04 LTS (Bionic Beaver) by the Stein release, but it's nearly upon us and we still have quite a few jobs, particularly Python 2.7 unit tests for most projects, running on older Ubuntu 16.04 LTS (Xenial Xerus) images instead. I propose that next Wednesday, March 13, we update the default node type in our Zuul job configurations from ubuntu-xenial to ubuntu-bionic for master branches of our projects.
Note that most projects are already running Python 3.6 unit testing which happens on ubuntu-bionic images and DevStack/Tempest testing is already happening on ubuntu-bionic as well, so the switch for any job definitions not calling an explicit node type should be a non-event. Nevertheless, I wanted to start a brief discussion on the list as we are coming close to RC target week and it would be good to have the cut-over happen before then as some projects may not see any additional changes in Stein after their first RC tag.
[*] https://governance.openstack.org/tc/reference/runtimes/stein.html -- Jeremy Stanley
+1 from me for moving ahead on this. -- Doug
On 2019-03-07 14:55:01 +0000 (+0000), Jeremy Stanley wrote:
We've committed to testing[*] on Ubuntu 18.04 LTS (Bionic Beaver) by the Stein release, but it's nearly upon us and we still have quite a few jobs, particularly Python 2.7 unit tests for most projects, running on older Ubuntu 16.04 LTS (Xenial Xerus) images instead. I propose that next Wednesday, March 13, we update the default node type in our Zuul job configurations from ubuntu-xenial to ubuntu-bionic for master branches of our projects. [...]
Change https://review.openstack.org/641897 and its dependencies will implement the proposed change in default node type. -- Jeremy Stanley
On 2019-03-08 01:54:52 +0000 (+0000), Jeremy Stanley wrote:
On 2019-03-07 14:55:01 +0000 (+0000), Jeremy Stanley wrote:
We've committed to testing[*] on Ubuntu 18.04 LTS (Bionic Beaver) by the Stein release, but it's nearly upon us and we still have quite a few jobs, particularly Python 2.7 unit tests for most projects, running on older Ubuntu 16.04 LTS (Xenial Xerus) images instead. I propose that next Wednesday, March 13, we update the default node type in our Zuul job configurations from ubuntu-xenial to ubuntu-bionic for master branches of our projects. [...]
Change https://review.openstack.org/641897 and its dependencies will implement the proposed change in default node type.
This has now merged. See another recent post[*] I made on the thread with things to watch out for and suggested fixes. I expect a number of projects are going to need to adjust job parents or set explicit nodesets for some project-specific py35 jobs on their master branches, and may need some non-tox/non-devstack jobs they're running on their stable branches adjusted to include branch-filtered variants with ubuntu-xenial nodesets too. [*] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003746.htm... -- Jeremy Stanley
On 2019-03-13 22:01:08 +0000 (+0000), Jeremy Stanley wrote:
On 2019-03-08 01:54:52 +0000 (+0000), Jeremy Stanley wrote: [...]
Change https://review.openstack.org/641897 and its dependencies will implement the proposed change in default node type.
This has now merged. See another recent post[*] I made on the thread with things to watch out for and suggested fixes. I expect a number of projects are going to need to adjust job parents or set explicit nodesets for some project-specific py35 jobs on their master branches, and may need some non-tox/non-devstack jobs they're running on their stable branches adjusted to include branch-filtered variants with ubuntu-xenial nodesets too.
[*] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003746.htm...
A quick update on initial fixes and problems we've identified which will hopefully be addressed quickly: Projects using the openstack-python35-jobs-no-constraints project-template on their master (or stable/stein) branches are currently insta-failing tox-35 builds due to running on ubuntu-bionic nodes which lack Python3.5. I have proposed an openstack-zuul-jobs change[1] to solve this. Sorry for not catching it before the transition. The fix[2] for nova-tox-functional-py35 to reparent it to a job which already runs on Xenial included a superfluous depends-on which prevented it from going into the gate once approved, so an edit has been pushed up which will hopefully merge soon. An variety of dashboard projects are relying on nodesource packages of NodeJS 4.x, which don't exist for Ubuntu Bionic. Monty has started another discussion thread[3] to look into options we can pursue, but in the short term is working on pinning those jobs to ubuntu-xenial nodes to give us more time to work out a better solution. [1] https://review.openstack.org/643180 [2] https://review.openstack.org/642940 [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003800.htm... -- Jeremy Stanley
On 2019-03-13 23:19:03 +0000 (+0000), Jeremy Stanley wrote: [...]
An variety of dashboard projects are relying on nodesource packages of NodeJS 4.x, which don't exist for Ubuntu Bionic. Monty has started another discussion thread to look into options we can pursue, but in the short term is working on pinning those jobs to ubuntu-xenial nodes to give us more time to work out a better solution.
Which is https://review.openstack.org/643182 (now approved, should take effect shortly). -- Jeremy Stanley
On 2019-03-13 23:19:03 +0000 (+0000), Jeremy Stanley wrote: [...]
The fix for nova-tox-functional-py35 to reparent it to a job which already runs on Xenial included a superfluous depends-on which prevented it from going into the gate once approved, so an edit has been pushed up which will hopefully merge soon. [...]
Due to a Catch-22 with another in-flight fix, this has now been squashed into https://review.openstack.org/643129 instead. -- Jeremy Stanley
On Wed, 13 Mar 2019 23:57:53 +0000, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2019-03-13 23:19:03 +0000 (+0000), Jeremy Stanley wrote: [...]
The fix for nova-tox-functional-py35 to reparent it to a job which already runs on Xenial included a superfluous depends-on which prevented it from going into the gate once approved, so an edit has been pushed up which will hopefully merge soon. [...]
Due to a Catch-22 with another in-flight fix, this has now been squashed into https://review.openstack.org/643129 instead.
I have fast-approved ^ to unblock the gate. -melanie
On 2019-03-07 14:55:01 +0000 (+0000), Jeremy Stanley wrote:
We've committed to testing[*] on Ubuntu 18.04 LTS (Bionic Beaver) by the Stein release, but it's nearly upon us and we still have quite a few jobs, particularly Python 2.7 unit tests for most projects, running on older Ubuntu 16.04 LTS (Xenial Xerus) images instead. I propose that next Wednesday, March 13, we update the default node type in our Zuul job configurations from ubuntu-xenial to ubuntu-bionic for master branches of our projects. [...]
As a quick follow-up, this change[1] is still scheduled for tomorrow. Here are couple of possible pain points for which you'll want to prepare: Any jobs which ultimately descend from the openstack-tox job definition[2] get a variant[3] on stable branches for ocata through rocky which forces use of the old ubuntu-xenial nodeset. If you have a job which needs to use ubuntu-xenial on stable branches and it doesn't descend from openstack-tox (or devstack, or legacy-base where similar branch filters are being applied) then you will need to create a similar variant of it or one of its parents to do the same. Further, any jobs which descend from one of the shared "py35" job definitions in the openstack-zuul-jobs repository have been patched[4] to force them to run on ubuntu-xenial even when triggered for a change on the master branch of a project. If you have a job which relies on Python 3.5 (or any other versioned tool which is no longer provided in Bionic) you will want to implement a similar nodeset modification for it or in one of its parents, or consider reparenting it on one of the shared "py35" definitions already provided in openstack-zuul-jobs. For those who are interested in checking this in advance, I've prepared a do-not-merge stack of job changes which descend from a clone of our base job with tomorrow's default node change already applied so that I could confirm the various nodeset pins and branch filters are working as intended. You can create a DNM change in your project with "Depends-On: https://review.openstack.org/642536" in a commit message footer and see what results Zuul reports. Note that this will only really provide useful results for jobs parented from the unittests[5] definition in the zuul-jobs repository (but that covers a vast majority of the unknown job set so is a good place to start). If you're trying this, remember to make unobtrusive changes to enough different kinds of files in your DNM change to overcome any possible file filters on some of the jobs you run. Also it may be a good idea to try it on a stable branch (if you have any) in addition to master just to make sure node selection is working correctly for your jobs there. I prepared a couple of tests against openstack/nova[6][7] which you can see for an example. [1] https://review.openstack.org/641897 [2] http://zuul.opendev.org/t/openstack/job/openstack-tox [3] https://opendev.org/openstack-infra/openstack-zuul-jobs/commit/a53afda [4] https://opendev.org/openstack-infra/openstack-zuul-jobs/commit/8507565 [5] https://review.openstack.org/642536 [6] https://review.openstack.org/642580 [7] https://review.openstack.org/642582 -- Jeremy Stanley
On 2019-03-12 19:08:23 +0000 (+0000), Jeremy Stanley wrote: [...]
If you have a job which relies on Python 3.5 (or any other versioned tool which is no longer provided in Bionic) you will want to implement a similar nodeset modification for it or in one of its parents, or consider reparenting it on one of the shared "py35" definitions already provided in openstack-zuul-jobs. [...]
Just a reminder, we're probably only a few hours away from when jobs like this will start breaking on master branches of projects. See my earlier message[1] for more detail, but when you encounter these the fixes are likely to be very trivial[2]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003746.htm... [2] https://review.openstack.org/642940 -- Jeremy Stanley
participants (3)
-
Doug Hellmann
-
Jeremy Stanley
-
melanie witt