[dev][infra][qa][tact-sig] Tox 4.0.0 breaking changes
Tox 4.0.0, a major rewrite, just appeared on PyPI in the past couple of hours, so jobs are breaking right and left. We're going to try pinning to latest 3.x in zuul-jobs for the near term while we work through some of the more obvious issues with it, but until then lots of tox-based Zuul jobs are going to be broken. I'll follow up to this thread once we think things should be stabilized and we can start trying to pick through what's going to need fixing for the new tox version.
On 2022-12-07 19:19:42 +0000 (+0000), Jeremy Stanley wrote:
Tox 4.0.0, a major rewrite, just appeared on PyPI in the past couple of hours, so jobs are breaking right and left. We're going to try pinning to latest 3.x in zuul-jobs for the near term
[...]
The ensure-tox version pin in zuul-jobs is in place and seems to be working for now, so it should be safe to recheck any jobs which failed from this. We'll continue trying to figure out what new tox is breaking in jobs, but help is always appreciated of course!
Another update... The ensure-tox role provided as part of the zuul-jobs standard library is going to undo its tox<4 cap on December 21, so jobs which are impacted will need to be fixed or get their own independent caps added before that date. See this announcement for further details:
https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/mess...
Projects which want to test-drive their fixes for tox v4 before that date should be able to set a commit footer like:
Depends-On: https://review.opendev.org/866943
---- On Thu, 08 Dec 2022 09:59:42 -0800 Jeremy Stanley wrote ---
It seems master jobs using tox<4[1] but all stable jobs using tox>=4.0[2] and failing. And both using ensure-tox
[1] https://zuul.opendev.org/t/openstack/build/8a4585f2961a4854ad96c7d2a188b557/... [2] https://zuul.opendev.org/t/openstack/build/78cddc1180ff40109bbe17df884d23d8/...
Another update... The ensure-tox role provided as part of the zuul-jobs standard library is going to undo its tox<4 cap on December 21, so jobs which are impacted will need to be fixed or get their own independent caps added before that date. See this announcement for further details:
Does it mean all the stable jobs also will start using the latest tox (>4.0) which is breaking the jobs and we need to fix them in stable branches too?
-gmann
https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/mess...
Projects which want to test-drive their fixes for tox v4 before that date should be able to set a commit footer like:
Depends-On: https://review.opendev.org/866943-- Jeremy Stanley
On 2022-12-08 12:44:17 -0800 (-0800), Ghanshyam Mann wrote: [...]
It seems master jobs using tox<4[1] but all stable jobs using tox>=4.0[2] and failing. And both using ensure-tox
[1] https://zuul.opendev.org/t/openstack/build/8a4585f2961a4854ad96c7d2a188b557/... [2] https://zuul.opendev.org/t/openstack/build/78cddc1180ff40109bbe17df884d23d8/...
[...]
Those are Tempest jobs, which do some of their own installing of tox directly rather than just relying on the version provided by ensure-tox. Check with the QA team, since they've been working on fixes for those.
Does it mean all the stable jobs also will start using the latest tox (>4.0) which is breaking the jobs and we need to fix them in stable branches too?
[...]
Unless we change those jobs to pin to an old version of tox for stable branches, yes.
On Thu, Dec 8, 2022 at 1:12 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2022-12-08 12:44:17 -0800 (-0800), Ghanshyam Mann wrote: [...]
It seems master jobs using tox<4[1] but all stable jobs using tox>=4.0[2] and failing. And both using ensure-tox
[1]
https://zuul.opendev.org/t/openstack/build/8a4585f2961a4854ad96c7d2a188b557/...
[2]
https://zuul.opendev.org/t/openstack/build/78cddc1180ff40109bbe17df884d23d8/... [...]
Those are Tempest jobs, which do some of their own installing of tox directly rather than just relying on the version provided by ensure-tox. Check with the QA team, since they've been working on fixes for those.
Does it mean all the stable jobs also will start using the latest tox (>4.0) which is breaking the jobs and we need to fix them in stable branches too?
[...]
Unless we change those jobs to pin to an old version of tox for stable branches, yes.
I'd be extremely in favor of this change. We should do everything we can to reduce the amount of effort needed for individual projects to keep stable branches passing tests.
-Jay Faulkner
---- On Thu, 08 Dec 2022 13:14:56 -0800 Jay Faulkner wrote ---
On Thu, Dec 8, 2022 at 1:12 PM Jeremy Stanley fungi@yuggoth.org> wrote: On 2022-12-08 12:44:17 -0800 (-0800), Ghanshyam Mann wrote: [...]
It seems master jobs using tox<4[1] but all stable jobs using tox>=4.0[2] and failing. And both using ensure-tox
[1] https://zuul.opendev.org/t/openstack/build/8a4585f2961a4854ad96c7d2a188b557/... [2] https://zuul.opendev.org/t/openstack/build/78cddc1180ff40109bbe17df884d23d8/...
[...]
Those are Tempest jobs, which do some of their own installing of tox directly rather than just relying on the version provided by ensure-tox. Check with the QA team, since they've been working on fixes for those.
Does it mean all the stable jobs also will start using the latest tox (>4.0) which is breaking the jobs and we need to fix them in stable branches too?
[...]
Unless we change those jobs to pin to an old version of tox for stable branches, yes.
I'd be extremely in favor of this change. We should do everything we can to reduce the amount of effort needed for individual projects to keep stable branches passing tests.
Yeah, I have pinned it on stable branches testing in devstack. It will take care of most of the devstack-based jobs unless there is an explicit installation of tox in other jobs but that can be pinned in a similar way.
https://review.opendev.org/q/I9a138af94dedc0d8ce5a0d519d75779415d3c30b+-bran...
-Jay Faulkner
---- On Thu, 08 Dec 2022 13:01:12 -0800 Jeremy Stanley wrote ---
On 2022-12-08 12:44:17 -0800 (-0800), Ghanshyam Mann wrote: [...]
It seems master jobs using tox<4[1] but all stable jobs using tox>=4.0[2] and failing. And both using ensure-tox
[1] https://zuul.opendev.org/t/openstack/build/8a4585f2961a4854ad96c7d2a188b557/... [2] https://zuul.opendev.org/t/openstack/build/78cddc1180ff40109bbe17df884d23d8/...
[...]
Those are Tempest jobs, which do some of their own installing of tox directly rather than just relying on the version provided by ensure-tox. Check with the QA team, since they've been working on fixes for those.
I was debugging that bug only.
I do not think it is Tempest's own installation. It is a devstack installation of tox (which I think uses ensure-tox on stable also unless we have any hidden installation overriding it) which is same on master as well as on stable but the master installation uses old tox which I am hoping due to ensure-tox capping tox but on stable it is not. I am confused by this different behaviour. I do not think devstack does any separate things for master and stable/zed for tox installation. Something from ensure-tox?
Does it mean all the stable jobs also will start using the latest tox (>4.0) which is breaking the jobs and we need to fix them in stable branches too?
[...]
Unless we change those jobs to pin to an old version of tox for stable branches, yes.
Ok, that means ensure-tox capping and uncapping are not useful for stable branch jobs. We should cap it in constraints also? or any specific installation of tox. Because we should not force stable branch jobs to use latest tox and break them and we fix them all.
=gmann
-- Jeremy Stanley
---- On Thu, 08 Dec 2022 13:16:04 -0800 Ghanshyam Mann wrote ---
---- On Thu, 08 Dec 2022 13:01:12 -0800 Jeremy Stanley wrote ---
On 2022-12-08 12:44:17 -0800 (-0800), Ghanshyam Mann wrote: [...]
It seems master jobs using tox<4[1] but all stable jobs using tox>=4.0[2] and failing. And both using ensure-tox
[1] https://zuul.opendev.org/t/openstack/build/8a4585f2961a4854ad96c7d2a188b557/... [2] https://zuul.opendev.org/t/openstack/build/78cddc1180ff40109bbe17df884d23d8/...
[...]
Those are Tempest jobs, which do some of their own installing of tox directly rather than just relying on the version provided by ensure-tox. Check with the QA team, since they've been working on fixes for those.
I was debugging that bug only.
I do not think it is Tempest's own installation. It is a devstack installation of tox (which I think uses ensure-tox on stable also unless we have any hidden installation overriding it) which is same on master as well as on stable but the master installation uses old tox which I am hoping due to ensure-tox capping tox but on stable it is not. I am confused by this different behaviour. I do not think devstack does any separate things for master and stable/zed for tox installation. Something from ensure-tox?
Another case of a mixup in tox installation is Devstack master Focal job which is using latest tox - https://zuul.opendev.org/t/openstack/build/aaad7312f68246c0979e1a43cfe96c3b/...
-gmann
Does it mean all the stable jobs also will start using the latest tox (>4.0) which is breaking the jobs and we need to fix them in stable branches too?
[...]
Unless we change those jobs to pin to an old version of tox for stable branches, yes.
Ok, that means ensure-tox capping and uncapping are not useful for stable branch jobs. We should cap it in constraints also? or any specific installation of tox. Because we should not force stable branch jobs to use latest tox and break them and we fix them all.
=gmann
-- Jeremy Stanley
On 2022-12-08 13:16:04 -0800 (-0800), Ghanshyam Mann wrote: [...]
I do not think it is Tempest's own installation. It is a devstack installation of tox (which I think uses ensure-tox on stable also unless we have any hidden installation overriding it) which is same on master as well as on stable but the master installation uses old tox which I am hoping due to ensure-tox capping tox but on stable it is not. I am confused by this different behaviour. I do not think devstack does any separate things for master and stable/zed for tox installation. Something from ensure-tox?
[...]
Not sure where you draw the line between DevStack and Tempest, for me they're part of the same suite split into different Git repositories. The problem area which was brought to my attention earlier today in #openstack-qa is in the configure_tempest function of DevStack's lib/tempest script which reinstalls tox rather than using the version already present on the server.
---- On Thu, 08 Dec 2022 13:38:41 -0800 Jeremy Stanley wrote ---
On 2022-12-08 13:16:04 -0800 (-0800), Ghanshyam Mann wrote: [...]
I do not think it is Tempest's own installation. It is a devstack installation of tox (which I think uses ensure-tox on stable also unless we have any hidden installation overriding it) which is same on master as well as on stable but the master installation uses old tox which I am hoping due to ensure-tox capping tox but on stable it is not. I am confused by this different behaviour. I do not think devstack does any separate things for master and stable/zed for tox installation. Something from ensure-tox?
[...]
Not sure where you draw the line between DevStack and Tempest, for me they're part of the same suite split into different Git repositories. The problem area which was brought to my attention earlier today in #openstack-qa is in the configure_tempest function of DevStack's lib/tempest script which reinstalls tox rather than using the version already present on the server.
I am debugging that bug but here I am asking something else here. My question is about different tox versions installed in the same type of jobs with devstack same scripts. Tox 4 is installed in the stable branch and master focal job and Tox 3 in the rest of the master jobs.
I am trying to understand why ensure-tox capping tox seems capping it for master jobs but not for stable branch jobs (no change int the Devstack scripts in master vs stable?
The only difference I see here is focal jobs (stable branch jobs or focal job on master) using latest tox.
-gmann
-- Jeremy Stanley
On 2022-12-08 13:56:45 -0800 (-0800), Ghanshyam Mann wrote: [...]
I am debugging that bug but here I am asking something else here. My question is about different tox versions installed in the same type of jobs with devstack same scripts. Tox 4 is installed in the stable branch and master focal job and Tox 3 in the rest of the master jobs.
I am trying to understand why ensure-tox capping tox seems capping it for master jobs but not for stable branch jobs (no change int the Devstack scripts in master vs stable?
The jobs you're talking about don't seem to use the ensure-tox role at all, so maybe that's where your confusion is arising? It's mostly used by jobs based (usually indirectly) on the "tox" parent job from zuul-jobs, so mostly unit tests and linters.
The only difference I see here is focal jobs (stable branch jobs or focal job on master) using latest tox.
I think you're either misunderstanding how those scripts work, or how Python packaging and pip work. If constraints lists are being supplied when the lib/tempest script installs its own copy of tox, then those can end up influencing which version of tox is installed even if tox is not in the constraints list (because current pip will try to satisfy constrained dependencies and limit available tox versions to those which work with any of its dependencies which have been constrained). If stable branches are forcing different versions of pip, that can also influence which versions of tox might end up installed, since older pip lacks a sane dependency solver and is just sort of "yolo" about the whole idea. There are lots of variables which can come into play.
Hi -
Also bit by sudden changes in tox 4, and I think it's important that the tox maintainers know that they did not adequately warn for these changes. our jobs should not be "breaking" without us having adequate warnings from previous tox versions.
My issue at https://github.com/tox-dev/tox/issues/2676 has been locked and the maintainer does not seem to think tox did anything incorrectly here. Pretty concerning for such a foundational project, so I would encourage new issues be raised at https://github.com/tox-dev/tox/issues/ https://github.com/tox-dev/tox/issues/2676 regarding individual backwards-incompatible changes that were made without warnings.
On Wed, Dec 7, 2022, at 2:19 PM, Jeremy Stanley wrote:
Tox 4.0.0, a major rewrite, just appeared on PyPI in the past couple of hours, so jobs are breaking right and left. We're going to try pinning to latest 3.x in zuul-jobs for the near term while we work through some of the more obvious issues with it, but until then lots of tox-based Zuul jobs are going to be broken. I'll follow up to this thread once we think things should be stabilized and we can start trying to pick through what's going to need fixing for the new tox version. -- Jeremy Stanley
*Attachments:*
- signature.asc
participants (4)
-
Ghanshyam Mann
-
Jay Faulkner
-
Jeremy Stanley
-
Mike Bayer