[stable][requirements][zuul] unpinned setuptools dependency on stable
Hi,
The latest setuptools (58.0) removed support for "use_2to3" [1] (deprecated since 46.2). Many OpenStack modules defines decorator==3.4.0 as lower constraints[2]. However, decorator 3.4.0 cannot be installed anymore with the latest setuptool as it depends on "use_2to3".
On master, this can be solved easily by bumping the dependency to decorator 4.0.0. On stable/xena we can still solve it the same way with a new RC. But on older stable branches such solution might be against the stable policy as it would require a major bump on our dependencies.
This issue is not limited to lower-constraints testing it just hit us there first. A similar break could happen in our upper constraints as well on old stable branches.
The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available.
During and after yesterday's nova meeting [3] we discussed possible solutions.
Option 1: Bump the major version of the decorator dependency on stable. Pros: * easy to implement Cons: * might against the policy / breaks downstream packagers * needs to be done in each affected project[3] separately * not future proof. After a future setuptools release we can see a similar break with another of our dependencies.
@Stable Cores: how do you feel about such bump?
Option 2: Pin the setuptools version during tox installation Pros: * single solution for all affected projects * future proof against breaks introduced by future setuptools releases Cons: * complex change as it probably requires to extend the base task in zuul itself
Option 3: turn off lower-constraints testing Pros: * easy to implement * some of us want to get rid of lower constraints anyhow Cons: * needs per project changes * not future proof against similar break affecting our upper constraints on old stable branches.
Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it
As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8].
@Stable Cores: what do you think what could be the way forward?
Cheers, gibi
[1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 [2] https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&lite... [3] https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.202... [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/... [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires [6] https://www.python.org/dev/peps/pep-0518/ [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini [8] https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/...
Hi,
Thanks for the detailed summary gibi!
Let me share my view:
Option 1: Since the time I am reviewing stable patches I remember that we usually avoided dependency's version bumps (except the case when the new pip dependency resolver started to work properly and revealed that lot of the lower constraints settings were broken) on stable branches. The written rule is "Following the Review guidelines. Specifically, not allowing backports of new features, *new* *dependencies*, or backward incompatible changes" [1].
Option 2: In OpenStack projects (who follows stable policy) we use global requirements, and upper constraints for stable branches. Why not use the same for the tools as well? It caused problem in the past as well (changes in virtualenv, setuptools, etc). In my opinion this would be better or future proof solution. Though it's true, we need to find the proper place and way to add the new constraints when installing tox in ensure-tox zuul role.
Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt!
Option 4: I'm not familiar with pyproject.toml, but I'm curious about the opinions of those who already used/uses it.
Long story short: I'd recommend option 2, as I think that is the best solution and fits the best to OpenStack stable way of working... If that is possible to add in our zuul job roles.
Nevertheless, I am curious about further opinions from the community. Thanks,
Előd
[1] https://docs.openstack.org/project-team-guide/stable-branches.html#active-ma...
On 2021. 09. 22. 14:11, Balazs Gibizer wrote:
Hi,
The latest setuptools (58.0) removed support for "use_2to3" [1] (deprecated since 46.2). Many OpenStack modules defines decorator==3.4.0 as lower constraints[2]. However, decorator 3.4.0 cannot be installed anymore with the latest setuptool as it depends on "use_2to3".
On master, this can be solved easily by bumping the dependency to decorator 4.0.0. On stable/xena we can still solve it the same way with a new RC. But on older stable branches such solution might be against the stable policy as it would require a major bump on our dependencies.
This issue is not limited to lower-constraints testing it just hit us there first. A similar break could happen in our upper constraints as well on old stable branches.
The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available.
During and after yesterday's nova meeting [3] we discussed possible solutions.
Option 1: Bump the major version of the decorator dependency on stable. Pros:
- easy to implement
Cons:
- might against the policy / breaks downstream packagers
- needs to be done in each affected project[3] separately
- not future proof. After a future setuptools release we can see a
similar break with another of our dependencies.
@Stable Cores: how do you feel about such bump?
Option 2: Pin the setuptools version during tox installation Pros:
- single solution for all affected projects
- future proof against breaks introduced by future setuptools releases
Cons:
- complex change as it probably requires to extend the base task in
zuul itself
Option 3: turn off lower-constraints testing Pros:
- easy to implement
- some of us want to get rid of lower constraints anyhow
Cons:
- needs per project changes
- not future proof against similar break affecting our upper
constraints on old stable branches.
Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it
As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8].
@Stable Cores: what do you think what could be the way forward?
Cheers, gibi
[1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 [2] https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&lite... [3] https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.202... [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/... [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires [6] https://www.python.org/dev/peps/pep-0518/ [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini [8] https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/...
On 2021-09-22 14:56:43 +0200 (+0200), Előd Illés wrote: [...]
Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt!
[...]
Do also note that all solutions in this space are effectively temporary. When we drafted the stable branch and extended maintenance policies, we accepted that there comes a time for any job where circumstances may prevent us from being able to effectively continue running it. If the complexity of the solution outweighs the *temporary* benefits of being able to run the job for longer, it may not be worth the effort.
While we almost certainly could come up with a non-default replacement for Zuul's standard tox job which allows us to use specific versions of tox, virtualenv, pip, setuptools, et cetera, that added complexity does not come for free. It has a maintenance cost which extends far into the future. Also think back on the times when we've reached for the "easy" solution of pinning such tools in other frameworks like DevStack: everyone's in a rush to stop the bleeding and get back to testing, but nobody cares about fixing the underlying issue and so such things end up pinned for years until that pinning breaks something else because we're blind to all the bitrot around us.
And then there's the fact that while we can "fix" things this way in CI jobs, these are ultimately tools for our developers which we just happen to also be able to run in the CI system, not the other way around. Pinning setuptools in the tox jobs does not solve this for developers running tox locally on their own systems. Are we going to start providing them with a list of specific versions of these tools which we're only able to test with, substantially increasing the complexity of setting up local dev environments and making it even harder for newcomers to contribute to the project?
The stable branch policy is a set of guidelines and suggestions, not an unbreakable covenant. It's there to help us make sound choices under normal circumstances, but we should not be afraid to come to a collective decision about making policy exceptions in unusual situations such as this one.
On Wed, Sep 22, 2021, at 6:15 AM, Jeremy Stanley wrote:
On 2021-09-22 14:56:43 +0200 (+0200), Előd Illés wrote: [...]
Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt!
[...]
Do also note that all solutions in this space are effectively temporary. When we drafted the stable branch and extended maintenance policies, we accepted that there comes a time for any job where circumstances may prevent us from being able to effectively continue running it. If the complexity of the solution outweighs the *temporary* benefits of being able to run the job for longer, it may not be worth the effort.
While we almost certainly could come up with a non-default replacement for Zuul's standard tox job which allows us to use specific versions of tox, virtualenv, pip, setuptools, et cetera, that added complexity does not come for free. It has a maintenance cost which extends far into the future. Also think back on the times when we've reached for the "easy" solution of pinning such tools in other frameworks like DevStack: everyone's in a rush to stop the bleeding and get back to testing, but nobody cares about fixing the underlying issue and so such things end up pinned for years until that pinning breaks something else because we're blind to all the bitrot around us.
And then there's the fact that while we can "fix" things this way in CI jobs, these are ultimately tools for our developers which we just happen to also be able to run in the CI system, not the other way around. Pinning setuptools in the tox jobs does not solve this for developers running tox locally on their own systems. Are we going to start providing them with a list of specific versions of these tools which we're only able to test with, substantially increasing the complexity of setting up local dev environments and making it even harder for newcomers to contribute to the project?
Related to this is the trap that we can often see passing CI as the goal. In reality the goal should be building quality reliable software that functions properly. The testing and CI gives us a measurement against that goal. We do pre merge testing so that we can avoid merging code that does not function. Occasionally external factors break functionality and in those cases the CI and testing have done their job. We are alerted to non functioning software that needs to be fixed.
In this case we need to be careful to avoid fixing this only in a CI context because we see passing CI as the goal. Instead we should try to fix the software so that it is functional for the CI system, developers on their local machines, and the individuals/orgs that ultimately deploy the software into production.
The stable branch policy is a set of guidelines and suggestions, not an unbreakable covenant. It's there to help us make sound choices under normal circumstances, but we should not be afraid to come to a collective decision about making policy exceptions in unusual situations such as this one. -- Jeremy Stanley
On Wed, Sep 22 2021 at 08:44:32 AM -0700, Clark Boylan cboylan@sapwetik.org wrote:
On Wed, Sep 22, 2021, at 6:15 AM, Jeremy Stanley wrote:
On 2021-09-22 14:56:43 +0200 (+0200), Előd Illés wrote: [...]
Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt!
[...]
Do also note that all solutions in this space are effectively temporary. When we drafted the stable branch and extended maintenance policies, we accepted that there comes a time for any job where circumstances may prevent us from being able to effectively continue running it. If the complexity of the solution outweighs the *temporary* benefits of being able to run the job for longer, it may not be worth the effort.
While we almost certainly could come up with a non-default replacement for Zuul's standard tox job which allows us to use specific versions of tox, virtualenv, pip, setuptools, et cetera, that added complexity does not come for free. It has a maintenance cost which extends far into the future. Also think back on the times when we've reached for the "easy" solution of pinning such tools in other frameworks like DevStack: everyone's in a rush to stop the bleeding and get back to testing, but nobody cares about fixing the underlying issue and so such things end up pinned for years until that pinning breaks something else because we're blind to all the bitrot around us.
And then there's the fact that while we can "fix" things this way in CI jobs, these are ultimately tools for our developers which we just happen to also be able to run in the CI system, not the other way around. Pinning setuptools in the tox jobs does not solve this for developers running tox locally on their own systems. Are we going to start providing them with a list of specific versions of these tools which we're only able to test with, substantially increasing the complexity of setting up local dev environments and making it even harder for newcomers to contribute to the project?
Related to this is the trap that we can often see passing CI as the goal. In reality the goal should be building quality reliable software that functions properly. The testing and CI gives us a measurement against that goal. We do pre merge testing so that we can avoid merging code that does not function. Occasionally external factors break functionality and in those cases the CI and testing have done their job. We are alerted to non functioning software that needs to be fixed.
In this case we need to be careful to avoid fixing this only in a CI context because we see passing CI as the goal. Instead we should try to fix the software so that it is functional for the CI system, developers on their local machines, and the individuals/orgs that ultimately deploy the software into production.
The [tox]requires based fix helps the CI and the developer locally as both are using tox. But it does not help individulas/orgs that are trying to deploy OpenStack with the lowest possible dependencies we say we support. To help them we have to signal either that our decorator lower constraint is now moved from 3.4.0 to 4.0.0 in all our stable branches OR that stable branches does not support setuptools 58.0.0 or newer. Which message we want to send?
Cheers, gibi
The stable branch policy is a set of guidelines and suggestions, not an unbreakable covenant. It's there to help us make sound choices under normal circumstances, but we should not be afraid to come to a collective decision about making policy exceptions in unusual situations such as this one. -- Jeremy Stanley
On Wed, Sep 22 2021 at 01:15:13 PM +0000, Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-09-22 14:56:43 +0200 (+0200), Előd Illés wrote: [...]
Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt!
[...]
Do also note that all solutions in this space are effectively temporary. When we drafted the stable branch and extended maintenance policies, we accepted that there comes a time for any job where circumstances may prevent us from being able to effectively continue running it. If the complexity of the solution outweighs the *temporary* benefits of being able to run the job for longer, it may not be worth the effort.
As the issue affects even the recent stable branches this would effectively mean we lose all the lower-constraints testing. And if later the same thing happen with one of our upper requirements then we will to lose all the testing. I think we cannot simply say that we ignore the problem by turning off the job.
While we almost certainly could come up with a non-default replacement for Zuul's standard tox job which allows us to use specific versions of tox, virtualenv, pip, setuptools, et cetera, that added complexity does not come for free. It has a maintenance cost which extends far into the future. Also think back on the times when we've reached for the "easy" solution of pinning such tools in other frameworks like DevStack: everyone's in a rush to stop the bleeding and get back to testing, but nobody cares about fixing the underlying issue and so such things end up pinned for years until that pinning breaks something else because we're blind to all the bitrot around us.
And then there's the fact that while we can "fix" things this way in CI jobs, these are ultimately tools for our developers which we just happen to also be able to run in the CI system, not the other way around. Pinning setuptools in the tox jobs does not solve this for developers running tox locally on their own systems. Are we going to start providing them with a list of specific versions of these tools which we're only able to test with, substantially increasing the complexity of setting up local dev environments and making it even harder for newcomers to contribute to the project?
The stable branch policy is a set of guidelines and suggestions, not an unbreakable covenant. It's there to help us make sound choices under normal circumstances, but we should not be afraid to come to a collective decision about making policy exceptions in unusual situations such as this one.
Does it mean you suggest bumping the decorator dependency to 4.0.0?
-- Jeremy Stanley
Le mer. 22 sept. 2021 à 18:05, Balazs Gibizer balazs.gibizer@est.tech a écrit :
On Wed, Sep 22 2021 at 01:15:13 PM +0000, Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-09-22 14:56:43 +0200 (+0200), Előd Illés wrote: [...]
Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt!
[...]
Do also note that all solutions in this space are effectively temporary. When we drafted the stable branch and extended maintenance policies, we accepted that there comes a time for any job where circumstances may prevent us from being able to effectively continue running it. If the complexity of the solution outweighs the *temporary* benefits of being able to run the job for longer, it may not be worth the effort.
As the issue affects even the recent stable branches this would effectively mean we lose all the lower-constraints testing. And if later the same thing happen with one of our upper requirements then we will to lose all the testing. I think we cannot simply say that we ignore the problem by turning off the job.
While we almost certainly could come up with a non-default replacement for Zuul's standard tox job which allows us to use specific versions of tox, virtualenv, pip, setuptools, et cetera, that added complexity does not come for free. It has a maintenance cost which extends far into the future. Also think back on the times when we've reached for the "easy" solution of pinning such tools in other frameworks like DevStack: everyone's in a rush to stop the bleeding and get back to testing, but nobody cares about fixing the underlying issue and so such things end up pinned for years until that pinning breaks something else because we're blind to all the bitrot around us.
And then there's the fact that while we can "fix" things this way in CI jobs, these are ultimately tools for our developers which we just happen to also be able to run in the CI system, not the other way around. Pinning setuptools in the tox jobs does not solve this for developers running tox locally on their own systems. Are we going to start providing them with a list of specific versions of these tools which we're only able to test with, substantially increasing the complexity of setting up local dev environments and making it even harder for newcomers to contribute to the project?
The stable branch policy is a set of guidelines and suggestions, not an unbreakable covenant. It's there to help us make sound choices under normal circumstances, but we should not be afraid to come to a collective decision about making policy exceptions in unusual situations such as this one.
Does it mean you suggest bumping the decorator dependency to 4.0.0?
-- Jeremy Stanley
I have to say fungi made great points. While option 2 has some ideal, option 1 seems the most pragmatic and won't divert us with developing efforts for getting there (AFAIK, we don't yet have a working solution for option 2).
Also, I don't know for all distros but the one I know doesn't use our lower constraints but rather redefine the dependencies by what they currently support. So, I'm even not sure bumping our lower dep here would create an issue for them if we tell them it's a minor release.
Operators, thoughts about it ?
---- On Wed, 22 Sep 2021 11:15:05 -0500 Sylvain Bauza sylvain.bauza@gmail.com wrote ----
Le mer. 22 sept. 2021 à 18:05, Balazs Gibizer balazs.gibizer@est.tech a écrit :
On Wed, Sep 22 2021 at 01:15:13 PM +0000, Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-09-22 14:56:43 +0200 (+0200), Előd Illés wrote: [...]
Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt!
[...]
Do also note that all solutions in this space are effectively temporary. When we drafted the stable branch and extended maintenance policies, we accepted that there comes a time for any job where circumstances may prevent us from being able to effectively continue running it. If the complexity of the solution outweighs the *temporary* benefits of being able to run the job for longer, it may not be worth the effort.
As the issue affects even the recent stable branches this would effectively mean we lose all the lower-constraints testing. And if later the same thing happen with one of our upper requirements then we will to lose all the testing. I think we cannot simply say that we ignore the problem by turning off the job.
While we almost certainly could come up with a non-default replacement for Zuul's standard tox job which allows us to use specific versions of tox, virtualenv, pip, setuptools, et cetera, that added complexity does not come for free. It has a maintenance cost which extends far into the future. Also think back on the times when we've reached for the "easy" solution of pinning such tools in other frameworks like DevStack: everyone's in a rush to stop the bleeding and get back to testing, but nobody cares about fixing the underlying issue and so such things end up pinned for years until that pinning breaks something else because we're blind to all the bitrot around us.
And then there's the fact that while we can "fix" things this way in CI jobs, these are ultimately tools for our developers which we just happen to also be able to run in the CI system, not the other way around. Pinning setuptools in the tox jobs does not solve this for developers running tox locally on their own systems. Are we going to start providing them with a list of specific versions of these tools which we're only able to test with, substantially increasing the complexity of setting up local dev environments and making it even harder for newcomers to contribute to the project?
The stable branch policy is a set of guidelines and suggestions, not an unbreakable covenant. It's there to help us make sound choices under normal circumstances, but we should not be afraid to come to a collective decision about making policy exceptions in unusual situations such as this one.
Does it mean you suggest bumping the decorator dependency to 4.0.0?
-- Jeremy Stanley
I have to say fungi made great points. While option 2 has some ideal, option 1 seems the most pragmatic and won't divert us with developing efforts for getting there (AFAIK, we don't yet have a working solution for option 2). Also, I don't know for all distros but the one I know doesn't use our lower constraints but rather redefine the dependencies by what they currently support. So, I'm even not sure bumping our lower dep here would create an issue for them if we tell them it's a minor release. Operators, thoughts about it ?
As per TC checks with distro/package maintainers on l-c usage, it is found that only Debian uses it and others do not care about it even on master branch[1].
I am also on the side of dropping the l-c from stable branches as soon as they are created instead of waiting for the broken gate and spend time on that where we know that they are hardly being used.
As Fungi mentioned, TC has updated the PTI also mentioning it clearly that testing the l-c is all up to project decisions and it is totally fine to drop them (many projects already dropped it from stable)
[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019877.h...
-gmann
On 2021-09-22 18:02:40 +0200 (+0200), Balazs Gibizer wrote: [...]
As the issue affects even the recent stable branches this would effectively mean we lose all the lower-constraints testing.
I've been strongly recommending dropping those jobs from all stable branches as soon as the branch is created. They're inherently broken by design, I did my best to explain that when the idea first arose, they don't really test what they set out to and the cost of keeping them functioning is disproportionately high. Many projects have already gotten rid of them, they're not mandated by the PTI anyway.
And if later the same thing happen with one of our upper requirements then we will to lose all the testing. I think we cannot simply say that we ignore the problem by turning off the job.
But that's not the current situation, right? I can come up with all sorts of scenarios where we might break upper bounds testing, that's merely one of them, we should evaluate our options for those on a case by case basis as they arise since the nature of the conflict will determine which solutions are viable and at what cost/benefit ratios.
Does it mean you suggest bumping the decorator dependency to 4.0.0?
In master, yes. On stable branches I recommend tossing the lower-constraints.txt based jobs out the nearest window.
On Wed, Sep 22, 2021, at 5:11 AM, Balazs Gibizer wrote:
Hi,
Snip
The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available.
During and after yesterday's nova meeting [3] we discussed possible solutions.
Option 1: Bump the major version of the decorator dependency on stable. Pros:
- easy to implement
Cons:
- might against the policy / breaks downstream packagers
- needs to be done in each affected project[3] separately
- not future proof. After a future setuptools release we can see a
similar break with another of our dependencies.
@Stable Cores: how do you feel about such bump?
Option 2: Pin the setuptools version during tox installation Pros:
- single solution for all affected projects
- future proof against breaks introduced by future setuptools releases
Cons:
- complex change as it probably requires to extend the base task in
zuul itself
Note as mentioned during the meeting yesterday I believe you actually need to pin virtualenv during the tox installation. If you want to pin setuptools it needs to happen when tox creates its virtualenvs.
One major con to this approach is now you've stopped testing that your software can deploy with newer versions of setuptools. It doesn't work right this moment, but as circumstances change you'll be without up to date information.
Option 3: turn off lower-constraints testing Pros:
- easy to implement
- some of us want to get rid of lower constraints anyhow
Cons:
- needs per project changes
- not future proof against similar break affecting our upper
constraints on old stable branches.
Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it
As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8].
This didn't work because the requires specification seems to only affect the meta venv that tox uses to bootstrap itself if necessary [9]. Basically this pinned setuptools in the wrong virtualenv. This might work if you pinned virtualenv here as suggested above.
@Stable Cores: what do you think what could be the way forward?
Cheers, gibi
[1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 [2] https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&lite... [3] https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.202... [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/... [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires [6] https://www.python.org/dev/peps/pep-0518/ [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini [8] https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/...
[9] https://tox.readthedocs.io/en/latest/example/basic.html#tox-auto-provisionin...
On Wed, Sep 22 2021 at 08:39:03 AM -0700, Clark Boylan cboylan@sapwetik.org wrote:
On Wed, Sep 22, 2021, at 5:11 AM, Balazs Gibizer wrote:
Hi,
Snip
The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available.
During and after yesterday's nova meeting [3] we discussed possible solutions.
Option 1: Bump the major version of the decorator dependency on stable. Pros:
- easy to implement
Cons:
- might against the policy / breaks downstream packagers
- needs to be done in each affected project[3] separately
- not future proof. After a future setuptools release we can see a
similar break with another of our dependencies.
@Stable Cores: how do you feel about such bump?
Option 2: Pin the setuptools version during tox installation Pros:
- single solution for all affected projects
- future proof against breaks introduced by future setuptools
releases Cons:
- complex change as it probably requires to extend the base task in
zuul itself
Note as mentioned during the meeting yesterday I believe you actually need to pin virtualenv during the tox installation. If you want to pin setuptools it needs to happen when tox creates its virtualenvs.
good point
One major con to this approach is now you've stopped testing that your software can deploy with newer versions of setuptools. It doesn't work right this moment, but as circumstances change you'll be without up to date information.
I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created.
Option 3: turn off lower-constraints testing Pros:
- easy to implement
- some of us want to get rid of lower constraints anyhow
Cons:
- needs per project changes
- not future proof against similar break affecting our upper
constraints on old stable branches.
Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it
As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8].
This didn't work because the requires specification seems to only affect the meta venv that tox uses to bootstrap itself if necessary [9]. Basically this pinned setuptools in the wrong virtualenv. This might work if you pinned virtualenv here as suggested above.
Thanks for the pointers. I think I managed to make this work locally. We will see if it works in CI too when [1] goes through the check pipeline.
Cheers, gibi
On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer balazs.gibizer@est.tech wrote:
I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created.
+1, this seems like a straightforward fix to me. https://review.opendev.org/c/openstack/requirements/+/810859 proposes what I think is the right change. Is there any downside?
On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote:
On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer balazs.gibizer@est.tech wrote:
I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created.
+1, this seems like a straightforward fix to me. https://review.opendev.org/c/openstack/requirements/+/810859 proposes what I think is the right change. Is there any downside?
Did that change actually fix the observed problem with tox jobs? Last I recall, the requirements/constraints on SetupTools were only effectively applied by DevStack because that includes a separate step for installing or downgrading it.
On Fri, Sep 24, 2021 at 3:38 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote:
On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer balazs.gibizer@est.tech wrote:
I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created.
+1, this seems like a straightforward fix to me. https://review.opendev.org/c/openstack/requirements/+/810859 proposes
what
I think is the right change. Is there any downside?
Did that change actually fix the observed problem with tox jobs? Last I recall, the requirements/constraints on SetupTools were only effectively applied by DevStack because that includes a separate step for installing or downgrading it.
I'm afraid I haven't yet tested it, or worked out how I would test it prior to merge - any hints for that?
My reasoning is based on reading the stack.sh stdout and underlying devstack code (for stable/ussuri):
+ tools/install_pip.sh:main:152 : pip_install_gr setuptools + inc/python:pip_install_gr:76 : local name=setuptools + inc/python:pip_install_gr:77 : local clean_name ++ inc/python:pip_install_gr:78 : get_from_global_requirements setuptools ++ inc/python:get_from_global_requirements:238 : local package=setuptools ++ inc/python:get_from_global_requirements:239 : local required_pkg +++ inc/python:get_from_global_requirements:240 : grep -i -h '^setuptools' /opt/stack/requirements/global-requirements.txt +++ inc/python:get_from_global_requirements:240 : cut -d# -f1 ++ inc/python:get_from_global_requirements:240 : required_pkg='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7''' ' ++ inc/python:get_from_global_requirements:241 : [[ setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='3.5'
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' == '' ]] ++ inc/python:get_from_global_requirements:244 : echo 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''' 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7'''' + inc/python:pip_install_gr:78 : clean_name='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5''' setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7'''' + inc/python:pip_install_gr:79 : pip_install 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''' 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7'''' Using python 3.6 to install setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' because python3_enabled=True
It clearly seems to be installing a version of setuptools as constrained by global-requirements.txt, and it appears that that installed version is the relevant factor, because runs with 58.* fail and older runs with 57.* succeeded.
On Fri, Sep 24, 2021, at 8:07 AM, Neil Jerram wrote:
On Fri, Sep 24, 2021 at 3:38 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote:
On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer balazs.gibizer@est.tech wrote:
I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created.
+1, this seems like a straightforward fix to me. https://review.opendev.org/c/openstack/requirements/+/810859 proposes what I think is the right change. Is there any downside?
Did that change actually fix the observed problem with tox jobs? Last I recall, the requirements/constraints on SetupTools were only effectively applied by DevStack because that includes a separate step for installing or downgrading it.
I'm afraid I haven't yet tested it, or worked out how I would test it prior to merge - any hints for that?
My reasoning is based on reading the stack.sh stdout and underlying devstack code (for stable/ussuri):
- tools/install_pip.sh:main:152 : pip_install_gr setuptools
- inc/python:pip_install_gr:76 : local name=setuptools
- inc/python:pip_install_gr:77 : local clean_name
++ inc/python:pip_install_gr:78 : get_from_global_requirements setuptools ++ inc/python:get_from_global_requirements:238 : local package=setuptools ++ inc/python:get_from_global_requirements:239 : local required_pkg +++ inc/python:get_from_global_requirements:240 : grep -i -h '^setuptools' /opt/stack/requirements/global-requirements.txt +++ inc/python:get_from_global_requirements:240 : cut -d# -f1 ++ inc/python:get_from_global_requirements:240 : required_pkg='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7''' ' ++ inc/python:get_from_global_requirements:241 : [[ setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='3.5'
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' == '' ]] ++ inc/python:get_from_global_requirements:244 : echo 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''' 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7''''
- inc/python:pip_install_gr:78 :
clean_name='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5''' setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7''''
- inc/python:pip_install_gr:79 : pip_install
'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''' 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7'''' Using python 3.6 to install setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' because python3_enabled=True
It clearly seems to be installing a version of setuptools as constrained by global-requirements.txt, and it appears that that installed version is the relevant factor, because runs with 58.* fail and older runs with 57.* succeeded.
Yup, as Fungi mentions this will work for devstack because devstack manages the setuptools installation. It won't work for tox because requirements and constraints are applied after tox+virtualenv have installed setuptools.
That said there may not be a one size fits all fix here and we'll have to address it several different ways. I think Fungi's concern is that we think a single fix has corrected the issue when that isn't the case.
On Fri, Sep 24, 2021 at 4:19 PM Clark Boylan cboylan@sapwetik.org wrote:
On Fri, Sep 24, 2021, at 8:07 AM, Neil Jerram wrote:
On Fri, Sep 24, 2021 at 3:38 PM Jeremy Stanley fungi@yuggoth.org
wrote:
On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote:
On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer
wrote:
I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created.
+1, this seems like a straightforward fix to me. https://review.opendev.org/c/openstack/requirements/+/810859
proposes what
I think is the right change. Is there any downside?
Did that change actually fix the observed problem with tox jobs? Last I recall, the requirements/constraints on SetupTools were only effectively applied by DevStack because that includes a separate step for installing or downgrading it.
I'm afraid I haven't yet tested it, or worked out how I would test it prior to merge - any hints for that?
My reasoning is based on reading the stack.sh stdout and underlying devstack code (for stable/ussuri):
- tools/install_pip.sh:main:152 : pip_install_gr setuptools
- inc/python:pip_install_gr:76 : local name=setuptools
- inc/python:pip_install_gr:77 : local clean_name
++ inc/python:pip_install_gr:78 : get_from_global_requirements setuptools ++ inc/python:get_from_global_requirements:238 : local package=setuptools ++ inc/python:get_from_global_requirements:239 : local required_pkg +++ inc/python:get_from_global_requirements:240 : grep -i -h '^setuptools' /opt/stack/requirements/global-requirements.txt +++ inc/python:get_from_global_requirements:240 : cut -d# -f1 ++ inc/python:get_from_global_requirements:240 :
required_pkg='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7'''
' ++ inc/python:get_from_global_requirements:241 : [[
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='3.5'
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7'
== '' ]] ++ inc/python:get_from_global_requirements:244 : echo
'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5''''
'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7''''
- inc/python:pip_install_gr:78 :
clean_name='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5'''
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7''''
- inc/python:pip_install_gr:79 : pip_install
'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='''3.5''''
'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='''2.7''''
Using python 3.6 to install
setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7'
because python3_enabled=True
It clearly seems to be installing a version of setuptools as constrained by global-requirements.txt, and it appears that that installed version is the relevant factor, because runs with 58.* fail and older runs with 57.* succeeded.
Yup, as Fungi mentions this will work for devstack because devstack manages the setuptools installation. It won't work for tox because requirements and constraints are applied after tox+virtualenv have installed setuptools.
That said there may not be a one size fits all fix here and we'll have to address it several different ways. I think Fungi's concern is that we think a single fix has corrected the issue when that isn't the case.
Ah right, I see more of the complexity now. Still, if a better overall solution can't be found quickly, I think it would be better to have some things working than nothing working.
I completely agree with gibi:
- on master (and thus, stable/xena) we need to support the latest setuptools - on older stable branches almost everything are already constrained (upper-constraints.txt), so I think that pinning setuptools on older branches is not that different than what we have now, plus it could eliminate a lot of possible similar issues in the future (I know, everything is 'temporary' as nothing will work forever, but if we want to make the stable maintenance easier then this is the better approach)
For me gibi's patch [1] looks quite promising for old stable branches. I know that it means that we have to create a similar patch for (at least) all broken old stable branches, but it unblocks the gate and seems quite trivial change.
Thanks,
Előd
[1] https://review.opendev.org/c/openstack/nova/+/810461
On 2021. 09. 22. 18:06, Balazs Gibizer wrote:
On Wed, Sep 22 2021 at 08:39:03 AM -0700, Clark Boylan cboylan@sapwetik.org wrote:
On Wed, Sep 22, 2021, at 5:11 AM, Balazs Gibizer wrote:
Hi,
Snip
The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available.
During and after yesterday's nova meeting [3] we discussed possible solutions.
Option 1: Bump the major version of the decorator dependency on stable. Pros: * easy to implement Cons: * might against the policy / breaks downstream packagers * needs to be done in each affected project[3] separately * not future proof. After a future setuptools release we can see a similar break with another of our dependencies.
@Stable Cores: how do you feel about such bump?
Option 2: Pin the setuptools version during tox installation Pros: * single solution for all affected projects * future proof against breaks introduced by future setuptools releases Cons: * complex change as it probably requires to extend the base task in zuul itself
Note as mentioned during the meeting yesterday I believe you actually need to pin virtualenv during the tox installation. If you want to pin setuptools it needs to happen when tox creates its virtualenvs.
good point
One major con to this approach is now you've stopped testing that your software can deploy with newer versions of setuptools. It doesn't work right this moment, but as circumstances change you'll be without up to date information.
I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created.
Option 3: turn off lower-constraints testing Pros: * easy to implement * some of us want to get rid of lower constraints anyhow Cons: * needs per project changes * not future proof against similar break affecting our upper constraints on old stable branches.
Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it
As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8].
This didn't work because the requires specification seems to only affect the meta venv that tox uses to bootstrap itself if necessary [9]. Basically this pinned setuptools in the wrong virtualenv. This might work if you pinned virtualenv here as suggested above.
Thanks for the pointers. I think I managed to make this work locally. We will see if it works in CI too when [1] goes through the check pipeline.
Cheers, gibi
On Wed, Sep 22, 2021 at 6:13 AM Balazs Gibizer balazs.gibizer@est.tech wrote:
Hi,
The latest setuptools (58.0) removed support for "use_2to3" [1] (deprecated since 46.2). Many OpenStack modules defines decorator==3.4.0 as lower constraints[2]. However, decorator 3.4.0 cannot be installed anymore with the latest setuptool as it depends on "use_2to3".
JFYI as I was looking into some other requirements issues yesterday, I hit this error with anyjson[0] 0.3.3 as well. It's used in a handful of projects[1] and there has not been a release since 2012[2] so this might be a problem in xena. I haven't checked the projects respective gates, but just want to highlight we'll probably have additional fallout from the setuptools change.
[0] https://github.com/pypa/setuptools/issues/1120#issuecomment-919239967 [1] https://codesearch.opendev.org/?q=anyjson&i=nope&literal=nope&fi... [2] https://pypi.org/project/anyjson/#history
On master, this can be solved easily by bumping the dependency to decorator 4.0.0. On stable/xena we can still solve it the same way with a new RC. But on older stable branches such solution might be against the stable policy as it would require a major bump on our dependencies.
This issue is not limited to lower-constraints testing it just hit us there first. A similar break could happen in our upper constraints as well on old stable branches.
The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available.
During and after yesterday's nova meeting [3] we discussed possible solutions.
Option 1: Bump the major version of the decorator dependency on stable. Pros:
- easy to implement
Cons:
- might against the policy / breaks downstream packagers
- needs to be done in each affected project[3] separately
- not future proof. After a future setuptools release we can see a
similar break with another of our dependencies.
@Stable Cores: how do you feel about such bump?
Option 2: Pin the setuptools version during tox installation Pros:
- single solution for all affected projects
- future proof against breaks introduced by future setuptools releases
Cons:
- complex change as it probably requires to extend the base task in
zuul itself
Option 3: turn off lower-constraints testing Pros:
- easy to implement
- some of us want to get rid of lower constraints anyhow
Cons:
- needs per project changes
- not future proof against similar break affecting our upper
constraints on old stable branches.
Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it
As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8].
@Stable Cores: what do you think what could be the way forward?
Cheers, gibi
[1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 [2]
https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&lite... [3]
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.202... [4]
https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/... [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires [6] https://www.python.org/dev/peps/pep-0518/ [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini [8]
https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/...
On 2021-09-24 07:19:03 -0600 (-0600), Alex Schultz wrote: [...]
JFYI as I was looking into some other requirements issues yesterday, I hit this error with anyjson[0] 0.3.3 as well. It's used in a handful of projects[1] and there has not been a release since 2012[2] so this might be a problem in xena. I haven't checked the projects respective gates, but just want to highlight we'll probably have additional fallout from the setuptools change.
[...]
Yes, we've also run into similar problems with pydot2 and funcparserlib, and I'm sure there's plenty more of what is effectively abandonware lingering in various projects' requirements lists. The long and short of it is that people with newer versions of SetupTools are going to be unable to install those, full stop. The maintainers of some of them may be spurred to action and release a new version, but in so doing may also drop support for older interpreters we still test with on some stable branches (this was the case with funcparserlib).
On the other hand, controlling what version of SetupTools others have and use isn't always possible, unlike runtime dependencies, so that really should be a solution of last resort. Making exceptions to stable branch policy in unusual circumstances such as this seems like a reasonable and more effective compromise.
On Fri, Sep 24, 2021 at 4:40 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-09-24 07:19:03 -0600 (-0600), Alex Schultz wrote: [...]
JFYI as I was looking into some other requirements issues yesterday, I hit this error with anyjson[0] 0.3.3 as well. It's used in a handful of projects[1] and there has not been a release since 2012[2] so this might be a problem in xena. I haven't checked the projects respective gates, but just want to highlight we'll probably have additional fallout from the setuptools change.
[...]
Yes, we've also run into similar problems with pydot2 and funcparserlib, and I'm sure there's plenty more of what is effectively abandonware lingering in various projects' requirements lists. The long and short of it is that people with newer versions of SetupTools are going to be unable to install those, full stop. The maintainers of some of them may be spurred to action and release a new version, but in so doing may also drop support for older interpreters we still test with on some stable branches (this was the case with funcparserlib).
Apparently, suds-jurko has the same problem, breaking oslo.vmware [1] and thus cinder.
Dmitry
[1] https://review.opendev.org/c/openstack/oslo.vmware/+/813377
On the other hand, controlling what version of SetupTools others have and use isn't always possible, unlike runtime dependencies, so that really should be a solution of last resort. Making exceptions to stable branch policy in unusual circumstances such as this seems like a reasonable and more effective compromise. -- Jeremy Stanley
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote:
Option 1: Bump the major version of the decorator dependency on stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0 which is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it).
Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking changes...).
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros rely on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
Option 4: utilize pyproject.toml[6] to specify build-time requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful, Cheers,
Thomas Goirand (zigo)
On Fri, Sep 24, 2021 at 9:22 PM Thomas Goirand zigo@debian.org wrote:
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote:
Option 1: Bump the major version of the decorator dependency on stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0 which is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it).
Irrelevant now because there are several more packages affected, not just Decorator.
Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking changes...).
Pretty sure no one has suggested pinning on master.
(And the subject of this email includes "stable", twice!)
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
+1, I've posted this: https://review.opendev.org/c/openstack/requirements/+/810859
Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros rely on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
How does this option help for the specific problem in hand? Namely:
- setuptools >=58.0.0 does not support use_2to3 option
- several packages that are pulled into OpenStack stable builds need use_2to3.
(Not saying it doesn't; I'm just ignorant here and would appreciate further explanation.)
Option 4: utilize pyproject.toml[6] to specify build-time requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful, Cheers,
Thomas Goirand (zigo)
On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand zigo@debian.org wrote:
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote:
Option 1: Bump the major version of the decorator dependency on stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0 which is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it).
Thanks for the info. So from Debian perspective it is OK to bump the decorator version on stable. As others noted in this thread it seems to be more than just decorator that broke. :/
Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking changes...).
I've no intention to pin it on master. Master needs to work with the latest and greatest. Also on master it is easier to fix / replace the dependencies that become broken with new setuptools.
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
ack
Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros rely on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
Option 4: utilize pyproject.toml[6] to specify build-time requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful,
Thanks!
Cheers, gibi
Cheers,
Thomas Goirand (zigo)
Hi again,
as I see there is no objection yet about using gibi's solution [1] (as I already summarized the situation in my previous mail [2]) for a fix for similar cases, so with a general stable core hat on, I *suggest* everyone to use that solution to pin the setuptools in tox for every failing cases (so that to avoid similar future errors as well).
[1] https://review.opendev.org/810461 [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059...
Előd
On 2021. 09. 27. 14:47, Balazs Gibizer wrote:
On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand zigo@debian.org wrote:
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote:
Option 1: Bump the major version of the decorator dependency on stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0 which is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it).
Thanks for the info. So from Debian perspective it is OK to bump the decorator version on stable. As others noted in this thread it seems to be more than just decorator that broke. :/
Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking changes...).
I've no intention to pin it on master. Master needs to work with the latest and greatest. Also on master it is easier to fix / replace the dependencies that become broken with new setuptools.
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
ack
Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros rely on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
Option 4: utilize pyproject.toml[6] to specify build-time requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful,
Thanks!
Cheers, gibi
Cheers,
Thomas Goirand (zigo)
But I don't think that solution works for devstack, does it? Is there a way to pin setuptools in a stable/ussuri devstack run, except by changing the stable branch of the requirements project?
On Mon, Sep 27, 2021 at 7:50 PM Előd Illés elod.illes@est.tech wrote:
Hi again,
as I see there is no objection yet about using gibi's solution [1] (as I already summarized the situation in my previous mail [2]) for a fix for similar cases, so with a general stable core hat on, I *suggest* everyone to use that solution to pin the setuptools in tox for every failing cases (so that to avoid similar future errors as well).
[1] https://review.opendev.org/810461 [2]
http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059...
Előd
On 2021. 09. 27. 14:47, Balazs Gibizer wrote:
On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand zigo@debian.org wrote:
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote:
Option 1: Bump the major version of the decorator dependency on stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0 which is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it).
Thanks for the info. So from Debian perspective it is OK to bump the decorator version on stable. As others noted in this thread it seems to be more than just decorator that broke. :/
Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking changes...).
I've no intention to pin it on master. Master needs to work with the latest and greatest. Also on master it is easier to fix / replace the dependencies that become broken with new setuptools.
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
ack
Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros rely on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
Option 4: utilize pyproject.toml[6] to specify build-time requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful,
Thanks!
Cheers, gibi
Cheers,
Thomas Goirand (zigo)
Is anyone helping to progress this? I just checked that stable/ussuri devstack is still broken.
Best wishes, Neil
On Tue, Sep 28, 2021 at 9:20 AM Neil Jerram neil@tigera.io wrote:
But I don't think that solution works for devstack, does it? Is there a way to pin setuptools in a stable/ussuri devstack run, except by changing the stable branch of the requirements project?
On Mon, Sep 27, 2021 at 7:50 PM Előd Illés elod.illes@est.tech wrote:
Hi again,
as I see there is no objection yet about using gibi's solution [1] (as I already summarized the situation in my previous mail [2]) for a fix for similar cases, so with a general stable core hat on, I *suggest* everyone to use that solution to pin the setuptools in tox for every failing cases (so that to avoid similar future errors as well).
[1] https://review.opendev.org/810461 [2]
http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059...
Előd
On 2021. 09. 27. 14:47, Balazs Gibizer wrote:
On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand zigo@debian.org wrote:
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote:
Option 1: Bump the major version of the decorator dependency on stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable),
for
which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0
which
is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it).
Thanks for the info. So from Debian perspective it is OK to bump the decorator version on stable. As others noted in this thread it seems to be more than just decorator that broke. :/
Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking changes...).
I've no intention to pin it on master. Master needs to work with the latest and greatest. Also on master it is easier to fix / replace the dependencies that become broken with new setuptools.
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
ack
Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros
rely
on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
Option 4: utilize pyproject.toml[6] to specify build-time
requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful,
Thanks!
Cheers, gibi
Cheers,
Thomas Goirand (zigo)
I can now confirm that https://review.opendev.org/c/openstack/requirements/+/810859 fixes my CI use case. (By temporarily using a fork of the requirements repo that includes that change.)
(Fix detail if needed here: https://github.com/projectcalico/networking-calico/pull/64/commits/cbed62824... )
Best wishes. Neil
On Mon, Oct 4, 2021 at 6:28 PM Neil Jerram neil@tigera.io wrote:
Is anyone helping to progress this? I just checked that stable/ussuri devstack is still broken.
Best wishes, Neil
On Tue, Sep 28, 2021 at 9:20 AM Neil Jerram neil@tigera.io wrote:
But I don't think that solution works for devstack, does it? Is there a way to pin setuptools in a stable/ussuri devstack run, except by changing the stable branch of the requirements project?
On Mon, Sep 27, 2021 at 7:50 PM Előd Illés elod.illes@est.tech wrote:
Hi again,
as I see there is no objection yet about using gibi's solution [1] (as I already summarized the situation in my previous mail [2]) for a fix for similar cases, so with a general stable core hat on, I *suggest* everyone to use that solution to pin the setuptools in tox for every failing cases (so that to avoid similar future errors as well).
[1] https://review.opendev.org/810461 [2]
http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059...
Előd
On 2021. 09. 27. 14:47, Balazs Gibizer wrote:
On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand zigo@debian.org wrote:
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote:
Option 1: Bump the major version of the decorator dependency on stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable),
for
which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0
which
is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it).
Thanks for the info. So from Debian perspective it is OK to bump the decorator version on stable. As others noted in this thread it seems to be more than just decorator that broke. :/
Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking
changes...).
I've no intention to pin it on master. Master needs to work with the latest and greatest. Also on master it is easier to fix / replace the dependencies that become broken with new setuptools.
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
ack
Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros
rely
on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
Option 4: utilize pyproject.toml[6] to specify build-time
requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful,
Thanks!
Cheers, gibi
Cheers,
Thomas Goirand (zigo)
Hi,
First, sorry for the slow response.
I think pinning setuptools in requirements for stable branches is also a good idea (up till wallaby). I can accept that.
Another thing is that the openstack projects that I've checked don't have issues in their CI regarding the unpinned setuptools. Mostly I saw/see the problem in unit test, static code check and similar tox targets.
Anyway, if this issue is there for devstack for others then I think we can cap setuptools, too, in requirements repository, if it is OK for everyone. My only concern is to cap it from the newest relevant stable branch where we need it. If I'm not mistaken most of the projects have fixed their related issue in Xena, so I guess Wallaby should be the first branch to cap setuptools.
Thanks,
Előd
On 2021. 10. 04. 20:16, Neil Jerram wrote:
I can now confirm that https://review.opendev.org/c/openstack/requirements/+/810859 https://review.opendev.org/c/openstack/requirements/+/810859 fixes my CI use case. (By temporarily using a fork of the requirements repo that includes that change.)
(Fix detail if needed here: https://github.com/projectcalico/networking-calico/pull/64/commits/cbed62824... https://github.com/projectcalico/networking-calico/pull/64/commits/cbed6282405957f7d60b6e0790c91fb852afe84c)
Best wishes. Neil
On Mon, Oct 4, 2021 at 6:28 PM Neil Jerram <neil@tigera.io mailto:neil@tigera.io> wrote:
Is anyone helping to progress this? I just checked that stable/ussuri devstack is still broken. Best wishes, Neil On Tue, Sep 28, 2021 at 9:20 AM Neil Jerram <neil@tigera.io <mailto:neil@tigera.io>> wrote: But I don't think that solution works for devstack, does it? Is there a way to pin setuptools in a stable/ussuri devstack run, except by changing the stable branch of the requirements project? On Mon, Sep 27, 2021 at 7:50 PM Előd Illés <elod.illes@est.tech> wrote: Hi again, as I see there is no objection yet about using gibi's solution [1] (as I already summarized the situation in my previous mail [2]) for a fix for similar cases, so with a general stable core hat on, I *suggest* everyone to use that solution to pin the setuptools in tox for every failing cases (so that to avoid similar future errors as well). [1] https://review.opendev.org/810461 <https://review.opendev.org/810461> [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059.html <http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059.html> Előd On 2021. 09. 27. 14:47, Balazs Gibizer wrote: > > > On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand > <zigo@debian.org <mailto:zigo@debian.org>> wrote: >> Hi Gibi! >> >> Thanks for bringing this up. >> >> As a distro package maintainer, here's my view. >> >> On 9/22/21 2:11 PM, Balazs Gibizer wrote: >>> Option 1: Bump the major version of the decorator dependency on >>> stable. >> >> Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for >> which I don't even maintain OpenStack anymore (that's OpenStack >> Newton...). So I don't see how switching to decorator 4.0.0 is a >> problem, and I don't understand how OpenStack could be using 3.4.0 which >> is in Jessie (ie: 6 years old Debian release). >> >> PyPi says Decorator 3.4.0 is from 2012: >> https://pypi.org/project/decorator/#history <https://pypi.org/project/decorator/#history> >> >> Do you have your release numbers correct? If so, then switching to >> Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) >> and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 >> feels a bit crazy (and I wasn't aware of it). > > Thanks for the info. So from Debian perspective it is OK to bump the > decorator version on stable. As others noted in this thread it seems > to be more than just decorator that broke. :/ > >> >>> Option 2: Pin the setuptools version during tox installation >> >> Please don't do this for the master branch, we need OpenStack to stay >> current with setuptools (yeah, even if this means breaking changes...). > > I've no intention to pin it on master. Master needs to work with the > latest and greatest. Also on master it is easier to fix / replace the > dependencies that become broken with new setuptools. > >> >> For already released OpenStack: I don't mind much if this is done (I >> could backport fixes if something breaks). > > ack > >> >>> Option 3: turn off lower-constraints testing >> >> I already expressed myself about this: this is dangerous as distros rely >> on it for setting lower bounds as low as possible (which is always >> preferred from a distro point of view). >> >>> Option 4: utilize pyproject.toml[6] to specify build-time requirements >> >> I don't know about pyproject.toml. >> >> Just my 2 cents, hoping it's useful, > > Thanks! > > Cheers, > gibi > >> Cheers, >> >> Thomas Goirand (zigo) >> > > >
On Thu, Oct 14, 2021 at 10:03 AM Előd Illés elod.illes@est.tech wrote:
Hi,
First, sorry for the slow response.
I think pinning setuptools in requirements for stable branches is also a good idea (up till wallaby). I can accept that.
Another thing is that the openstack projects that I've checked don't have issues in their CI regarding the unpinned setuptools. Mostly I saw/see the problem in unit test, static code check and similar tox targets.
Anyway, if this issue is there for devstack for others then I think we can cap setuptools, too, in requirements repository, if it is OK for everyone. My only concern is to cap it from the newest relevant stable branch where we need it. If I'm not mistaken most of the projects have fixed their related issue in Xena, so I guess Wallaby should be the first branch to cap setuptools.
I don't know if someone was working on it already but I proposed the patch on Ussuri: https://review.opendev.org/c/openstack/requirements/+/821878 If this gets accepted, I'll backport it to stable/train as well.
Without that patch, it's impossible to use Devstack outside of the gate to deploy Ussuri or Train.
Thanks,
Thanks,
Előd
On 2021. 10. 04. 20:16, Neil Jerram wrote:
I can now confirm that https://review.opendev.org/c/openstack/requirements/+/810859 fixes my CI use case. (By temporarily using a fork of the requirements repo that includes that change.)
(Fix detail if needed here: https://github.com/projectcalico/networking-calico/pull/64/commits/cbed62824... )
Best wishes. Neil
On Mon, Oct 4, 2021 at 6:28 PM Neil Jerram neil@tigera.io wrote:
Is anyone helping to progress this? I just checked that stable/ussuri devstack is still broken.
Best wishes, Neil
On Tue, Sep 28, 2021 at 9:20 AM Neil Jerram neil@tigera.io wrote:
But I don't think that solution works for devstack, does it? Is there a way to pin setuptools in a stable/ussuri devstack run, except by changing the stable branch of the requirements project?
On Mon, Sep 27, 2021 at 7:50 PM Előd Illés elod.illes@est.tech elod.illes@est.tech wrote:
Hi again,
as I see there is no objection yet about using gibi's solution [1] (as I already summarized the situation in my previous mail [2]) for a fix for similar cases, so with a general stable core hat on, I *suggest* everyone to use that solution to pin the setuptools in tox for every failing cases (so that to avoid similar future errors as well).
[1] https://review.opendev.org/810461 [2]
http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059...
Előd
On 2021. 09. 27. 14:47, Balazs Gibizer wrote:
On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand zigo@debian.org wrote:
Hi Gibi!
Thanks for bringing this up.
As a distro package maintainer, here's my view.
On 9/22/21 2:11 PM, Balazs Gibizer wrote: > Option 1: Bump the major version of the decorator dependency on > stable.
Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable),
for
which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0
which
is in Jessie (ie: 6 years old Debian release).
PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history
Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with
3.4.0
feels a bit crazy (and I wasn't aware of it).
Thanks for the info. So from Debian perspective it is OK to bump the decorator version on stable. As others noted in this thread it seems to be more than just decorator that broke. :/
> Option 2: Pin the setuptools version during tox installation
Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking
changes...).
I've no intention to pin it on master. Master needs to work with the latest and greatest. Also on master it is easier to fix / replace the dependencies that become broken with new setuptools.
For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks).
ack
> Option 3: turn off lower-constraints testing
I already expressed myself about this: this is dangerous as distros
rely
on it for setting lower bounds as low as possible (which is always preferred from a distro point of view).
> Option 4: utilize pyproject.toml[6] to specify build-time
requirements
I don't know about pyproject.toml.
Just my 2 cents, hoping it's useful,
Thanks!
Cheers, gibi
Cheers,
Thomas Goirand (zigo)
participants (11)
-
Alex Schultz
-
Balazs Gibizer
-
Clark Boylan
-
Dmitry Tantsur
-
Előd Illés
-
Emilien Macchi
-
Ghanshyam Mann
-
Jeremy Stanley
-
Neil Jerram
-
Sylvain Bauza
-
Thomas Goirand