[all] DevStack jobs broken due to setuptools not available for Python 2
Hi all, I noticed DevStack jobs fail all over the place [1] due to: UnsupportedPythonVersion: Package 'setuptools' requires a different Python: 2.7.17 not in '>=3.5' Bug reported in [2]. Notice USE_PYTHON3=True does not help as stack.sh is hardcoded to versionless Python. [1] https://zuul.opendev.org/t/openstack/builds?result=RETRY_LIMIT&result=FAILURE [2] https://bugs.launchpad.net/devstack/+bug/1859350 -yoctozepto
On Sun, Jan 12, 2020 at 07:20:27PM +0100, Radosław Piliszek wrote:
I noticed DevStack jobs fail all over the place [1] due to: UnsupportedPythonVersion: Package 'setuptools' requires a different Python: 2.7.17 not in '>=3.5'
I think there's a wide variety of things going on here. Firstly, I think pip should be not be trying to install this ... you clearly felt the same thing and have filed [1] where it seems that it might be due to the wheels we create not specifying "data-requires-python" in our links to the wheel. This is the first I've heard of this ... we will need to look into this wrt to our wheel building and I have filed [2]. The plain "virtualenv" call that sets up the requirements virtualenv should be using Python 3 I think; proposed in [3]. This would avoid the issue by using python3 on master. The other places calling "virtualenv" appear to be related to TRACK_DEPENDS, which I think we can remove now to avoid further confusion. Proposed in [4] However, this leaves devstack-gate which is used by grenade. I *think* that [5] will work if the older branch of devstack also installs with python3. The short answer is, yes, this is a big mess :/ -i [1] https://github.com/pypa/pip/issues/7586#issuecomment-573460206 [2] https://storyboard.openstack.org/#!/story/2007084 [3] https://review.opendev.org/702162 [4] https://review.opendev.org/702163 [5] https://review.opendev.org/702126
---- On Mon, 13 Jan 2020 00:05:54 -0600 Ian Wienand <iwienand@redhat.com> wrote ----
On Sun, Jan 12, 2020 at 07:20:27PM +0100, Radosław Piliszek wrote:
I noticed DevStack jobs fail all over the place [1] due to: UnsupportedPythonVersion: Package 'setuptools' requires a different Python: 2.7.17 not in '>=3.5'
I think there's a wide variety of things going on here.
Firstly, I think pip should be not be trying to install this ... you clearly felt the same thing and have filed [1] where it seems that it might be due to the wheels we create not specifying "data-requires-python" in our links to the wheel. This is the first I've heard of this ... we will need to look into this wrt to our wheel building and I have filed [2].
The plain "virtualenv" call that sets up the requirements virtualenv should be using Python 3 I think; proposed in [3]. This would avoid the issue by using python3 on master.
The other places calling "virtualenv" appear to be related to TRACK_DEPENDS, which I think we can remove now to avoid further confusion. Proposed in [4]
However, this leaves devstack-gate which is used by grenade. I *think* that [5] will work if the older branch of devstack also installs with python3.
Yes, grenade master jobs use py3 in both (new and older devstack) which is expected testing behaviour. We avoided (or at least not done yet) any mixed (py2->py3) upgrade testing. -gmann
The short answer is, yes, this is a big mess :/
-i
[1] https://github.com/pypa/pip/issues/7586#issuecomment-573460206 [2] https://storyboard.openstack.org/#!/story/2007084 [3] https://review.opendev.org/702162 [4] https://review.opendev.org/702163 [5] https://review.opendev.org/702126
It turned out to be real mess to fix devstack and devstack-gate and whatever else could be impacted. The current goal is to get rid of setuptools wheel so that it does not interfere in Zuul. As more packages go py3-only we will be forced to generate proper metadata in HTML indices... -yoctozepto pon., 13 sty 2020 o 15:08 Ghanshyam Mann <gmann@ghanshyammann.com> napisał(a):
---- On Mon, 13 Jan 2020 00:05:54 -0600 Ian Wienand <iwienand@redhat.com> wrote ----
On Sun, Jan 12, 2020 at 07:20:27PM +0100, Radosław Piliszek wrote:
I noticed DevStack jobs fail all over the place [1] due to: UnsupportedPythonVersion: Package 'setuptools' requires a different Python: 2.7.17 not in '>=3.5'
I think there's a wide variety of things going on here.
Firstly, I think pip should be not be trying to install this ... you clearly felt the same thing and have filed [1] where it seems that it might be due to the wheels we create not specifying "data-requires-python" in our links to the wheel. This is the first I've heard of this ... we will need to look into this wrt to our wheel building and I have filed [2].
The plain "virtualenv" call that sets up the requirements virtualenv should be using Python 3 I think; proposed in [3]. This would avoid the issue by using python3 on master.
The other places calling "virtualenv" appear to be related to TRACK_DEPENDS, which I think we can remove now to avoid further confusion. Proposed in [4]
However, this leaves devstack-gate which is used by grenade. I *think* that [5] will work if the older branch of devstack also installs with python3.
Yes, grenade master jobs use py3 in both (new and older devstack) which is expected testing behaviour. We avoided (or at least not done yet) any mixed (py2->py3) upgrade testing.
-gmann
The short answer is, yes, this is a big mess :/
-i
[1] https://github.com/pypa/pip/issues/7586#issuecomment-573460206 [2] https://storyboard.openstack.org/#!/story/2007084 [3] https://review.opendev.org/702162 [4] https://review.opendev.org/702163 [5] https://review.opendev.org/702126
On 2020-01-13 15:16:58 +0100 (+0100), Radosław Piliszek wrote:
It turned out to be real mess to fix devstack and devstack-gate and whatever else could be impacted. The current goal is to get rid of setuptools wheel so that it does not interfere in Zuul. As more packages go py3-only we will be forced to generate proper metadata in HTML indices... [...]
Yes, pardon me as I'm still catching up. Based on what I've read so far, the PyPI simple API provides some additional information with its package indices indicating which Python releases are supported by a given package. Our wheel cache is just being served from Apache with mod_autoindex providing basic indexing of the files, so does not have that information to provide. We've discussed a number of possible solutions to the problem. One option is to (temporarily) stop using the pre-built wheel cache, but that presupposes that it doesn't do much for us in the first place. That cache is there to provide pre-built wheels for packages which otherwise take a *very* long time to build from sdist and for which there is on usable wheel on PyPI (at least for the platforms on which we're running our jobs). Another option is to stop unnecessarily copying wheels already available on PyPI into that cache. It's used as a sieve, so that jobs can pull wheels from it when they exist but will still fall back to PyPI for any the cache doesn't contain. I suspect that the majority of projects dropping compatibility with older Python releases publish usable wheels for our platforms on PyPI already, so their presence in our cache is redundant. We could remove the latest Setuptools release from the wheel-mirror volume as a short-term solution, but will need to temporarily stop further updates to the cache since the job which builds it would just put that file right back. -- Jeremy Stanley
On Mon, Jan 13, 2020, at 6:43 AM, Jeremy Stanley wrote:
On 2020-01-13 15:16:58 +0100 (+0100), Radosław Piliszek wrote:
It turned out to be real mess to fix devstack and devstack-gate and whatever else could be impacted. The current goal is to get rid of setuptools wheel so that it does not interfere in Zuul. As more packages go py3-only we will be forced to generate proper metadata in HTML indices... [...]
Yes, pardon me as I'm still catching up. Based on what I've read so far, the PyPI simple API provides some additional information with its package indices indicating which Python releases are supported by a given package. Our wheel cache is just being served from Apache with mod_autoindex providing basic indexing of the files, so does not have that information to provide.
PyPI supplies this via the 'data-requires-python' element attributes in the html source of the index. You can see these viewing the source of eg https://pypi.org/simple/setuptools/. The problem with relying on this is not all packages supply this data (it is provided at package upload time iirc) and not all versions of pip support it. While we rely on mod_autoindex for the per package indexes (this is where data-requires-python goes) we do generate the top level index manually via https://opendev.org/openstack/project-config/src/branch/master/roles/copy-wh.... It is possible this script could be extended to write index files for each package as well. If we can determine what python versions are required we could then include that info. However, as mentioned above this won't fix all cases. I think the simplest option would be to simply stop building and mirroring wheels for packages which already have wheels on pypi. Let the wheel mirror be a true fallback for slow building packages like lxml and libvirt-python. Note that setuptools is a bit of an exception here because it is the bootstrap module. With setuptools in place we can control python specific package version selections using environment markers. This is already something we do a fair bit, https://opendev.org/openstack/requirements/src/branch/master/global-requirem..., which means we have the tooling in place to manage it. Clark
On 2020-01-13 14:43:56 +0000 (+0000), Jeremy Stanley wrote:
On 2020-01-13 15:16:58 +0100 (+0100), Radosław Piliszek wrote:
It turned out to be real mess to fix devstack and devstack-gate and whatever else could be impacted. The current goal is to get rid of setuptools wheel so that it does not interfere in Zuul. As more packages go py3-only we will be forced to generate proper metadata in HTML indices... [...]
Yes, pardon me as I'm still catching up. Based on what I've read so far, the PyPI simple API provides some additional information with its package indices indicating which Python releases are supported by a given package. Our wheel cache is just being served from Apache with mod_autoindex providing basic indexing of the files, so does not have that information to provide.
We've discussed a number of possible solutions to the problem. One option is to (temporarily) stop using the pre-built wheel cache, but that presupposes that it doesn't do much for us in the first place. That cache is there to provide pre-built wheels for packages which otherwise take a *very* long time to build from sdist and for which there is on usable wheel on PyPI (at least for the platforms on which we're running our jobs).
This was proposed via https://review.opendev.org/702166 for the record.
Another option is to stop unnecessarily copying wheels already available on PyPI into that cache. It's used as a sieve, so that jobs can pull wheels from it when they exist but will still fall back to PyPI for any the cache doesn't contain. I suspect that the majority of projects dropping compatibility with older Python releases publish usable wheels for our platforms on PyPI already, so their presence in our cache is redundant. We could remove the latest Setuptools release from the wheel-mirror volume as a short-term solution, but will need to temporarily stop further updates to the cache since the job which builds it would just put that file right back.
I've proposed https://review.opendev.org/702244 for a smaller-scale version of this now (explicitly blacklisting wheels for pip, setuptools and virtualenv) but we can expand it to something more thorough once it's put through its paces. If we merge that, then we can manually delete the affected setuptools wheel from our wheel-mirror volume and not have to worry about it coming back. -- Jeremy Stanley
On 2020-01-13 15:41:44 +0000 (+0000), Jeremy Stanley wrote: [...]
I've proposed https://review.opendev.org/702244 for a smaller-scale version of this now (explicitly blacklisting wheels for pip, setuptools and virtualenv) but we can expand it to something more thorough once it's put through its paces. If we merge that, then we can manually delete the affected setuptools wheel from our wheel-mirror volume and not have to worry about it coming back.
This has since merged, and as of 16:30 UTC (roughly an hour ago) I deleted all copies of the setuptools-45.0.0-py2.py3-none-any.whl file from our AFS volumes. We're testing now to see if previously broken jobs are working again, but suspect things should be back to normal. -- Jeremy Stanley
Hi,
On 13 Jan 2020, at 18:32, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-13 15:41:44 +0000 (+0000), Jeremy Stanley wrote: [...]
I've proposed https://review.opendev.org/702244 for a smaller-scale version of this now (explicitly blacklisting wheels for pip, setuptools and virtualenv) but we can expand it to something more thorough once it's put through its paces. If we merge that, then we can manually delete the affected setuptools wheel from our wheel-mirror volume and not have to worry about it coming back.
This has since merged, and as of 16:30 UTC (roughly an hour ago) I deleted all copies of the setuptools-45.0.0-py2.py3-none-any.whl file from our AFS volumes. We're testing now to see if previously broken jobs are working again, but suspect things should be back to normal.
Thx Jeremy for quick fix for this issue. It seems that, at least for Neutron, all jobs are working again :)
-- Jeremy Stanley
— Slawek Kaplonski Senior software engineer Red Hat
participants (6)
-
Clark Boylan
-
Ghanshyam Mann
-
Ian Wienand
-
Jeremy Stanley
-
Radosław Piliszek
-
Slawek Kaplonski