Open Stack

On Mon, Jan 13, 2020, at 6:43 AM, Jeremy Stanley wrote:
> On 2020-01-13 15:16:58 +0100 (+0100), Radosław Piliszek wrote:
> > It turned out to be real mess to fix devstack and devstack-gate
> > and whatever else could be impacted. The current goal is to get
> > rid of setuptools wheel so that it does not interfere in Zuul. As
> > more packages go py3-only we will be forced to generate proper
> > metadata in HTML indices...
> [...]
> 
> Yes, pardon me as I'm still catching up. Based on what I've read so
> far, the PyPI simple API provides some additional information with
> its package indices indicating which Python releases are supported
> by a given package. Our wheel cache is just being served from Apache
> with mod_autoindex providing basic indexing of the files, so does
> not have that information to provide.
> 

PyPI supplies this via the 'data-requires-python' element attributes in the html source of the index. You can see these viewing the source of eg https://pypi.org/simple/setuptools/. The problem with relying on this is not all packages supply this data (it is provided at package upload time iirc) and not all versions of pip support it.

While we rely on mod_autoindex for the per package indexes (this is where data-requires-python goes) we do generate the top level index manually via https://opendev.org/openstack/project-config/src/branch/master/roles/copy-wheels/files/wheel-index.sh. It is possible this script could be extended to write index files for each package as well. If we can determine what python versions are required we could then include that info. However, as mentioned above this won't fix all cases.

I think the simplest option would be to simply stop building and mirroring wheels for packages which already have wheels on pypi. Let the wheel mirror be a true fallback for slow building packages like lxml and libvirt-python.

Note that setuptools is a bit of an exception here because it is the bootstrap module. With setuptools in place we can control python specific package version selections using environment markers. This is already something we do a fair bit, https://opendev.org/openstack/requirements/src/branch/master/global-requirements.txt#L47-L48, which means we have the tooling in place to manage it.

Clark