[requirements][qa] new pip resolver & our constraints needs
The PyPA team is working on a new resolver for pip. As part of that work, they have had some questions about the way OpenStack uses the constraints feature. They’ve been great about taking input, and prometheanfire has been doing some testing to ensure the new work doesn’t break compatibility [1] (thanks Matthew!). There is a new question from the pip maintainers about whether constraints need to support “nameless” entries (by referring to a URL in a constraints file instead of using package names) [2]. I don’t see anything in the upper-constraints.txt that looks like a URL, but I don’t know how teams might be configuring their lower-constraints.txt or whether we do anything in devstack to munge the constraints list to point to local packages as part of LIBS_FROM_GIT handling. Is anyone aware of any uses of URLs in constraints files anywhere within OpenStack? Doug [1] https://review.opendev.org/#/c/726186/ [2] https://github.com/pypa/pip/issues/8210
On 2020-05-10 12:39:46 -0400 (-0400), Doug Hellmann wrote: [...]
Is anyone aware of any uses of URLs in constraints files anywhere within OpenStack? [...]
Granted it's only indexing master branches, but this quick search shows no occurrences: http://codesearch.openstack.org/?q=https%3F%3A&i=nope&files=lower-constraints.txt -- Jeremy Stanley
On 20-05-10 17:14:45, Jeremy Stanley wrote:
On 2020-05-10 12:39:46 -0400 (-0400), Doug Hellmann wrote: [...]
Is anyone aware of any uses of URLs in constraints files anywhere within OpenStack? [...]
Granted it's only indexing master branches, but this quick search shows no occurrences:
http://codesearch.openstack.org/?q=https%3F%3A&i=nope&files=lower-constraints.txt
I think we disallow it in the requirements-check job. Yep, parse_urls is set to false by default. openstack_requirements/requirement.py def parse_line(req_line, permit_urls=False): """Parse a single line of a requirements file. requirements files here are a subset of pip requirements files: we don't try to parse URL entries, or pip options like -f and -e. Those are not permitted in global-requirements.txt. If encountered in a synchronised file such as requirements.txt or test-requirements.txt, they are illegal but currently preserved as-is. They may of course be used by local test configurations, just not committed into the OpenStack reference branches. :param permit_urls: If True, urls are parsed into Requirement tuples. By default they are not, because they cannot be reflected into setuptools kwargs, and thus the default is conservative. When urls are permitted, -e *may* be supplied at the start of the line. """ -- Matthew Thode
The PyPA team is working on a new resolver for pip. As part of that work, they have had some questions about the way OpenStack uses the constraints feature. They’ve been great about taking input, and prometheanfire has been doing some testing to ensure the new work doesn’t break compatibility [1] (thanks Matthew!).
There is a new question from the pip maintainers about whether constraints need to support “nameless” entries (by referring to a URL in a constraints file instead of using package names) [2]. I don’t see anything in the upper- constraints.txt that looks like a URL, but I don’t know how teams might be configuring their lower-constraints.txt or whether we do anything in devstack to munge the constraints list to point to local packages as part of LIBS_FROM_GIT handling.
Is anyone aware of any uses of URLs in constraints files anywhere within OpenStack? yes we sometimes have urls to git repos.
On Sun, 2020-05-10 at 12:39 -0400, Doug Hellmann wrote: they are not in constraits files but the are in test-requiremente or requirements.txt its not the best example but networking-ovs-dpdk has neutron for several reasons https://opendev.org/x/networking-ovs-dpdk/src/branch/master/test-requirement... -e git+https://github.com/openstack/neutron.git@master#egg=neutron in that url i have @master which can be a brach,tag or commit so you could use the same to tack an unreleased version fo an external depncy in a constraits file. im not sure if we require this for anythign i know in white box we had to also use the url syntax cor crundini and iniparse because the python 3 version were not released on pypi yet https://opendev.org/x/whitebox-tempest-plugin/commit/3ef1dded7d18eeec48e75c7... if it was not packaged on pypi ever and we supported lower constratins... then i can see us using git+https://github.com/pixelb/crudini.git@0.9.3#egg=crudini to be our lower constraint. right now we are installing master as they still have not pushed 0.9.3 to pypi https://github.com/pixelb/crudini/issues/58 this is the only why i think it would be useful to have support for the urls in constraits.
Doug
[1] https://review.opendev.org/#/c/726186/ [2] https://github.com/pypa/pip/issues/8210
On May 11, 2020, at 8:30 AM, Sean Mooney <smooney@redhat.com> wrote:
The PyPA team is working on a new resolver for pip. As part of that work, they have had some questions about the way OpenStack uses the constraints feature. They’ve been great about taking input, and prometheanfire has been doing some testing to ensure the new work doesn’t break compatibility [1] (thanks Matthew!).
There is a new question from the pip maintainers about whether constraints need to support “nameless” entries (by referring to a URL in a constraints file instead of using package names) [2]. I don’t see anything in the upper- constraints.txt that looks like a URL, but I don’t know how teams might be configuring their lower-constraints.txt or whether we do anything in devstack to munge the constraints list to point to local packages as part of LIBS_FROM_GIT handling.
Is anyone aware of any uses of URLs in constraints files anywhere within OpenStack? yes we sometimes have urls to git repos.
On Sun, 2020-05-10 at 12:39 -0400, Doug Hellmann wrote: they are not in constraits files but the are in test-requiremente or requirements.txt
OK, the question is specifically about whether unnamed dependencies listed in *constraints* are somehow expected to work. Having URLs in requirements lists isn’t an issue. jrosser pointed out on IRC that openstack-ansible uses URLs with egg= so I’ve suggested describing that upstream in case it’s relevant.
its not the best example but networking-ovs-dpdk has neutron for several reasons https://opendev.org/x/networking-ovs-dpdk/src/branch/master/test-requirement... <https://opendev.org/x/networking-ovs-dpdk/src/branch/master/test-requirements.txt#L14>
-e git+https://github.com/openstack/neutron.git@master#egg=neutron <git+https://github.com/openstack/neutron.git@master#egg=neutron>
in that url i have @master which can be a brach,tag or commit so you could use the same to tack an unreleased version fo an external depncy in a constraits file.
im not sure if we require this for anythign
i know in white box we had to also use the url syntax cor crundini and iniparse because the python 3 version were not released on pypi yet
https://opendev.org/x/whitebox-tempest-plugin/commit/3ef1dded7d18eeec48e75c7... <https://opendev.org/x/whitebox-tempest-plugin/commit/3ef1dded7d18eeec48e75c715df73da09237c965>
if it was not packaged on pypi ever and we supported lower constratins... then i can see us using git+https://github.com/pixelb/crudini.git@0.9.3#egg=crudini <git+https://github.com/pixelb/crudini.git@0.9.3#egg=crudini> to be our lower constraint.
right now we are installing master as they still have not pushed 0.9.3 to pypi https://github.com/pixelb/crudini/issues/58 <https://github.com/pixelb/crudini/issues/58>
this is the only why i think it would be useful to have support for the urls in constraits.
It sounds like this might be another case to describe on that pypa GitHub issue.
Doug
[1] https://review.opendev.org/#/c/726186/ <https://review.opendev.org/#/c/726186/> [2] https://github.com/pypa/pip/issues/8210 <https://github.com/pypa/pip/issues/8210>
participants (4)
-
Doug Hellmann
-
Jeremy Stanley
-
Matthew Thode
-
Sean Mooney