[all] Lower-constraints in some projects broken - update your repos

Doug Hellmann doug at doughellmann.com
Sun Apr 19 17:09:30 UTC 2020

> On Apr 19, 2020, at 10:37 AM, Doug Hellmann <doug at doughellmann.com> wrote:
>> On Apr 18, 2020, at 1:29 PM, Jeremy Stanley <fungi at yuggoth.org> wrote:
>> On 2020-04-18 18:11:58 +0200 (+0200), Andreas Jaeger wrote:
>>> More fun:
>>> change https://review.opendev.org/720877 removes:
>>> install_command = pip install {opts} {packages}
>>> And suddenly lower-constraints fails - and the failure looks correct, so
>>> removing it is fine but will cause a surprise ;(
>> After analyzing this over IRC, it looks like
>> openstack/networking-hyperv probably has an incoherent
>> lower-constraints.txt file because it asks for (one of many
>> examples) neutron-lib==1.28.0 and oslo.concurrency==3.25.0 but
>> neutron-lib 1.28.0 declares it requires oslo.concurrency>=3.26.0 so
>> this won't work.
>> The lower constraints jobs are built on the premise that they're
>> working with a satisfiable set of constraints. Any time you
>> alter a lower bound for any dependency or add a new dependency,
>> you'll need to *fully* rebuild your lower-constraints.txt because of
>> the possible knock-on effects such a change can have on the
>> transitive dependencies tracked there.
>> https://review.opendev.org/675572 replaced neutron-lib>=1.18.0 with
>> neutron-lib>=1.28.0 in the requirements.txt, but only edited the
>> corresponding entry in lower-constraints.txt instead of taking all
>> of neutron-lib's dependencies (and their dependencies, and so on)
>> into account. The result is a lower constraints job which isn't
>> testing what they think it's testing.
>> The good news is that's all fixable. The bad news is that this
>> problem is probably widespread, because the places I expected to
>> talk about how to properly maintain your lower-constraints.txt file
>> provide minimal and potentially misleading advice:
>> https://docs.openstack.org/project-team-guide/dependency-management.html#adding-a-new-dependency
>> https://docs.openstack.org/project-team-guide/dependency-management.html#updating-the-minimum-version-of-a-dependency
>> Neither of those suggest rebuilding your lower-constraints.txt when
>> you alter a lower bound or add a new requirement. Further, I thought
>> I remembered there being a utility in the openstack/requirements
>> repository for correctly generating a lower-constraints.txt file,
>> but this does not actually exist that I can find.
> IIRC, the scripting I did way back when [1] was pretty naive. It looked at requirements.txt, converted >= to == and assumed either that any second order dependencies installed as a result of that set of constraints were the lowest likely to work or that teams would take care of editing the list.
> There did used to be an update-requirements tool in the requirements repo, but it was removed as part of that work and I’m not sure it ever dealt with lower bounds.
> [1] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html <http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html>
>> The more I think back, the more I'm starting to remember that I
>> suggested the only thorough way (short of reimplementing pip's
>> version selection logic ourselves) to generate a lower constraints
>> file from the lower bounds in a requirements set would be by using
>> an altered copy of pip which inverted its version comparisons so
>> that it selected the lowest available versions of packages which
>> would satisfy every version range rather than the highest. I don't
>> think anybody has made that, and so I expect instead we've
>> collectively hand-waved with an assumption that people would be able
>> to figure out a working set of lower constraints instead (which is
>> in my opinion rather unrealistic given the complexity of the
>> transitive dependency trees of even some of our medium-sized
>> projects).

https://github.com/pypa/pip/pull/8086 <https://github.com/pypa/pip/pull/8086> might be useful. It doesn’t force the use of lower bounds, but it should result in the minimum versions being selected where possible.

>> The upshot of this is that correctly calculating a lower constraint
>> set for any project is likely to involve a lot of trial and error,
>> probably far more than most projects are going to want to bear. If
>> we had a tool we could run to generate a coherent
>> lower-constraints.txt for a project, then this might be different,
>> but I suspect that too is going to be more hassle than anyone wants
>> to invest their time in creating.
> Yes, it seems that very few distributions actually want the *lowest* values, either. What they seem to want is something that matches what they are already packaging, which is likely to be less than the version in the upper constraints list.
>> -- 
>> Jeremy Stanley

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200419/0bc6df2f/attachment-0001.html>

More information about the openstack-discuss mailing list