Open Stack

> On Apr 18, 2020, at 1:29 PM, Jeremy Stanley <fungi at yuggoth.org> wrote:
> 
> On 2020-04-18 18:11:58 +0200 (+0200), Andreas Jaeger wrote:
>> More fun:
>> 
>> change https://review.opendev.org/720877 removes:
>> 
>> install_command = pip install {opts} {packages}
>> 
>> And suddenly lower-constraints fails - and the failure looks correct, so
>> removing it is fine but will cause a surprise ;(
> 
> After analyzing this over IRC, it looks like
> openstack/networking-hyperv probably has an incoherent
> lower-constraints.txt file because it asks for (one of many
> examples) neutron-lib==1.28.0 and oslo.concurrency==3.25.0 but
> neutron-lib 1.28.0 declares it requires oslo.concurrency>=3.26.0 so
> this won't work.
> 
> The lower constraints jobs are built on the premise that they're
> working with a satisfiable set of constraints. Any time you
> alter a lower bound for any dependency or add a new dependency,
> you'll need to *fully* rebuild your lower-constraints.txt because of
> the possible knock-on effects such a change can have on the
> transitive dependencies tracked there.
> 
> https://review.opendev.org/675572 replaced neutron-lib>=1.18.0 with
> neutron-lib>=1.28.0 in the requirements.txt, but only edited the
> corresponding entry in lower-constraints.txt instead of taking all
> of neutron-lib's dependencies (and their dependencies, and so on)
> into account. The result is a lower constraints job which isn't
> testing what they think it's testing.
> 
> The good news is that's all fixable. The bad news is that this
> problem is probably widespread, because the places I expected to
> talk about how to properly maintain your lower-constraints.txt file
> provide minimal and potentially misleading advice:
> 
> https://docs.openstack.org/project-team-guide/dependency-management.html#adding-a-new-dependency
> 
> https://docs.openstack.org/project-team-guide/dependency-management.html#updating-the-minimum-version-of-a-dependency
> 
> Neither of those suggest rebuilding your lower-constraints.txt when
> you alter a lower bound or add a new requirement. Further, I thought
> I remembered there being a utility in the openstack/requirements
> repository for correctly generating a lower-constraints.txt file,
> but this does not actually exist that I can find.

IIRC, the scripting I did way back when [1] was pretty naive. It looked at requirements.txt, converted >= to == and assumed either that any second order dependencies installed as a result of that set of constraints were the lowest likely to work or that teams would take care of editing the list.

There did used to be an update-requirements tool in the requirements repo, but it was removed as part of that work and I’m not sure it ever dealt with lower bounds.

[1] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html

> 
> The more I think back, the more I'm starting to remember that I
> suggested the only thorough way (short of reimplementing pip's
> version selection logic ourselves) to generate a lower constraints
> file from the lower bounds in a requirements set would be by using
> an altered copy of pip which inverted its version comparisons so
> that it selected the lowest available versions of packages which
> would satisfy every version range rather than the highest. I don't
> think anybody has made that, and so I expect instead we've
> collectively hand-waved with an assumption that people would be able
> to figure out a working set of lower constraints instead (which is
> in my opinion rather unrealistic given the complexity of the
> transitive dependency trees of even some of our medium-sized
> projects).
> 
> The upshot of this is that correctly calculating a lower constraint
> set for any project is likely to involve a lot of trial and error,
> probably far more than most projects are going to want to bear. If
> we had a tool we could run to generate a coherent
> lower-constraints.txt for a project, then this might be different,
> but I suspect that too is going to be more hassle than anyone wants
> to invest their time in creating.

Yes, it seems that very few distributions actually want the *lowest* values, either. What they seem to want is something that matches what they are already packaging, which is likely to be less than the version in the upper constraints list.


> -- 
> Jeremy Stanley