Open Stack

On 2020-04-18 18:11:58 +0200 (+0200), Andreas Jaeger wrote:
> More fun:
> 
> change https://review.opendev.org/720877 removes:
> 
> install_command = pip install {opts} {packages}
> 
> And suddenly lower-constraints fails - and the failure looks correct, so
> removing it is fine but will cause a surprise ;(

After analyzing this over IRC, it looks like
openstack/networking-hyperv probably has an incoherent
lower-constraints.txt file because it asks for (one of many
examples) neutron-lib==1.28.0 and oslo.concurrency==3.25.0 but
neutron-lib 1.28.0 declares it requires oslo.concurrency>=3.26.0 so
this won't work.

The lower constraints jobs are built on the premise that they're
working with a satisfiable set of constraints. Any time you
alter a lower bound for any dependency or add a new dependency,
you'll need to *fully* rebuild your lower-constraints.txt because of
the possible knock-on effects such a change can have on the
transitive dependencies tracked there.

https://review.opendev.org/675572 replaced neutron-lib>=1.18.0 with
neutron-lib>=1.28.0 in the requirements.txt, but only edited the
corresponding entry in lower-constraints.txt instead of taking all
of neutron-lib's dependencies (and their dependencies, and so on)
into account. The result is a lower constraints job which isn't
testing what they think it's testing.

The good news is that's all fixable. The bad news is that this
problem is probably widespread, because the places I expected to
talk about how to properly maintain your lower-constraints.txt file
provide minimal and potentially misleading advice:

https://docs.openstack.org/project-team-guide/dependency-management.html#adding-a-new-dependency

https://docs.openstack.org/project-team-guide/dependency-management.html#updating-the-minimum-version-of-a-dependency

Neither of those suggest rebuilding your lower-constraints.txt when
you alter a lower bound or add a new requirement. Further, I thought
I remembered there being a utility in the openstack/requirements
repository for correctly generating a lower-constraints.txt file,
but this does not actually exist that I can find.

The more I think back, the more I'm starting to remember that I
suggested the only thorough way (short of reimplementing pip's
version selection logic ourselves) to generate a lower constraints
file from the lower bounds in a requirements set would be by using
an altered copy of pip which inverted its version comparisons so
that it selected the lowest available versions of packages which
would satisfy every version range rather than the highest. I don't
think anybody has made that, and so I expect instead we've
collectively hand-waved with an assumption that people would be able
to figure out a working set of lower constraints instead (which is
in my opinion rather unrealistic given the complexity of the
transitive dependency trees of even some of our medium-sized
projects).

The upshot of this is that correctly calculating a lower constraint
set for any project is likely to involve a lot of trial and error,
probably far more than most projects are going to want to bear. If
we had a tool we could run to generate a coherent
lower-constraints.txt for a project, then this might be different,
but I suspect that too is going to be more hassle than anyone wants
to invest their time in creating.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200418/ed9be81d/attachment.sig>