On Thu, 2021-01-21 at 14:50 +0000, Jeremy Stanley wrote:
On 2021-01-21 09:30:19 +0000 (+0000), Stephen Finucane wrote: [...]
What we have doesn't work, and direct dependencies are the only things we can truly control. In the scenario you're suggesting, not only do we need to track dependencies, but we also need to track the dependencies of dependencies, and the dependencies of the dependencies of the dependencies, and the dependencies of the dependencies of the dependencies of the dependencies etc. etc. down the rabbit hole. For each of these indirect dependencies, of which there may be many, we need to figure out what the minimum version is for each of these indirect dependencies is manually, because as has been noted many times there is no standardized machinery in place in pip etc. to find (and test) the minimum dependency versions supported by a package. Put another way, if we depend on package foo, which depends on package bar, which depends on package baz, we can state our own informed minimum version for foo, but we will need to inspect foo to find a minimum version of bar that is suitable, and we will need to inspect baz to find a minimum version of baz that is suitable. An impossible ask. [...]
Where this begins to fall apart, as I mentioned earlier, is that the larger your transitive dependency set, the more likely it is that a direct dependency is *also* an indirect dependency (maybe many layers down). If a dependency of your dependency updates to a version which insists on a newer version of some other direct dependency of yours than what you've set in lower-constraints.txt, then your jobs are going to break and need lower bounds adjustments or additional indirect dependencies added to the lower-constraints.txt to roll them back to versions which worked with the others you've set. Unlike upper-constraints.txt where it's assumed that a complete transitive set of dependencies is covered, this will mean additional churn in your stable branches over time.
Ah, I didn't grasp this particular point when you initially raised it. I've spent some time playing around with pip (there's a fun code base) to no avail, so this (the reduced dependency set) is still the best idea I've got, flaws and all. With that said, our 'upper-constraints.txt' should still capture all dependencies, including those indirect ones, no? As such, none of those should be increasing on stable branches, which means the set of transitive dependencies should remain fixed once the branch is cut?
Or is the idea that we would only every do lower bounds checking on the release under development, and then remove those jobs when we branch?
This is also an option if it proves to be particularly painful. It does feel like this would indicate a failure of our upper-constraints to cap a dependency, even if its indirect, but I realize there are limits to what we can achieve here. Stephen