Open Stack

On 15 April 2015 at 09:35, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>
>
> On Sun, Apr 12, 2015 at 6:09 PM, Robert Collins <robertc at robertcollins.net>
> wrote:
>>
>> On 13 April 2015 at 12:53, Monty Taylor <mordred at inaugust.com> wrote:
>>
>> > If we pin the stable branches with hard pins of direct and indirect
>> > dependencies, we can have our stable branch artifacts be installable.
>> > Thats awesome. IF there is a bugfix release or a security update to a
>> > dependent library - someone can propose it. Otherwise, the stable
>> > release should not be moving.
>>
>> Can we do that in stable branches? We've still got the problem of
>> bumping dependencies across multiple packages.
>
> What do you mean bumping dependencies across mulitple packages?

nova depends on oslo.messaging.
Both nova and oslo.messaging have $foo in install_requires. A hard pin
of $foo==X will then be applied by both nova and oslo.messaging.

AIUI we've moved to installing deps from PyPI (which is great), but
lets analyze both cases.

a) we install oslo.messaging from PyPI.
  - we go to change our version of foo to X+1 in nova
  - the gate job that overrides install_requires (by editing
requirements.txt from memory, which feeds into install_requires via
our reflection logic) will then fail when oslo.messaging is evaluated,
because X+1 != X and oslo.messaging requires ==X.

b) we install oslo.messaging from git
 - we go to change our version of foo to X+1 in nova
 - we edit oslo.messaging's install_requires as well as nova's
 - the commit succeeds, but oslo.messaging git and nova git are no
longer coinstallable, because oslo.messaging's foo==X has not been
changed in git

> We cannot do this today with 'pip install -r requirements.txt' but we can
> with 'pip install -r --no-deps requirements.txt'  if requirements includes
> all transitive dependencies. And then we have to figure out transitive
> dependencies for all projects etc.

pip install -r a-requirements-file-from-pip-freeze should always work,
because all transitive dependencies are included, and the versions
selected are mutually co-installable (assuming the environment that
was pip freeze's was safely constructed [there is one bug
https://github.com/pypa/pip/issues/2687 I believe that could be
triggered as we use multiple pip install runs to install stuff]).

The issues we have with our current requirements.txt and the different
ways it might be used are the source of us having had to try --no-deps
before: we have global requirements broad because it takes time to
propogate a change across repos, and we can't bump lower versions when
that breaks not-yet-ready repos (because we use global requirements to
override in the gate).

The issue inside pip that triggers this is:
 - when a requirement is added, we initially store the constraint for it
 - then we evaluate it against the indices in use, finding a version
that matches the constraints and selecting that.
 - if a later source of requirements (e.g. the install_requires from
the wheel or sdist of something we selected) has the same requirement,
and the selected version does not meet the constraints this new
requirement expressed, pip errors.

What needs to happen is that we need to be able to backtrack the
selection state back to where we selected the version that didn't
work, and re-evaluate without that one. This is of course
theoretically NPC because as we discover each version the conflict
might change each time as well... and the need to actually run sdist
egg_info to find out the requirements from each version adds even more
pain.

In principle though, a solid cache layer + a SAT tool should eat it up
for breakfast - but we need to rearrange a chunk of internal code to
be able to express it this way. OTOH I'm in the middle of doing that
because I need much of the same plumbing to be able to do
setup_requires in the way the setuptools and pip folk want. So... its
coming.

***but***

none of that matters, because the output from freeze - transitive
locked versions - will Just Work today.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud