Open Stack

You may find the code for pip-compile
https://github.com/nvie/pip-tools/tree/future of interest for this, as I
think they may already have a solution for the deep dependency analysis.


I've started experimenting with it for git-upstream cause GitPython have
a habbit of breaking stuff through a couple of releases now :-(


What I like is:
* Doesn't require an extra tool before using 'pip install'
** Some may want to regen the dependencies, but it's optional and the
common python dev approach is retained
* Stable releases are guaranteed to use the versions of dependencies
they were released and verified against
* Improves on the guarantee of gated branch CI
** The idea that if you sync with upstream any test failures are due to
your local changes
** Which is not always true if updated deps can break stuff


On the flip side:
* You remain exposed to security issues in python code until you
manually update
* Development cycle doesn't move forward automatically, may not see
compatibility issues until late when forced to move forward one of the deps


Think the cons can be handled by some additional CI jobs to update the
pins on a regular basis and pass it through the standard gates and
potentially to auto approve during development cycles if they pass
(already getting the latest matching ones so no big diff here). Some
decisions on trade off around whether this should be done for stable
releases automatically or periodically requiring manual approval would
have to be made.


Did I say how much I like the fact that it doesn't require another tool
before just being able to use 'pip install'?


To experiment with it:
virtualenv .venv/pip-tools
source .venv/pip-tools/bin/activate
pip install git+https://github.com/nvie/pip-tools.git@future

Regards,
Darragh Bailey

"Nothing is foolproof to a sufficiently talented fool" - Unknown

On 22/01/15 03:45, Joshua Harlow wrote:
> A slightly better version that starts to go deeper (and downloads
> dependencies of dependencies and extracts there egg_info to get at
> these dependencies...)
>
> https://gist.github.com/harlowja/555ea019aef4e901897b
>
> Output @ http://paste.ubuntu.com/9813919/
>
> When ran on the same 'test.txt' mentioned below...
>
> Happy hacking!
>
> -Josh
>
> Joshua Harlow wrote:
>> A run that shows more of the happy/desired path:
>>
>> $ cat test.txt
>> six>1
>> taskflow<0.5
>> $ python pippin.py -r test.txt
>> Initial package set:
>> - six ['>1']
>> - taskflow ['<0.5']
>> Deep package set:
>> - six ['==1.9.0']
>> - taskflow ['==0.4.0']
>>
>> -Josh
>>
>> Joshua Harlow wrote:
>>> Another thing that I just started whipping together:
>>>
>>> https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
>>>
>>> The idea for the above is to use pip to download dependencies, but
>>> figure out what versions will work using our own resolver (and our own
>>> querying of 'http://pypi.python.org/pypi/%s/json') that just does a
>>> very
>>> deep search of all requirements (and requirements of requirements...).
>>>
>>> The idea for that is that the probe() function in that gist will
>>> 'freeze' a single requirement then dive down into further requirements
>>> and ensure compatibility while that 'diving' (aka, recursion into
>>> further requirements) is underway. If a incompatibility is found then
>>> the recursion will back-track and try a to freeze a different
>>> version of
>>> a desired package (and repeat...).
>>>
>>> To me this kind of deep finding would be a potential way of making this
>>> work in a way that basically only uses pip for downloading (and does
>>> the
>>> deep matching/probing) on our own since once the algorithm above
>>> doesn't
>>> backtrack and finds a matching set of requirements that will all work
>>> together the program can exit (and this set can then be used as the
>>> master set for openstack; at that point we might have to tell people to
>>> not use pip, or to only use pip --download to fetch the compatible
>>> versions).
>>>
>>> It's not completed but it could be complementary to what others are
>>> working on; feel free to hack away :)
>>>
>>> So far the following works:
>>>
>>> $ cat test.txt
>>> six>1
>>> taskflow>1
>>>
>>> $ python pippin.py -r test.txt
>>> Initial package set:
>>> - six ['>1']
>>> - taskflow ['>1']
>>> Traceback (most recent call last):
>>> File "pippin.py", line 168, in <module>
>>> main()
>>> File "pippin.py", line 162, in main
>>> matches = probe(initial, {})
>>> File "pippin.py", line 139, in probe
>>> result = probe(requirements, gathered)
>>> File "pippin.py", line 129, in probe
>>> m = find_match(pkg_name, req)
>>> File "pippin.py", line 112, in find_match
>>> return match_available(req.req, find_versions(pkg_name))
>>> File "pippin.py", line 108, in match_available
>>> " matches '%s' (tried %s)" % (req, looked_in))
>>> __main__.NotFoundException: No requirement found that matches
>>> 'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
>>> '0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
>>>
>>> I suspect all that is needed to add is the code that is marked with
>>> FIXME/TODO there and this kind of recursive back-tracking might just do
>>> the trick...
>>>
>>> -Josh
>>>
>>> Joe Gordon wrote:
>>>>
>>>>
>>>> On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon <joe.gordon0 at gmail.com
>>>> <mailto:joe.gordon0 at gmail.com>> wrote:
>>>>
>>>> We can side step the dependency graphing and ordering issue by
>>>> looking at the list of curently installed packages via pip freeze
>>>> and not installing dependencies (pip install --no-deps)
>>>>
>>>> After looking into this further here are the known issues:
>>>>
>>>> * Partial capping won't work [0], so we need to pin all
>>>> dependencies, we can generate this list per file via "pip install
>>>> -r" and "pip freeze", but this doesn't address the issue of apt-get
>>>> vs pip install. For example in the stable gate we use suds 0.4.1 but
>>>> only suds 0.4.0 is available via pip.
>>>> * Not all packages are installed in are standard dsvm-tempest env,
>>>> so using pip-freeze from that isn't enough
>>>> * We need to run this per requirements file and move to using pip
>>>> install --no-deps everywhere. As the global-requirements sync
>>>> wouldn't work the first time since files don't list all transient
>>>> dependencies yet.
>>>> * We can still break if a package version is removed from pypi
>>>> * in pip-freeze we sometimes install versions lower then our minimum
>>>> version (python-libvirt!)
>>>>
>>>>
>>>> Exploring a few ideas here: https://review.openstack.org/#/c/147451/4
>>>>
>>>>
>>>>
>>>> [0]
>>>> http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley <fungi at yuggoth.org
>>>> <mailto:fungi at yuggoth.org>> wrote:
>>>>
>>>> On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
>>>> [...]
>>>> > The other thing that happened was partial capping doesn't work,
>>>> > because something else moves forward and breaks you from below. So
>>>> > the patch will need to hit everything at once.
>>>>
>>>> Right, and we _have_ to start using stable branches on all
>>>> clients/libraries to backport fixes as part of that. This means that
>>>> the stable branch management workflow is about to become pervasive
>>>> across some teams who were previously (blissfully?) ignorant of it.
>>>>
>>>> > Unresolved entirely is the tertiary dependencies (not direct
>>>> > dependencies of any OpenStack project). That will need another
>>>> > mechanism to seed them before any installation happens.
>>>> [...]
>>>>
>>>> I won't go so far as to call it intractable, but I took a stab at it
>>>> about a year ago and building the dependency graph properly to be
>>>> able to do a depth-first ordering is nontrivial (enough that after
>>>> about a week hacking on possible solutions I gave up and switched to
>>>> more productive tasks). The primary complications I ran into were
>>>> identifying setup_requires in transitive dependencies and dealing
>>>> with platform/version-specific dependencies. That said, there's a
>>>> very good chance that more recent improvements in setuptools, pip
>>>> and virtualenv could make this task easier.
>>>>
>>>> > That's the things I can think off from the top of my head.
>>>>
>>>> The implementation, from a devstack-gate perspective, is also going
>>>> to require a decision on whether we stick with stable/relname for
>>>> branches of libraries too or switch to some extended branch mapping
>>>> mechanism to be able to track stable/relnum branches for those. And
>>>> we're going to need more jobs to ensure that clients (specifically)
>>>> retain backward-compatibility from an appdev and end user
>>>> perspective since they'll no longer get any testing as server
>>>> dependencies on stable branches (due to being capped there).
>>>> -- 
>>>> Jeremy Stanley
>>>>
>>>> __________________________________________________________________________
>>>>
>>>>
>>>>
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>> __________________________________________________________________________
>>>>
>>>>
>>>>
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev