Open Stack

A slightly better version that starts to go deeper (and downloads 
dependencies of dependencies and extracts there egg_info to get at these 
dependencies...)

https://gist.github.com/harlowja/555ea019aef4e901897b

Output @ http://paste.ubuntu.com/9813919/

When ran on the same 'test.txt' mentioned below...

Happy hacking!

-Josh

Joshua Harlow wrote:
> A run that shows more of the happy/desired path:
>
> $ cat test.txt
> six>1
> taskflow<0.5
> $ python pippin.py -r test.txt
> Initial package set:
> - six ['>1']
> - taskflow ['<0.5']
> Deep package set:
> - six ['==1.9.0']
> - taskflow ['==0.4.0']
>
> -Josh
>
> Joshua Harlow wrote:
>> Another thing that I just started whipping together:
>>
>> https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
>>
>> The idea for the above is to use pip to download dependencies, but
>> figure out what versions will work using our own resolver (and our own
>> querying of 'http://pypi.python.org/pypi/%s/json') that just does a very
>> deep search of all requirements (and requirements of requirements...).
>>
>> The idea for that is that the probe() function in that gist will
>> 'freeze' a single requirement then dive down into further requirements
>> and ensure compatibility while that 'diving' (aka, recursion into
>> further requirements) is underway. If a incompatibility is found then
>> the recursion will back-track and try a to freeze a different version of
>> a desired package (and repeat...).
>>
>> To me this kind of deep finding would be a potential way of making this
>> work in a way that basically only uses pip for downloading (and does the
>> deep matching/probing) on our own since once the algorithm above doesn't
>> backtrack and finds a matching set of requirements that will all work
>> together the program can exit (and this set can then be used as the
>> master set for openstack; at that point we might have to tell people to
>> not use pip, or to only use pip --download to fetch the compatible
>> versions).
>>
>> It's not completed but it could be complementary to what others are
>> working on; feel free to hack away :)
>>
>> So far the following works:
>>
>> $ cat test.txt
>> six>1
>> taskflow>1
>>
>> $ python pippin.py -r test.txt
>> Initial package set:
>> - six ['>1']
>> - taskflow ['>1']
>> Traceback (most recent call last):
>> File "pippin.py", line 168, in <module>
>> main()
>> File "pippin.py", line 162, in main
>> matches = probe(initial, {})
>> File "pippin.py", line 139, in probe
>> result = probe(requirements, gathered)
>> File "pippin.py", line 129, in probe
>> m = find_match(pkg_name, req)
>> File "pippin.py", line 112, in find_match
>> return match_available(req.req, find_versions(pkg_name))
>> File "pippin.py", line 108, in match_available
>> " matches '%s' (tried %s)" % (req, looked_in))
>> __main__.NotFoundException: No requirement found that matches
>> 'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
>> '0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
>>
>> I suspect all that is needed to add is the code that is marked with
>> FIXME/TODO there and this kind of recursive back-tracking might just do
>> the trick...
>>
>> -Josh
>>
>> Joe Gordon wrote:
>>>
>>>
>>> On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon <joe.gordon0 at gmail.com
>>> <mailto:joe.gordon0 at gmail.com>> wrote:
>>>
>>> We can side step the dependency graphing and ordering issue by
>>> looking at the list of curently installed packages via pip freeze
>>> and not installing dependencies (pip install --no-deps)
>>>
>>> After looking into this further here are the known issues:
>>>
>>> * Partial capping won't work [0], so we need to pin all
>>> dependencies, we can generate this list per file via "pip install
>>> -r" and "pip freeze", but this doesn't address the issue of apt-get
>>> vs pip install. For example in the stable gate we use suds 0.4.1 but
>>> only suds 0.4.0 is available via pip.
>>> * Not all packages are installed in are standard dsvm-tempest env,
>>> so using pip-freeze from that isn't enough
>>> * We need to run this per requirements file and move to using pip
>>> install --no-deps everywhere. As the global-requirements sync
>>> wouldn't work the first time since files don't list all transient
>>> dependencies yet.
>>> * We can still break if a package version is removed from pypi
>>> * in pip-freeze we sometimes install versions lower then our minimum
>>> version (python-libvirt!)
>>>
>>>
>>> Exploring a few ideas here: https://review.openstack.org/#/c/147451/4
>>>
>>>
>>>
>>> [0]
>>> http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html
>>>
>>>
>>>
>>> On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley <fungi at yuggoth.org
>>> <mailto:fungi at yuggoth.org>> wrote:
>>>
>>> On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
>>> [...]
>>> > The other thing that happened was partial capping doesn't work,
>>> > because something else moves forward and breaks you from below. So
>>> > the patch will need to hit everything at once.
>>>
>>> Right, and we _have_ to start using stable branches on all
>>> clients/libraries to backport fixes as part of that. This means that
>>> the stable branch management workflow is about to become pervasive
>>> across some teams who were previously (blissfully?) ignorant of it.
>>>
>>> > Unresolved entirely is the tertiary dependencies (not direct
>>> > dependencies of any OpenStack project). That will need another
>>> > mechanism to seed them before any installation happens.
>>> [...]
>>>
>>> I won't go so far as to call it intractable, but I took a stab at it
>>> about a year ago and building the dependency graph properly to be
>>> able to do a depth-first ordering is nontrivial (enough that after
>>> about a week hacking on possible solutions I gave up and switched to
>>> more productive tasks). The primary complications I ran into were
>>> identifying setup_requires in transitive dependencies and dealing
>>> with platform/version-specific dependencies. That said, there's a
>>> very good chance that more recent improvements in setuptools, pip
>>> and virtualenv could make this task easier.
>>>
>>> > That's the things I can think off from the top of my head.
>>>
>>> The implementation, from a devstack-gate perspective, is also going
>>> to require a decision on whether we stick with stable/relname for
>>> branches of libraries too or switch to some extended branch mapping
>>> mechanism to be able to track stable/relnum branches for those. And
>>> we're going to need more jobs to ensure that clients (specifically)
>>> retain backward-compatibility from an appdev and end user
>>> perspective since they'll no longer get any testing as server
>>> dependencies on stable branches (due to being capped there).
>>> --
>>> Jeremy Stanley
>>>
>>> __________________________________________________________________________
>>>
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> __________________________________________________________________________
>>>
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev