Open Stack

For those that have been following this escapade I wrote up the 
following, hopefully it's useful to someone ;)

https://github.com/harlowja/pippin#how-it-works

Example(s) of actual runs are also @

https://github.com/harlowja/pippin/tree/master/examples

^ Does not include the full output files (since it's a lot of 
information/data that wouldn't fit easily on git...)

-Josh

Joshua Harlow wrote:
> Cool,
>
> I've got to try that out today to see what it's doing.
>
> I've also shoved my little program up @
> https://github.com/harlowja/pippin (the pip-tools one is definitely more
> elegantly coded than mine, haha).
>
> Feel free to fork it (modify, run, or ...)
>
> Basic instructions to use it:
>
> https://github.com/harlowja/pippin#pippin
>
> -Josh
>
> Bailey, Darragh wrote:
>> You may find the code for pip-compile
>> https://github.com/nvie/pip-tools/tree/future of interest for this, as I
>> think they may already have a solution for the deep dependency analysis.
>>
>>
>> I've started experimenting with it for git-upstream cause GitPython have
>> a habbit of breaking stuff through a couple of releases now :-(
>>
>>
>> What I like is:
>> * Doesn't require an extra tool before using 'pip install'
>> ** Some may want to regen the dependencies, but it's optional and the
>> common python dev approach is retained
>> * Stable releases are guaranteed to use the versions of dependencies
>> they were released and verified against
>> * Improves on the guarantee of gated branch CI
>> ** The idea that if you sync with upstream any test failures are due to
>> your local changes
>> ** Which is not always true if updated deps can break stuff
>>
>>
>> On the flip side:
>> * You remain exposed to security issues in python code until you
>> manually update
>> * Development cycle doesn't move forward automatically, may not see
>> compatibility issues until late when forced to move forward one of the
>> deps
>>
>>
>> Think the cons can be handled by some additional CI jobs to update the
>> pins on a regular basis and pass it through the standard gates and
>> potentially to auto approve during development cycles if they pass
>> (already getting the latest matching ones so no big diff here). Some
>> decisions on trade off around whether this should be done for stable
>> releases automatically or periodically requiring manual approval would
>> have to be made.
>>
>>
>> Did I say how much I like the fact that it doesn't require another tool
>> before just being able to use 'pip install'?
>>
>>
>> To experiment with it:
>> virtualenv .venv/pip-tools
>> source .venv/pip-tools/bin/activate
>> pip install git+https://github.com/nvie/pip-tools.git@future
>>
>> Regards,
>> Darragh Bailey
>>
>> "Nothing is foolproof to a sufficiently talented fool" - Unknown
>>
>> On 22/01/15 03:45, Joshua Harlow wrote:
>>> A slightly better version that starts to go deeper (and downloads
>>> dependencies of dependencies and extracts there egg_info to get at
>>> these dependencies...)
>>>
>>> https://gist.github.com/harlowja/555ea019aef4e901897b
>>>
>>> Output @ http://paste.ubuntu.com/9813919/
>>>
>>> When ran on the same 'test.txt' mentioned below...
>>>
>>> Happy hacking!
>>>
>>> -Josh
>>>
>>> Joshua Harlow wrote:
>>>> A run that shows more of the happy/desired path:
>>>>
>>>> $ cat test.txt
>>>> six>1
>>>> taskflow<0.5
>>>> $ python pippin.py -r test.txt
>>>> Initial package set:
>>>> - six ['>1']
>>>> - taskflow ['<0.5']
>>>> Deep package set:
>>>> - six ['==1.9.0']
>>>> - taskflow ['==0.4.0']
>>>>
>>>> -Josh
>>>>
>>>> Joshua Harlow wrote:
>>>>> Another thing that I just started whipping together:
>>>>>
>>>>> https://gist.github.com/harlowja/5e39ec5ca9e3f0d9a21f
>>>>>
>>>>> The idea for the above is to use pip to download dependencies, but
>>>>> figure out what versions will work using our own resolver (and our own
>>>>> querying of 'http://pypi.python.org/pypi/%s/json') that just does a
>>>>> very
>>>>> deep search of all requirements (and requirements of requirements...).
>>>>>
>>>>> The idea for that is that the probe() function in that gist will
>>>>> 'freeze' a single requirement then dive down into further requirements
>>>>> and ensure compatibility while that 'diving' (aka, recursion into
>>>>> further requirements) is underway. If a incompatibility is found then
>>>>> the recursion will back-track and try a to freeze a different
>>>>> version of
>>>>> a desired package (and repeat...).
>>>>>
>>>>> To me this kind of deep finding would be a potential way of making
>>>>> this
>>>>> work in a way that basically only uses pip for downloading (and does
>>>>> the
>>>>> deep matching/probing) on our own since once the algorithm above
>>>>> doesn't
>>>>> backtrack and finds a matching set of requirements that will all work
>>>>> together the program can exit (and this set can then be used as the
>>>>> master set for openstack; at that point we might have to tell
>>>>> people to
>>>>> not use pip, or to only use pip --download to fetch the compatible
>>>>> versions).
>>>>>
>>>>> It's not completed but it could be complementary to what others are
>>>>> working on; feel free to hack away :)
>>>>>
>>>>> So far the following works:
>>>>>
>>>>> $ cat test.txt
>>>>> six>1
>>>>> taskflow>1
>>>>>
>>>>> $ python pippin.py -r test.txt
>>>>> Initial package set:
>>>>> - six ['>1']
>>>>> - taskflow ['>1']
>>>>> Traceback (most recent call last):
>>>>> File "pippin.py", line 168, in<module>
>>>>> main()
>>>>> File "pippin.py", line 162, in main
>>>>> matches = probe(initial, {})
>>>>> File "pippin.py", line 139, in probe
>>>>> result = probe(requirements, gathered)
>>>>> File "pippin.py", line 129, in probe
>>>>> m = find_match(pkg_name, req)
>>>>> File "pippin.py", line 112, in find_match
>>>>> return match_available(req.req, find_versions(pkg_name))
>>>>> File "pippin.py", line 108, in match_available
>>>>> " matches '%s' (tried %s)" % (req, looked_in))
>>>>> __main__.NotFoundException: No requirement found that matches
>>>>> 'taskflow>1' (tried ['0.6.1', '0.6.0', '0.5.0', '0.4.0', '0.3.21',
>>>>> '0.2', '0.1.3', '0.1.2', '0.1.1', '0.1'])
>>>>>
>>>>> I suspect all that is needed to add is the code that is marked with
>>>>> FIXME/TODO there and this kind of recursive back-tracking might
>>>>> just do
>>>>> the trick...
>>>>>
>>>>> -Josh
>>>>>
>>>>> Joe Gordon wrote:
>>>>>>
>>>>>> On Fri, Jan 16, 2015 at 12:25 PM, Joe Gordon<joe.gordon0 at gmail.com
>>>>>> <mailto:joe.gordon0 at gmail.com>> wrote:
>>>>>>
>>>>>> We can side step the dependency graphing and ordering issue by
>>>>>> looking at the list of curently installed packages via pip freeze
>>>>>> and not installing dependencies (pip install --no-deps)
>>>>>>
>>>>>> After looking into this further here are the known issues:
>>>>>>
>>>>>> * Partial capping won't work [0], so we need to pin all
>>>>>> dependencies, we can generate this list per file via "pip install
>>>>>> -r" and "pip freeze", but this doesn't address the issue of apt-get
>>>>>> vs pip install. For example in the stable gate we use suds 0.4.1 but
>>>>>> only suds 0.4.0 is available via pip.
>>>>>> * Not all packages are installed in are standard dsvm-tempest env,
>>>>>> so using pip-freeze from that isn't enough
>>>>>> * We need to run this per requirements file and move to using pip
>>>>>> install --no-deps everywhere. As the global-requirements sync
>>>>>> wouldn't work the first time since files don't list all transient
>>>>>> dependencies yet.
>>>>>> * We can still break if a package version is removed from pypi
>>>>>> * in pip-freeze we sometimes install versions lower then our minimum
>>>>>> version (python-libvirt!)
>>>>>>
>>>>>>
>>>>>> Exploring a few ideas here: https://review.openstack.org/#/c/147451/4
>>>>>>
>>>>>>
>>>>>>
>>>>>> [0]
>>>>>> http://lists.openstack.org/pipermail/openstack-dev/2015-January/054156.html
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 16, 2015 at 5:07 AM, Jeremy Stanley<fungi at yuggoth.org
>>>>>> <mailto:fungi at yuggoth.org>> wrote:
>>>>>>
>>>>>> On 2015-01-15 08:44:58 -0500 (-0500), Sean Dague wrote:
>>>>>> [...]
>>>>>>> The other thing that happened was partial capping doesn't work,
>>>>>>> because something else moves forward and breaks you from below. So
>>>>>>> the patch will need to hit everything at once.
>>>>>> Right, and we _have_ to start using stable branches on all
>>>>>> clients/libraries to backport fixes as part of that. This means that
>>>>>> the stable branch management workflow is about to become pervasive
>>>>>> across some teams who were previously (blissfully?) ignorant of it.
>>>>>>
>>>>>>> Unresolved entirely is the tertiary dependencies (not direct
>>>>>>> dependencies of any OpenStack project). That will need another
>>>>>>> mechanism to seed them before any installation happens.
>>>>>> [...]
>>>>>>
>>>>>> I won't go so far as to call it intractable, but I took a stab at it
>>>>>> about a year ago and building the dependency graph properly to be
>>>>>> able to do a depth-first ordering is nontrivial (enough that after
>>>>>> about a week hacking on possible solutions I gave up and switched to
>>>>>> more productive tasks). The primary complications I ran into were
>>>>>> identifying setup_requires in transitive dependencies and dealing
>>>>>> with platform/version-specific dependencies. That said, there's a
>>>>>> very good chance that more recent improvements in setuptools, pip
>>>>>> and virtualenv could make this task easier.
>>>>>>
>>>>>>> That's the things I can think off from the top of my head.
>>>>>> The implementation, from a devstack-gate perspective, is also going
>>>>>> to require a decision on whether we stick with stable/relname for
>>>>>> branches of libraries too or switch to some extended branch mapping
>>>>>> mechanism to be able to track stable/relnum branches for those. And
>>>>>> we're going to need more jobs to ensure that clients (specifically)
>>>>>> retain backward-compatibility from an appdev and end user
>>>>>> perspective since they'll no longer get any testing as server
>>>>>> dependencies on stable branches (due to being capped there).
>>>>>> --
>>>>>> Jeremy Stanley
>>>>>>
>>>>>> __________________________________________________________________________
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>> __________________________________________________________________________
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> __________________________________________________________________________
>>>>
>>>>
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> __________________________________________________________________________
>>>
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev