Open Stack

Hi All,
    I confess up front that I'm pretty green in the is area and there is a lot
of history that I just don't have.  That wont stop me from asking/opening the
discussion.

As I ask in $subject: why do we install with --upgrade in our tox environments?
Are there issues this is fixing/hiding?

I've been seeing a few failures with requirements updates on stable/kilo.  I
expect that this applies to liberty and master BUT we're seeing less of it as
constraints are a good thing on those branches.

I'll use glance_store as an example[2] https://review.openstack.org/#/c/265182
I want to be clear this isn't about this specific library (glance_store or
requests), I'm seeing something similar with testtools, fixtures and other libraries.

Looking at:
 http://logs.openstack.org/18/277018/2/check/gate-glance_store-python27/1691013/tox/py27-1.log.txt
 http://logs.openstack.org/18/277018/2/check/gate-glance_store-python27/1691013/tox/py27-2.log.txt

In py-1.log we (edited for clarity):
pip install --allow-all-external --allow-insecure netaddr -U -rrequirements.txt -rtest-requirements.txt

---
Collecting python-cinderclient<1.2.0,>=1.1.0 (from -r /home/jenkins/workspace/gate-glance_store-python27/requirements.txt (line 8))
  Downloading http://mirror.iad.rax.openstack.org/pypi/packages/py2.py3/p/python-cinderclient/python_cinderclient-1.1.2-py2.py3-none-any.whl (202kB)
...
Collecting requests!=2.4.0,<2.8.0,>=2.2.0 (from -r /home/jenkins/workspace/gate-glance_store-python27/test-requirements.txt (line 6))
  Downloading http://mirror.iad.rax.openstack.org/pypi/packages/2.7/r/requests/requests-2.7.0-py2.py3-none-any.whl (470kB)
---

So we installed requests 2.7.0 as per our current g-r specification.  IIUC We
use the spec from test-requirements as all the requirements+specs from
requirements.txt and test-requirements.txt are processed before looking at the
requirements of each library.  So when we look for the requests library while
processing python-cinderclient requirements we already have a spec that's been
satisfied and move on.

Then in py-2.log we (edited for clarity):
pip install --allow-all-external --allow-insecure netaddr -U -e <the git repo>
---
Requirement already up-to-date: python-cinderclient<1.2.0,>=1.1.0 in ./.tox/py27/lib/python2.7/site-packages (from glance-store==0.4.1.dev16)
...
Collecting requests!=2.4.0,>=2.2.0 (from python-cinderclient<1.2.0,>=1.1.0->glance-store==0.4.1.dev16)
  Downloading http://mirror.iad.rax.openstack.org/pypi/packages/2.7/r/requests/requests-2.9.1-py2.py3-none-any.whl (501kB)
---

Here we upgrade requests because the python-cinderclient is less restrictive[3]
Here we're only looking at requirements.txt which doesn't have a requests
specification so when we process python-cinderclient's requirements (with -U)
we see a "better" requests library install that and then "go bang" [4]

I *think* this particular failure would be "fixed" if we didn't install our
packages with -U.

I know that people are working on enhancing the pip dependency resolver but
that isn't work we can use today.

Again there are alternate solutions for this specific issue but I feel like
removing -U would fix a class of problems, perhaps it'll create another I don't
know.

Discuss :)

Yours Tony.

[1] Footnote deleted in editing and I'm too lazy renumber the rest :D
[2] Just because it's the one I have open in my browser
[3] See https://review.openstack.org/#/c/265182
[4] http://logs.openstack.org/18/277018/2/check/gate-glance_store-python27/1691013/console.html#_2016-02-10_18_35_39_818
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160211/38700de8/attachment.pgp>