[openstack-dev] [all][requirements] Why do we use pip install -U as our install_command

Clark Boylan cboylan at sapwetik.org
Thu Feb 11 03:09:54 UTC 2016


On Wed, Feb 10, 2016, at 05:46 PM, Tony Breeds wrote:
> Hi All,
>     I confess up front that I'm pretty green in the is area and there is
>     a lot
> of history that I just don't have.  That wont stop me from asking/opening
> the
> discussion.
> 
> As I ask in $subject: why do we install with --upgrade in our tox
> environments?
> Are there issues this is fixing/hiding?
> 
> I've been seeing a few failures with requirements updates on stable/kilo.
>  I
> expect that this applies to liberty and master BUT we're seeing less of
> it as
> constraints are a good thing on those branches.
> 
> I'll use glance_store as an example[2]
> https://review.openstack.org/#/c/265182
> I want to be clear this isn't about this specific library (glance_store
> or
> requests), I'm seeing something similar with testtools, fixtures and
> other libraries.
> 
> Looking at:
>  http://logs.openstack.org/18/277018/2/check/gate-glance_store-python27/1691013/tox/py27-1.log.txt
>  http://logs.openstack.org/18/277018/2/check/gate-glance_store-python27/1691013/tox/py27-2.log.txt
> 
> In py-1.log we (edited for clarity):
> pip install --allow-all-external --allow-insecure netaddr -U
> -rrequirements.txt -rtest-requirements.txt
> 
> ---
> Collecting python-cinderclient<1.2.0,>=1.1.0 (from -r
> /home/jenkins/workspace/gate-glance_store-python27/requirements.txt (line
> 8))
>   Downloading
>   http://mirror.iad.rax.openstack.org/pypi/packages/py2.py3/p/python-cinderclient/python_cinderclient-1.1.2-py2.py3-none-any.whl
>   (202kB)
> ...
> Collecting requests!=2.4.0,<2.8.0,>=2.2.0 (from -r
> /home/jenkins/workspace/gate-glance_store-python27/test-requirements.txt
> (line 6))
>   Downloading
>   http://mirror.iad.rax.openstack.org/pypi/packages/2.7/r/requests/requests-2.7.0-py2.py3-none-any.whl
>   (470kB)
> ---
> 
> So we installed requests 2.7.0 as per our current g-r specification. 
> IIUC We
> use the spec from test-requirements as all the requirements+specs from
> requirements.txt and test-requirements.txt are processed before looking
> at the
> requirements of each library.  So when we look for the requests library
> while
> processing python-cinderclient requirements we already have a spec that's
> been
> satisfied and move on.
> 
> Then in py-2.log we (edited for clarity):
> pip install --allow-all-external --allow-insecure netaddr -U -e <the git
> repo>
> ---
> Requirement already up-to-date: python-cinderclient<1.2.0,>=1.1.0 in
> ./.tox/py27/lib/python2.7/site-packages (from glance-store==0.4.1.dev16)
> ...
> Collecting requests!=2.4.0,>=2.2.0 (from
> python-cinderclient<1.2.0,>=1.1.0->glance-store==0.4.1.dev16)
>   Downloading
>   http://mirror.iad.rax.openstack.org/pypi/packages/2.7/r/requests/requests-2.9.1-py2.py3-none-any.whl
>   (501kB)
> ---
> 
> Here we upgrade requests because the python-cinderclient is less
> restrictive[3]
> Here we're only looking at requirements.txt which doesn't have a requests
> specification so when we process python-cinderclient's requirements (with
> -U)
> we see a "better" requests library install that and then "go bang" [4]
> 
> I *think* this particular failure would be "fixed" if we didn't install
> our
> packages with -U.
> 
> I know that people are working on enhancing the pip dependency resolver
> but
> that isn't work we can use today.
> 
> Again there are alternate solutions for this specific issue but I feel
> like
> removing -U would fix a class of problems, perhaps it'll create another I
> don't
> know.
> 
> Discuss :)
> 
> Yours Tony.
> 
> [1] Footnote deleted in editing and I'm too lazy renumber the rest :D
> [2] Just because it's the one I have open in my browser
> [3] See https://review.openstack.org/#/c/265182
> [4]
> http://logs.openstack.org/18/277018/2/check/gate-glance_store-python27/1691013/console.html#_2016-02-10_18_35_39_818

The reason that I remember off the top of my head is because we spent
far too much time telling people to run `tox -r` when their code failed
during Jenkins testing but ran just fine locally. It removes a
significant amount of debugging overhead to have everyone using a
relatively consistent set of packages whenever they rerun tests.

Clark



More information about the OpenStack-dev mailing list