Open Stack

Excerpts from Clark Boylan's message of 2017-10-20 13:14:13 -0700:
> On Fri, Oct 20, 2017, at 11:17 AM, Clark Boylan wrote:
> > On Fri, Oct 20, 2017, at 07:23 AM, Doug Hellmann wrote:
> > > It sounds like the PyPI/PyPA folks are planning some major changes to
> > > pip internals, soon.
> > > 
> > > I know pbr uses setuptools, and I don't think it uses pip, but if
> > > someone has time to verify that it would be helpful.
> > > 
> > > We'll also want to watch out for breakage in normal use of pip 10. If
> > > they're making changes this big, they may miss something in their own
> > > test coverage that affects our jobs.
> > > 
> > 
> > After a quick skim of PBR I don't think we use pip internals anywhere.
> > Its all executed via the command itself. That said we should test this
> > so I've put up https://review.openstack.org/513825 (others should feel
> > free to iterate on it if it doesn't work) to install latest pip master
> > in a devstack run.
> 
> The current issue this change is facing can be seen at
> http://logs.openstack.org/25/513825/4/check/legacy-tempest-dsvm-py35/c31deb2/logs/devstacklog.txt.gz#_2017-10-20_20_07_54_838.
> The tl;dr is that for distutils installed packages (basically all the
> distro installed python packges) pip refuses to uninstall them in order
> to perform upgrades because it can't reliably determine where all the
> files are. I think this is a new pip 10 behavior.
> 
> In the general case I think this means we can not rely on global pip
> installs anymore. This may be a good thing to bring up with upstream
> PyPA as I expect it will break a lot of people in a lot of places (it
> will break infra for example too).
> 
> Clark
> 

Yes, it would be good for someone who understands the cases where we're
doing these sorts of global package installations using pip to post on
distutils-sig explaining the breakage and asking for some time to let us
work around the issue.

We have a couple of ways to mitigate it, depending on the situation.

1. Pin the affected packages to the system package versions in our
   constraints list.

2. Install into a virtualenv that ignores system packages, avoiding the
   need to upgrade at all.

3. Remove the system package before installing from pip (why is it there
   in the first place?).

It's not clear to me which is the best approach.  I've added
[devstack][qa] to the subject line to draw some more attention to
this thread from folks familiar with these jobs.

Doug