Open Stack

On Thu, Apr 5, 2018 at 9:27 PM Clark Boylan <cboylan at sapwetik.org> wrote:

> On Mon, Apr 2, 2018, at 9:13 AM, Clark Boylan wrote:
> > On Mon, Apr 2, 2018, at 8:06 AM, Matthew Thode wrote:
> > > On 18-03-31 15:00:27, Jeremy Stanley wrote:
> > > > According to a notice[1] posted to the pypa-announce and
> > > > distutils-sig mailing lists, pip 10.0.0.b1 is on PyPI now and 10.0.0
> > > > is expected to be released in two weeks (over the April 14/15
> > > > weekend). We know it's at least going to start breaking[2] DevStack
> > > > and we need to come up with a plan for addressing that, but we don't
> > > > know how much more widespread the problem might end up being so
> > > > encourage everyone to try it out now where they can.
> > > >
> > >
> > > I'd like to suggest locking down pip/setuptools/wheel like openstack
> > > ansible is doing in
> > >
> https://github.com/openstack/openstack-ansible/blob/master/global-requirement-pins.txt
> > >
> > > We could maintain it as a separate constraints file (or infra could
> > > maintian it, doesn't mater).  The file would only be used for the
> > > initial get-pip install.
> >
> > In the past we've done our best to avoid pinning these tools because 1)
> > we've told people they should use latest for openstack to work and 2) it
> > is really difficult to actually control what versions of these tools end
> > up on your systems if not latest.
> >
> > I would strongly push towards addressing the distutils package deletion
> > problem that we've run into with pip10 instead. One of the approaches
> > thrown out that pabelanger is working on is to use a common virtualenv
> > for devstack and avoid the system package conflict entirely.
>
> I was mistaken and pabelanger was working to get devstack's USE_VENV
> option working which installs each service (if the service supports it)
> into its own virtualenv. There are two big drawbacks to this. This first is
> that we would lose coinstallation of all the openstack services which is
> one way we ensure they all work together at the end of the day. The second
> is that not all services in "base" devstack support USE_VENV and I doubt
> many plugins do either (neutron apparently doesn't?).
>
> I've since worked out a change that passes tempest using a global
> virtualenv installed devstack at https://review.openstack.org/#/c/558930/.
> This needs to be cleaned up so that we only check for and install the
> virtualenv(s) once and we need to handle mixed python2 and python3
> environments better (so that you can run a python2 swift and python3
> everything else).
>
> The other major issue we've run into is that nova file injection (which is
> tested by tempest) seems to require either libguestfs or nbd. libguestfs
> bindings for python aren't available on pypi and instead we get them from
> system packaging. This means if we want libguestfs support we have to
> enable system site packages when using virtualenvs. The alternative is to
> use nbd which apparently isn't preferred by nova and doesn't work under
> current devstack anyways.
>
> Why is this a problem? Well the new pip10 behavior that breaks devstack is
> pip10's refusable to remove distutils installed packages. Distro packages
> by and large are distutils packaged which means if you mix system packages
> and pip installed packages there is a good chance something will break (and
> it does break for current devstack). I'm not sure that using a virtualenv
> with system site packages enabled will sufficiently protect us from this
> case (but we should test it further). Also it feels wrong to enable system
> packages in a virtualenv if the entire point is avoiding system python
> packages.
>
> I'm not sure what the best option is here but if we can show that system
> site packages with virtualenvs is viable with pip10 and people want to move
> forward with devstack using a global virtualenv we can work to clean up
> this change and make it mergeable.
>

Thanks Clark for looking into this.

One of the things that will break using a global virtual env is the
"all-plugin" Tempest tox environment, which is still used in a few places
[0].
The "all-plugin" tox environment is deprecated anyways, so this may
actually push things moving in the right direction.

Some background the "all-plugin": Tempest plugins used to live in tree for
many projects - for Tempest to discover those plugins, "all-plugin"
installs Tempest in a virtual environment with system site-packages
enabled. After the Tempest plugin community goal in Queens, most plugins
are now hosted in a dedicated repository, and "all-plugin" should not be
needed anymore.

Andrea Frittoli (andreaf)

[0] http://codesearch.openstack.org/?q=all-plugin&i=nope&files=&repos=


>
> Clark
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180407/8b31b6a1/attachment.html>