Following up on this issue, I've sent up a patch with a permanent fix. There are a few issues related to various releases of virtualenv and distribution packaging that make this a bit tricky. For CentOS, Loci follows the recommentdations of the RDO team to not install EPEL[1], since there may be incompatabilities between RDO and EPEL. Because of this, we can't assume that pip will be available by default. virtualenv is available, and is a way to get a version of pip that can bootstrap the pip and updated virtualenv installation. Handling of symlinks to virtualenvs has been buggy and inconsistent for quite a while. For the CentOS packaged version of virtualenv, existing python libraries are copied directly over to the virtualenv. In 16.4.0, this behavior changed to symlink to the parent library location. In the instance where you have nested virtualenvs (which Loci does, because this is how it bootstraps the build environment), this means the build venv links back to the bootstrap venv[2]. Previously we deleted the bootstrap venv. The fix is to hold on to it, preserving the chain of links[3]. This solution isn't ideal, and it's worthwhile to rethink how we get pip and other dependencies onto the build hosts. It's a bigger discussion that the Loci team will have in the coming months. Regardless of that, Loci builds now work with the temporary patch that landed a few days ago and with the more "permanent" patch that is in flight. [1] https://docs.openstack.org/install-guide/environment-packages-rdo.html [2] https://github.com/pypa/virtualenv/pull/1309 [3] https://review.openstack.org/#/c/636447/2
On Feb 11, 2019, at 5:12 PM, Chris Hoge <chris@openstack.org> wrote:
A patch for a temporary fix is up for review.
https://review.openstack.org/#/c/636252/
We’ll be looking into a more permanent fix in the coming days.
On Feb 11, 2019, at 4:38 PM, Chris Hoge <chris@openstack.org> wrote:
It appears the lastest release of virtualenv has broken Loci builds. I believe the root cause is an update in how symlinks are handled. Before the release, the python libraries installed in the:
/var/lib/openstack/lib64/python2.7/lib-dynload
directory (this is on CentOS, Ubuntu and Suse vary) were direct instances of the library. For example:
-rwxr-xr-x. 1 root root 62096 Oct 30 23:46 itertoolsmodule.so
Now, the build points to a long-destroyed symlink that is an artifact of the requirements build process. For example:
lrwxrwxrwx. 1 root root 56 Feb 11 23:01 itertoolsmodule.so -> /tmp/venv/lib64/python2.7/lib-dynload/itertoolsmodule.so
We will investigate how to make the build more robust, repair this, and will report back soon. Until then, you should expect any fresh builds to not be functional, despite the apparent success in building the container.
Thanks, Chris
[1] https://virtualenv.pypa.io/en/stable/changes/#release-history