Open Stack

That is indeed a really good move which can be summarized as moving towards unadulterated system images, or at least not modified in a way that would bring them to a state where they are "unsupported".

Preinstalling already existing system packages or setting up mirrors is ok, but other than this probably not.

virtualenv/pip/setuptools prooved to be a very good source of issues.

Thanks
Sorin

> On 30 May 2020, at 01:43, Ian Wienand <iwienand at redhat.com> wrote:
> 
> Hello,
> 
> This is to notify the community of the planned upcoming removal of the
> "pip-and-virtualenv" element from our infra image builds.
> 
> This is part of the "cleanup test node python" spec documented at [1].
> 
> tl;dr
> -----
> 
> - pip and virtualenv tools will no longer be pre-installed on the base
>  images we use for testing, but will be installed by jobs themselves.
> - most current jobs will depend on common Zuul base jobs that have
>  been updated to install these tools already *i.e. in most cases
>  there should be nothing to do*
> - in general, jobs should ensure they depend on roles that install
>  these tools if they require them
> - if you do find your job failing due to the lack of either of these
>  tools, use the ensure-pip or ensure-virtualenv roles provided in
>  zuul-jobs
> - we have "-plain" node types that implement this now.  If you think
>  there might be a problem, switch the job to run run against one of
>  these node types described below for testing.
> 
> History
> -------
> 
> The "pip-and-virtualenv" element is part of diskimage-builder [2] and
> installs the latest version of pip and virtualenv, and related tools
> like setuptools, into the system environment for the daily builds.
> This has a long history of working around various issues, but has
> become increasingly problematic to maintain.
> 
> One of the more noticable effects is that to prevent the distribution
> packages overwriting the upstream installs, we put various pip,
> setuptools and virtualenv packages on "hold" in CI images.  This
> prevents tests resinstalling packaged tools which can create, in
> short, a big mess.  Both destroying the extant environment and having
> the tools on hold have been problems for various projects at various
> times.
> 
> Another other problem is what happens when you call "pip" or
> "virtualenv" has diverged.  It used to be that "pip" would install the
> package under Python2, while pip3 would install under python3.
> However, modern distributions now have "pip" installing under Python
> 3.  To keep having "pip" install Python 2 packages on these platforms
> is not just wrong, but drags in Python 2 dependences to our images
> that aren't required.
> 
> The addition of Python 3's venv and the split with virtualenv makes
> things even more confusing again.
> 
> Future
> ------
> 
> As famously said "the only winning move is not to play".  [3]
> 
> By dropping this element, images will not have non-packaged
> pip/virtualenv/setuptools/etc. pre-installed.  No packages will be on
> hold.
> 
> The "ensure-pip" role [4] will ensure that dependencies for "pip:"
> commands will work.  Because of the way base jobs are setup, it is
> most likely that this role has already run.  If you wish to use a
> virtual environment to install a tool, I would suggest using the
> "ensure_pip_virtualenv_cmd" this role exports.  This will default to
> "python3 -m venv".  An example is [5].
> 
> In a Python 3 word, you probably do not *need* virtualenv; "python3 -m
> venv" is available and works after the "ensure-pip" role is run.
> However, if you require some of the features that virtualenv provides
> that venv does not (explained at [6]) there is a role
> "ensure-virtualenv" [7].  For example, we do this on the devstack
> branches because it is common to use "virtualenv" there due to the
> long history [8].
> 
> If you need specific versions of pip or virtualenv, etc. beyond the
> system-packaged versions installed with the above, you should have
> your job configure these.  There is absolutely no problem with jobs
> installing differing versions of pip/virtuatlenv/etc in any way they
> want -- we just don't want the base images to have any of that by
> default.  Of course, you should consider if you're building/testing
> something that is actually useful outside the gate, but that is a
> global concern.
> 
> Testing
> -------
> 
> We have built parallel nodes with the suffix "-plain" where you can
> test any jobs in the new environment.  For example [9] speculatively
> tests devstack.  The node types available are
> 
> centos-7-plain
> centos-8-plain
> ubuntu-xenial-plain
> ubuntu-bionic-plain
> 
> The newer focal images do not have pip pre-installed, neither do the
> faster moving Fedora images, any SUSE images, or any ARM64 images.
> 
> Rollout
> -------
> 
> We would like to make the switch soon, to shake out any issues early
> in the cycle.  This would mean on or about the 8th June.
> 
> Thanks,
> 
> -i
> 
> [1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html
> [2] https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/pip-and-virtualenv
> [3] https://en.wikipedia.org/wiki/WarGames
> [4] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#role-ensure-pip
> [5] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/bindep/tasks/install.yaml#L9
> [6] https://virtualenv.pypa.io/en/latest/
> [7] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#role-ensure-virtualenv
> [8] https://opendev.org/openstack/devstack/commit/23cfb9e6ebc63a4da4577c0ef9e3450b9c946fa7
> [9] https://review.opendev.org/#/c/712211/11/.zuul.yaml
> 
>