Open Stack

I recently discovered that this problem goes even deeper, please read all comments on https://github.com/tox-dev/tox/issues/1565

Due to this it seems to be impossible to define zuul jobs that use a specific python version regardless the environment name. If you want to force "linters" job to use python3.8 only under CI/CD, is impossible.

Using basepython=pythonX.Y combined with ignore_basepython_conflict=False seems the only way to enforce version use, and it comes at the cost of not being flexible for developers (as they may not have the exact version that we want to use on CI jobs).

While for unittest jobs we do use pyXY in the environment name, we do not have the same for "linters", "docs". Alternative to add `{pyXY}-linters` seems to be the only option to trick it.

Another (dangerous) approach would be to assure that the only python version available on each nodeset used by tox job is the one we want to use for testing.

I engaged with Bernat Gabor (tox maintainer) on Gitter/tox-dev about this issue and apparently there is no solution. Tox v4 has some changes planned but is far away.


* --discover cannot be used to enforce python detection
* tox own interpreter version cannot be used to enforce version being picked (at least this is what the maintainer told me)

Sadly, after spending a good number of hours on that I am more confused that I was when I started.

With Zuul nodepool images that can change over night on multiple zuul servers (at least 4 i use), it seems that tox-* jobs are joing to be a permanent source of surprises, where we can discover that what they say they test is not really what they did.

Anyone missing a `make foo` command? ;)

Cheers
Sorin

> On 11 May 2020, at 21:02, Sean Mooney <smooney at redhat.com> wrote:
> 
> On Mon, 2020-05-11 at 10:34 -0700, Clark Boylan wrote:
>> Hello everyone,
>> 
>> This has come up a few times on IRC so we are probably well overdue for a email about it. Long story short, if your
>> tox.ini config sets basepython [0] to `python3` and you also use py35, py36, py37, or py38 test targets there is a
>> good chance you are not testing what you intend to be testing. You also need to set ignore_basepython_conflict to true
>> [1].
>> 
>> The reason for this is basepython acts as an override for the python executable to be used when creating tox
>> virtualenvs. `python3` on most platforms indicates a specific python3 version: Ubuntu Xenial 3.5, Ubuntu Bionic and
>> CentOS 8 3.6, and so on. This means that even though you are asking for python 3.7 via py37 you'll get whatever
>> version `python3` is on the running platform. To address this we can set ignore_basepython_conflict and tox will use
>> the version specified as part of the target and not the basepython override.
>> 
>> You might wonder why using basepython is useful at all given this situation. The reason for it is the default python
>> used by tox for virtualenvs is the version of python tox was installed under. This means that if tox is running under
>> python2 it will use python2 for virtualenvs when no other version is set. Since many software projects are now trying
>> to drop python2 support they want to explicitly force python3 in the default case. basepython gets us halfway there,
>> ignore_basepython_conflict the rest of the way.
>> 
>> [0] https://tox.readthedocs.io/en/latest/config.html#conf-basepython
>> [1] https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conflict
>> 
>> Hopefully this helps explain some of tox's odd behavior in a beneficial way. Now go and check your tox.ini files :)
> yep and to reinforce that point we also have this big warning comment nova so that people know  why we do this.
> https://github.com/openstack/nova/blob/master/tox.ini#L4-L7
> in later version of tox ignore_basepython_conflict = True will be the default
> as that is generally less surprisng behavior but with our current min version both are needed.
>> Clark