Tox basepython and Python3
Hello everyone, This has come up a few times on IRC so we are probably well overdue for a email about it. Long story short, if your tox.ini config sets basepython [0] to `python3` and you also use py35, py36, py37, or py38 test targets there is a good chance you are not testing what you intend to be testing. You also need to set ignore_basepython_conflict to true [1]. The reason for this is basepython acts as an override for the python executable to be used when creating tox virtualenvs. `python3` on most platforms indicates a specific python3 version: Ubuntu Xenial 3.5, Ubuntu Bionic and CentOS 8 3.6, and so on. This means that even though you are asking for python 3.7 via py37 you'll get whatever version `python3` is on the running platform. To address this we can set ignore_basepython_conflict and tox will use the version specified as part of the target and not the basepython override. You might wonder why using basepython is useful at all given this situation. The reason for it is the default python used by tox for virtualenvs is the version of python tox was installed under. This means that if tox is running under python2 it will use python2 for virtualenvs when no other version is set. Since many software projects are now trying to drop python2 support they want to explicitly force python3 in the default case. basepython gets us halfway there, ignore_basepython_conflict the rest of the way. [0] https://tox.readthedocs.io/en/latest/config.html#conf-basepython [1] https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conf... Hopefully this helps explain some of tox's odd behavior in a beneficial way. Now go and check your tox.ini files :) Clark
On a related note, the proliferation of tested Python versions has led many projects to enable the skip_missing_interpreters option in their tox.ini files. Please don't, this is just plain DANGEROUS. I know it's nice that when you've got a bunch of Python versions in your default tox envlist some of which a typical developer may not have installed, they can still run `tox` and not get errors about those. However, it also means that if you run `tox -e py38` and don't have any Python 3.8 interpreter, tox will happily say it did nothing successfully. Yes it's fairly obvious when you see it happen locally. It's far less obvious when you add a Python 3.8 job in the gate but don't make sure that interpreter is actually installed then and get back a +1 from Zuul when tox ran no tests at all. An alternative solution, which some projects like Zuul have switched to, is not listing a bunch of specific pyXY versions in the tox envlist, but just putting "py3" instead. This will cause folks who are running `tox` to get unit tests with whatever their default python3 interpreter is, but also they'll get a clear error if they don't have any python3 interpreter at all. If someone has Python 3.8 installed and it isn't their default python3 but they still want to test with it, they can of course do `tox -e py38` and that will work as expected. This also means you no longer have to update the envlist in tox.ini every time you add or remove support for a specific interpreter version. Besides, tox.ini is not a good place to list what versions of the interpreter your project supports, that's what trove classifiers in the setup.cfg file are for. -- Jeremy Stanley
On Mon, 2020-05-11 at 18:20 +0000, Jeremy Stanley wrote:
On a related note, the proliferation of tested Python versions has led many projects to enable the skip_missing_interpreters option in their tox.ini files. Please don't, this is just plain DANGEROUS.
The 'py3' option listed below is a better one, but for those that *really* want this behavior: alias 'tox=tox --skip-missing-interpreters' Your tests all "pass" locally, but the gate continues to properly test things. Alternatively, use Fedora where all supported Python versions are packaged by default and this isn't an issue :P Stephen
I know it's nice that when you've got a bunch of Python versions in your default tox envlist some of which a typical developer may not have installed, they can still run `tox` and not get errors about those. However, it also means that if you run `tox -e py38` and don't have any Python 3.8 interpreter, tox will happily say it did nothing successfully. Yes it's fairly obvious when you see it happen locally. It's far less obvious when you add a Python 3.8 job in the gate but don't make sure that interpreter is actually installed then and get back a +1 from Zuul when tox ran no tests at all.
An alternative solution, which some projects like Zuul have switched to, is not listing a bunch of specific pyXY versions in the tox envlist, but just putting "py3" instead. This will cause folks who are running `tox` to get unit tests with whatever their default python3 interpreter is, but also they'll get a clear error if they don't have any python3 interpreter at all. If someone has Python 3.8 installed and it isn't their default python3 but they still want to test with it, they can of course do `tox -e py38` and that will work as expected. This also means you no longer have to update the envlist in tox.ini every time you add or remove support for a specific interpreter version. Besides, tox.ini is not a good place to list what versions of the interpreter your project supports, that's what trove classifiers in the setup.cfg file are for.
On Tue, 2020-05-12 at 10:48 +0100, Stephen Finucane wrote:
On Mon, 2020-05-11 at 18:20 +0000, Jeremy Stanley wrote:
On a related note, the proliferation of tested Python versions has led many projects to enable the skip_missing_interpreters option in their tox.ini files. Please don't, this is just plain DANGEROUS. why is this dangourse it wont casue the ci jobs to be skipped since we guarentee the interperter will be present
i was planning to add this to cybrog in https://review.opendev.org/#/c/708705/1/tox.ini and had planned to add it to nova too sowhat is the reason for considering it dangerous. i think we shoudl be adding that to all project by default so if there is a stong reason not to do this i would like to hear why.
The 'py3' option listed below is a better one, but for those that *really* want this behavior:
alias 'tox=tox --skip-missing-interpreters'
Your tests all "pass" locally, but the gate continues to properly test things.
Alternatively, use Fedora where all supported Python versions are packaged by default and this isn't an issue :P
Stephen
I know it's nice that when you've got a bunch of Python versions in your default tox envlist some of which a typical developer may not have installed, they can still run `tox` and not get errors about those. However, it also means that if you run `tox -e py38` and don't have any Python 3.8 interpreter, tox will happily say it did nothing successfully. Yes it's fairly obvious when you see it happen locally. It's far less obvious when you add a Python 3.8 job in the gate but don't make sure that interpreter is actually installed then and get back a +1 from Zuul when tox ran no tests at all.
An alternative solution, which some projects like Zuul have switched to, is not listing a bunch of specific pyXY versions in the tox envlist, but just putting "py3" instead. This will cause folks who are running `tox` to get unit tests with whatever their default python3 interpreter is, but also they'll get a clear error if they don't have any python3 interpreter at all. If someone has Python 3.8 installed and it isn't their default python3 but they still want to test with it, they can of course do `tox -e py38` and that will work as expected. This also means you no longer have to update the envlist in tox.ini every time you add or remove support for a specific interpreter version. Besides, tox.ini is not a good place to list what versions of the interpreter your project supports, that's what trove classifiers in the setup.cfg file are for.
On 2020-05-12 13:43:12 +0100 (+0100), Sean Mooney wrote:
On Tue, 2020-05-12 at 10:48 +0100, Stephen Finucane wrote:
On Mon, 2020-05-11 at 18:20 +0000, Jeremy Stanley wrote:
On a related note, the proliferation of tested Python versions has led many projects to enable the skip_missing_interpreters option in their tox.ini files. Please don't, this is just plain DANGEROUS.
why is this dangourse it wont casue the ci jobs to be skipped since we guarentee the interperter will be present
Yes, the current tox-py38 in zuul-jobs sets "python_version: 3.8" in the vars list which the parent job passes to the ensure-python role. This is true for all current the tox-pyXY jobs there except tox-py27 (I'm not entirely sure why the discrepancy). The ensure-python role, in its present state, will make sure the relevant pythonX.Y and pythonX.Y-dev packages are installed on Debian and Ubuntu nodes, but does nothing on other platforms. So it's probably fairly safe for upstream testing with Zuul if the project is only inheriting from the predefined tox-py3* jobs and using the default (Ubuntu based) nodeset. If the project defines its own job and neglects to set python_version or doesn't descend from the generic tox job in zuul-jobs or sets a different nodeset, then you'll get a successful build which ran no tests. That's a lot of caveats. We saw this first hand when OpenStack initially added its own py38 jobs, and have seen it crop up in third-party CI systems (as well as confuse some new developers when they're running tox locally). -- Jeremy Stanley
On Mon, 2020-05-11 at 10:34 -0700, Clark Boylan wrote:
Hello everyone,
This has come up a few times on IRC so we are probably well overdue for a email about it. Long story short, if your tox.ini config sets basepython [0] to `python3` and you also use py35, py36, py37, or py38 test targets there is a good chance you are not testing what you intend to be testing. You also need to set ignore_basepython_conflict to true [1].
The reason for this is basepython acts as an override for the python executable to be used when creating tox virtualenvs. `python3` on most platforms indicates a specific python3 version: Ubuntu Xenial 3.5, Ubuntu Bionic and CentOS 8 3.6, and so on. This means that even though you are asking for python 3.7 via py37 you'll get whatever version `python3` is on the running platform. To address this we can set ignore_basepython_conflict and tox will use the version specified as part of the target and not the basepython override.
You might wonder why using basepython is useful at all given this situation. The reason for it is the default python used by tox for virtualenvs is the version of python tox was installed under. This means that if tox is running under python2 it will use python2 for virtualenvs when no other version is set. Since many software projects are now trying to drop python2 support they want to explicitly force python3 in the default case. basepython gets us halfway there, ignore_basepython_conflict the rest of the way.
[0] https://tox.readthedocs.io/en/latest/config.html#conf-basepython [1] https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conf...
Hopefully this helps explain some of tox's odd behavior in a beneficial way. Now go and check your tox.ini files :) yep and to reinforce that point we also have this big warning comment nova so that people know why we do this. https://github.com/openstack/nova/blob/master/tox.ini#L4-L7 in later version of tox ignore_basepython_conflict = True will be the default as that is generally less surprisng behavior but with our current min version both are needed. Clark
I recently discovered that this problem goes even deeper, please read all comments on https://github.com/tox-dev/tox/issues/1565 Due to this it seems to be impossible to define zuul jobs that use a specific python version regardless the environment name. If you want to force "linters" job to use python3.8 only under CI/CD, is impossible. Using basepython=pythonX.Y combined with ignore_basepython_conflict=False seems the only way to enforce version use, and it comes at the cost of not being flexible for developers (as they may not have the exact version that we want to use on CI jobs). While for unittest jobs we do use pyXY in the environment name, we do not have the same for "linters", "docs". Alternative to add `{pyXY}-linters` seems to be the only option to trick it. Another (dangerous) approach would be to assure that the only python version available on each nodeset used by tox job is the one we want to use for testing. I engaged with Bernat Gabor (tox maintainer) on Gitter/tox-dev about this issue and apparently there is no solution. Tox v4 has some changes planned but is far away. * --discover cannot be used to enforce python detection * tox own interpreter version cannot be used to enforce version being picked (at least this is what the maintainer told me) Sadly, after spending a good number of hours on that I am more confused that I was when I started. With Zuul nodepool images that can change over night on multiple zuul servers (at least 4 i use), it seems that tox-* jobs are joing to be a permanent source of surprises, where we can discover that what they say they test is not really what they did. Anyone missing a `make foo` command? ;) Cheers Sorin
On 11 May 2020, at 21:02, Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2020-05-11 at 10:34 -0700, Clark Boylan wrote:
Hello everyone,
This has come up a few times on IRC so we are probably well overdue for a email about it. Long story short, if your tox.ini config sets basepython [0] to `python3` and you also use py35, py36, py37, or py38 test targets there is a good chance you are not testing what you intend to be testing. You also need to set ignore_basepython_conflict to true [1].
The reason for this is basepython acts as an override for the python executable to be used when creating tox virtualenvs. `python3` on most platforms indicates a specific python3 version: Ubuntu Xenial 3.5, Ubuntu Bionic and CentOS 8 3.6, and so on. This means that even though you are asking for python 3.7 via py37 you'll get whatever version `python3` is on the running platform. To address this we can set ignore_basepython_conflict and tox will use the version specified as part of the target and not the basepython override.
You might wonder why using basepython is useful at all given this situation. The reason for it is the default python used by tox for virtualenvs is the version of python tox was installed under. This means that if tox is running under python2 it will use python2 for virtualenvs when no other version is set. Since many software projects are now trying to drop python2 support they want to explicitly force python3 in the default case. basepython gets us halfway there, ignore_basepython_conflict the rest of the way.
[0] https://tox.readthedocs.io/en/latest/config.html#conf-basepython [1] https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conf...
Hopefully this helps explain some of tox's odd behavior in a beneficial way. Now go and check your tox.ini files :) yep and to reinforce that point we also have this big warning comment nova so that people know why we do this. https://github.com/openstack/nova/blob/master/tox.ini#L4-L7 in later version of tox ignore_basepython_conflict = True will be the default as that is generally less surprisng behavior but with our current min version both are needed. Clark
On 5/12/20 9:44 AM, Sorin Sbarnea wrote:
I recently discovered that this problem goes even deeper, please read all comments on https://github.com/tox-dev/tox/issues/1565
Due to this it seems to be impossible to define zuul jobs that use a specific python version regardless the environment name. If you want to force "linters" job to use python3.8 only under CI/CD, is impossible.
Using basepython=pythonX.Y combined with ignore_basepython_conflict=False seems the only way to enforce version use, and it comes at the cost of not being flexible for developers (as they may not have the exact version that we want to use on CI jobs).
While for unittest jobs we do use pyXY in the environment name, we do not have the same for "linters", "docs". Alternative to add `{pyXY}-linters` seems to be the only option to trick it.
Is there a reason you need a specific py3 version for these jobs? Normally, at least in OpenStack, we just want these jobs to not use py27. Which version of py3 is used to build docs is usually not as big of a concern. Sean
On Tue, May 12, 2020, at 7:44 AM, Sorin Sbarnea wrote:
I recently discovered that this problem goes even deeper, please read all comments on https://github.com/tox-dev/tox/issues/1565
Due to this it seems to be impossible to define zuul jobs that use a specific python version regardless the environment name. If you want to force "linters" job to use python3.8 only under CI/CD, is impossible.
Using basepython=pythonX.Y combined with ignore_basepython_conflict=False seems the only way to enforce version use, and it comes at the cost of not being flexible for developers (as they may not have the exact version that we want to use on CI jobs).
While for unittest jobs we do use pyXY in the environment name, we do not have the same for "linters", "docs". Alternative to add `{pyXY}-linters` seems to be the only option to trick it.
I don't think that is a trick, this is an intentional feature of tox to solve the problem you have. If you want a specific version of python to be used tox aims to help you do that via the pyXY* environments. https://tox.readthedocs.io/en/latest/config.html#tox-environments clearly describes this behavior.
Another (dangerous) approach would be to assure that the only python version available on each nodeset used by tox job is the one we want to use for testing.
I engaged with Bernat Gabor (tox maintainer) on Gitter/tox-dev about this issue and apparently there is no solution. Tox v4 has some changes planned but is far away.
* --discover cannot be used to enforce python detection * tox own interpreter version cannot be used to enforce version being picked (at least this is what the maintainer told me)
Sadly, after spending a good number of hours on that I am more confused that I was when I started.
With Zuul nodepool images that can change over night on multiple zuul servers (at least 4 i use), it seems that tox-* jobs are joing to be a permanent source of surprises, where we can discover that what they say they test is not really what they did.
In the case of linting, without a specific python version set, they are doing what you asked: run the linters without a specific version of python and use what is available. The problem I described earlier was asking for a specific python version and not getting that version. Running tox -e linters does not ask for a specific version. The workaround you described above is the actual tox solution to this problem. You should run py38-linters if that is your intent.
Anyone missing a `make foo` command? ;)
Cheers Sorin
---- On Tue, 12 May 2020 10:01:40 -0500 Clark Boylan <cboylan@sapwetik.org> wrote ----
On Tue, May 12, 2020, at 7:44 AM, Sorin Sbarnea wrote:
I recently discovered that this problem goes even deeper, please read all comments on https://github.com/tox-dev/tox/issues/1565
Due to this it seems to be impossible to define zuul jobs that use a specific python version regardless the environment name. If you want to force "linters" job to use python3.8 only under CI/CD, is impossible.
Using basepython=pythonX.Y combined with ignore_basepython_conflict=False seems the only way to enforce version use, and it comes at the cost of not being flexible for developers (as they may not have the exact version that we want to use on CI jobs).
While for unittest jobs we do use pyXY in the environment name, we do not have the same for "linters", "docs". Alternative to add `{pyXY}-linters` seems to be the only option to trick it.
I don't think that is a trick, this is an intentional feature of tox to solve the problem you have. If you want a specific version of python to be used tox aims to help you do that via the pyXY* environments. https://tox.readthedocs.io/en/latest/config.html#tox-environments clearly describes this behavior.
Another (dangerous) approach would be to assure that the only python version available on each nodeset used by tox job is the one we want to use for testing.
I engaged with Bernat Gabor (tox maintainer) on Gitter/tox-dev about this issue and apparently there is no solution. Tox v4 has some changes planned but is far away.
* --discover cannot be used to enforce python detection * tox own interpreter version cannot be used to enforce version being picked (at least this is what the maintainer told me)
Sadly, after spending a good number of hours on that I am more confused that I was when I started.
With Zuul nodepool images that can change over night on multiple zuul servers (at least 4 i use), it seems that tox-* jobs are joing to be a permanent source of surprises, where we can discover that what they say they test is not really what they did.
In the case of linting, without a specific python version set, they are doing what you asked: run the linters without a specific version of python and use what is available. The problem I described earlier was asking for a specific python version and not getting that version. Running tox -e linters does not ask for a specific version.
The workaround you described above is the actual tox solution to this problem. You should run py38-linters if that is your intent.
Correct. For all other tox env like for unit tests which can have a conflict for python version, we already use ignore_basepython_conflict=True with tox=3.1 as min version in all projects (if I have not missed any). That is what we did during py2.7 drop where we defined the basepython=python3 in common testenv. -gmann
Anyone missing a `make foo` command? ;)
Cheers Sorin
participants (7)
-
Clark Boylan
-
Ghanshyam Mann
-
Jeremy Stanley
-
Sean McGinnis
-
Sean Mooney
-
Sorin Sbarnea
-
Stephen Finucane