[all][qa][infra] Restore unit tests for intermediate python versions
Hello, As you may know, current master runs unit tests with only Python 3.9 and Python 3.12. My understanding about this set up is that we test the minimum boundary and the maximum boundary and assume the change may work with all the versions between these two. However in some of my recent work I had to introduce the switch in the middle of these two versions. For example https://review.opendev.org/c/openstack/neutron/+/931271 introduces the logic to distinguish Python < 3.10 from Python >= 3.10 , and we don't specifically test that boundary. I'm afraid that this may potentially cause uncaught problems. So my question here is ... Can we test all supported python versions, not only min/max, at least in check jobs ? This would helps us catch any regressions caused by such implementation. I know there could be tradeoffs between test coverage and infra resources, but IMO this is the test coverage we should fill. Thank you, -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
On Fri, Oct 4, 2024, at 9:30 AM, Takashi Kajinami wrote:
Hello,
As you may know, current master runs unit tests with only Python 3.9 and Python 3.12. My understanding about this set up is that we test the minimum boundary and the maximum boundary and assume the change may work with all the versions between these two.
However in some of my recent work I had to introduce the switch in the middle of these two versions. For example https://review.opendev.org/c/openstack/neutron/+/931271 introduces the logic to distinguish Python < 3.10 from Python >= 3.10 , and we don't specifically test that boundary. I'm afraid that this may potentially cause uncaught problems.
One approach that wasn't discussed in the change is to continue to use the existing code (pkg_resources) until python3.12 then only change the behavior for 3.12. That approach would generally work if we are only testing the boundaries since we're flipping the behavior on the boundary.
So my question here is ... Can we test all supported python versions, not only min/max, at least in check jobs ? This would helps us catch any regressions caused by such implementation.
We can. The main limitation is that we may not have a test platform for every version of python depending on how the distro releases align with python releases. Currently I think we have a platform for every release though (centos/rocky 3.9, jammy 3.10, bookworm 3.11, noble 3.12). If we end up without a platform for a version of python jobs may need to be updated to install python from source or some other method.
I know there could be tradeoffs between test coverage and infra resources, but IMO this is the test coverage we should fill.
Yes, the main concern was simply to avoid unnecessary effort. In the case of unittest jobs the vast majority of those are much cheaper in terms of resources than tempest jobs. I think if we stuck to this primarily for unittests the impact would be minimal. If we tried to expand the tempest matrix that would become unwieldy.
Thank you,
-- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
On 10/5/24 01:47, Clark Boylan wrote:
On Fri, Oct 4, 2024, at 9:30 AM, Takashi Kajinami wrote:
Hello,
As you may know, current master runs unit tests with only Python 3.9 and Python 3.12. My understanding about this set up is that we test the minimum boundary and the maximum boundary and assume the change may work with all the versions between these two.
However in some of my recent work I had to introduce the switch in the middle of these two versions. For example https://review.opendev.org/c/openstack/neutron/+/931271 introduces the logic to distinguish Python < 3.10 from Python >= 3.10 , and we don't specifically test that boundary. I'm afraid that this may potentially cause uncaught problems.
One approach that wasn't discussed in the change is to continue to use the existing code (pkg_resources) until python3.12 then only change the behavior for 3.12. That approach would generally work if we are only testing the boundaries since we're flipping the behavior on the boundary.
For this case we need python 3.11 job to test the boundary. Also this leaves noisy deprecation warnings in < Python 3.11 .
So my question here is ... Can we test all supported python versions, not only min/max, at least in check jobs ? This would helps us catch any regressions caused by such implementation.
We can. The main limitation is that we may not have a test platform for every version of python depending on how the distro releases align with python releases. Currently I think we have a platform for every release though (centos/rocky 3.9, jammy 3.10, bookworm 3.11, noble 3.12). If we end up without a platform for a version of python jobs may need to be updated to install python from source or some other method.
acknowledged
I know there could be tradeoffs between test coverage and infra resources, but IMO this is the test coverage we should fill.
Yes, the main concern was simply to avoid unnecessary effort. In the case of unittest jobs the vast majority of those are much cheaper in terms of resources than tempest jobs. I think if we stuck to this primarily for unittests the impact would be minimal. If we tried to expand the tempest matrix that would become unwieldy.
Yes. I don't intend to run all tests in all versions but at least having unit test jobs, which are supposed to be cheap, may avoid 0 test coverage and would be enough now. In case we find any real problems then we can expand the scope to other jobs.
Thank you,
-- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
On Fri, Oct 4, 2024, at 10:26 AM, Takashi Kajinami wrote:
On 10/5/24 01:47, Clark Boylan wrote:
On Fri, Oct 4, 2024, at 9:30 AM, Takashi Kajinami wrote:
Hello,
As you may know, current master runs unit tests with only Python 3.9 and Python 3.12. My understanding about this set up is that we test the minimum boundary and the maximum boundary and assume the change may work with all the versions between these two.
However in some of my recent work I had to introduce the switch in the middle of these two versions. For example https://review.opendev.org/c/openstack/neutron/+/931271 introduces the logic to distinguish Python < 3.10 from Python >= 3.10 , and we don't specifically test that boundary. I'm afraid that this may potentially cause uncaught problems.
One approach that wasn't discussed in the change is to continue to use the existing code (pkg_resources) until python3.12 then only change the behavior for 3.12. That approach would generally work if we are only testing the boundaries since we're flipping the behavior on the boundary.
For this case we need python 3.11 job to test the boundary. Also this leaves noisy deprecation warnings in < Python 3.11 .
The lower bound (currently 3.9) would continue to test the old behavior (pkg_resources) and the upper bound (python3.12) would test the new behavior (importlib). It would be covered as well as all of the other upper and lower bound testing we've done. Python3.10 and 3.11 would be "covered" by the 3.9 test ensuring that pkg_resources continues to work. When 3.9 is dropped 3.10 would take over this responsibility. Eventually we'll only have the importlib behavior and python3.12/3.13/etc would cover it.
So my question here is ... Can we test all supported python versions, not only min/max, at least in check jobs ? This would helps us catch any regressions caused by such implementation.
We can. The main limitation is that we may not have a test platform for every version of python depending on how the distro releases align with python releases. Currently I think we have a platform for every release though (centos/rocky 3.9, jammy 3.10, bookworm 3.11, noble 3.12). If we end up without a platform for a version of python jobs may need to be updated to install python from source or some other method.
acknowledged
I know there could be tradeoffs between test coverage and infra resources, but IMO this is the test coverage we should fill.
Yes, the main concern was simply to avoid unnecessary effort. In the case of unittest jobs the vast majority of those are much cheaper in terms of resources than tempest jobs. I think if we stuck to this primarily for unittests the impact would be minimal. If we tried to expand the tempest matrix that would become unwieldy.
Yes. I don't intend to run all tests in all versions but at least having unit test jobs, which are supposed to be cheap, may avoid 0 test coverage and would be enough now.
In case we find any real problems then we can expand the scope to other jobs.
Thank you,
-- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
On 2024-10-05 01:30:30 +0900 (+0900), Takashi Kajinami wrote: [...]
to distinguish Python < 3.10 from Python >= 3.10 [...]
This is a questionable choice of implementation, given some distros in the past have backported features to older releases or reverted removals they saw as regressions. When possible, it makes more sense to test for the existence/absence of the feature you want to use rather than blindly assuming the (C)Python version will serve as a faithful proxy. An up-side to this approach is that you don't need to worry as much about version boundaries, as long as you test versions both with and without the feature/behavior in question. -- Jeremy Stanley
On 10/5/24 02:16, Jeremy Stanley wrote:
On 2024-10-05 01:30:30 +0900 (+0900), Takashi Kajinami wrote: [...]
to distinguish Python < 3.10 from Python >= 3.10 [...]
This is a questionable choice of implementation, given some distros in the past have backported features to older releases or reverted removals they saw as regressions. When possible, it makes more sense to test for the existence/absence of the feature you want to use rather than blindly assuming the (C)Python version will serve as a faithful proxy. An up-side to this approach is that you don't need to worry as much about version boundaries, as long as you test versions both with and without the feature/behavior in question.
The super annoying thing with that change is that importlib.metadata.entry_point has been existing since python 3.8 but it changed the behavior in python 3.10 . So checking existance of modules/attributes didn't work and I avoided implementing hanky inspect check to analyze accepted arguments. An alternative solution may be relying on importlib_metadata for all versions because it works for Python 3.12, but I was unsure if requiring unnecessary 3rd party dependency to make the code older version would be a good choice.
participants (3)
-
Clark Boylan
-
Jeremy Stanley
-
Takashi Kajinami