On Fri, Oct 4, 2024, at 10:26 AM, Takashi Kajinami wrote:
On 10/5/24 01:47, Clark Boylan wrote:
On Fri, Oct 4, 2024, at 9:30 AM, Takashi Kajinami wrote:
Hello,
As you may know, current master runs unit tests with only Python 3.9 and Python 3.12. My understanding about this set up is that we test the minimum boundary and the maximum boundary and assume the change may work with all the versions between these two.
However in some of my recent work I had to introduce the switch in the middle of these two versions. For example https://review.opendev.org/c/openstack/neutron/+/931271 introduces the logic to distinguish Python < 3.10 from Python >= 3.10 , and we don't specifically test that boundary. I'm afraid that this may potentially cause uncaught problems.
One approach that wasn't discussed in the change is to continue to use the existing code (pkg_resources) until python3.12 then only change the behavior for 3.12. That approach would generally work if we are only testing the boundaries since we're flipping the behavior on the boundary.
For this case we need python 3.11 job to test the boundary. Also this leaves noisy deprecation warnings in < Python 3.11 .
The lower bound (currently 3.9) would continue to test the old behavior (pkg_resources) and the upper bound (python3.12) would test the new behavior (importlib). It would be covered as well as all of the other upper and lower bound testing we've done. Python3.10 and 3.11 would be "covered" by the 3.9 test ensuring that pkg_resources continues to work. When 3.9 is dropped 3.10 would take over this responsibility. Eventually we'll only have the importlib behavior and python3.12/3.13/etc would cover it.
So my question here is ... Can we test all supported python versions, not only min/max, at least in check jobs ? This would helps us catch any regressions caused by such implementation.
We can. The main limitation is that we may not have a test platform for every version of python depending on how the distro releases align with python releases. Currently I think we have a platform for every release though (centos/rocky 3.9, jammy 3.10, bookworm 3.11, noble 3.12). If we end up without a platform for a version of python jobs may need to be updated to install python from source or some other method.
acknowledged
I know there could be tradeoffs between test coverage and infra resources, but IMO this is the test coverage we should fill.
Yes, the main concern was simply to avoid unnecessary effort. In the case of unittest jobs the vast majority of those are much cheaper in terms of resources than tempest jobs. I think if we stuck to this primarily for unittests the impact would be minimal. If we tried to expand the tempest matrix that would become unwieldy.
Yes. I don't intend to run all tests in all versions but at least having unit test jobs, which are supposed to be cheap, may avoid 0 test coverage and would be enough now.
In case we find any real problems then we can expand the scope to other jobs.
Thank you,
-- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit