Hello,
I have just ran over the mailing listing post OpenStack 2024.2 testing runtime by Ganshyam Mann [1] on dropping support for python3.8 for the upcoming Openstack 2024.2 release. This post and some small contributions to some openstack components have lead me to write this post.
What I would like to suggest to modernize the Python-Stack used for OpenStack. Such a modernization includes:
- A current python version - A modern and up to date static code analysis - Introduce type annotations
Let me explain what I mean:
A current python version: The current supported Python versions are Python 3.8 to Python 3.11 [2]. As of writing, OpenStack Yoga was just moved to unmaintained. To install this release, e.g. on an Ubuntu Jammy operating system the packages are included in the Ubuntu-Packages. The default Python version on an Ubuntu 22.04 operating system is Python3.10. And looking at the support of the Python versions [3] Python 3.8 to 3.10 only receive security updates but the end is near for Python 3.8 to be out of support. The policy defined which Python version should be used for testing [4] will result that Python 3.8 is still used for the OpenStack 2024.1 release [5]. And this will also result in required support for Python3.8 in the upcoming releases of OpenStack. I my opinion the policy should be reworked and the OpenStack releases should be more aggressive on dropping support for older Python versions. Modern features of the language cannot be otherwise be used, therefor improvements in speed, security and general toolchain/ecosystem cannot be fully leveraged. By defining the lower bound of the Python Version we are limiting ourself to this version but we might also reduce the features we can use because of deprecations of the newer versions .
A modern and up to date static code analysis: Currently flake8 and pylint are used both of which are modern analysis tools. But their configuration is in my opinion not up to date. Lets have a look at the configuration of the neutron-lib [6]. As seen a lot of rules are disabled by the configuration and are needed at the current moment at the state of the project. In addition some rules need to be adjusted such as the line length (80 characters is a very aggressive limit and on most modern displays a line length of 100 to 120 can fit without any problems, personal opinion). The adjustment of the static code analysis besides opinionated rules will lead me to my third point to modernize the Python stack used
Introduce type annotations Python type annotations are well supported in the current Python versions and through the typing package additional types are added. The introduction of type annotations will improve the developer experience. This is also motivated by the Python foundation in pep-484 [7]. Because of the type annotation all modern IDEs can provide (more precise) type hints. The type hints can and in in my experience will reduce the introduction of bugs (accessing members or calling methods, no logical bugs) and will shorten the time to develop a new feature or fix a logical bug and do not run in runtime exceptions. The datatypes passed to a method are clear and we do not need to dig down a rabbithole to find out what is passed to a method. The name of a variable does not always tell the truth. As an example I want to look at a patch [8] for the neutron-vpnaas component. Christian Rohmann and I worked on this patch for the component together (this has also lead me to write this email) to improve the reconciling for VPNaaS. It took as along time to find out that a RouterInformation object is passed in to a method instead of a router but the variable was named router. We both agreed on that in the first place the naming of the variable was wrong but type annotations would have saved us a good amount of time. This does not mean that the naming is not important and that type annotations will save the world, the context which includes the name of variables is important and type annotations will add additional information to the context. And I have just touched the surface on what is possible with the introduction of type annotations, thinking of automatic document generation, test generation, even code generation,... As a reference to my point modernizing the static analysis this can then also include the introduction of mypy to check the types used. Of course such a change cannot be done in one big commit and needs time but I suggest to start adding type annotations and improve the overall code quality of the OpenStack services, starting with the oslo packages as they are used by mostly all OpenStack components.
Of course such changes cannot be done for such a huge project in one day. It took a long time to remove support for Python 2. So I do not expect such changes to happen immediately. But it would be nice to have a discussion about it and maybe create a road map and adjust some policy on supported Python versions and create new policies how to write Python code and what additional tools should be used. I can also understand if not everything is possible but the last two points need to be addressed in my opinion. These changes will lead overall to better code quality but most important the introduction to the big code base will be easier for new developers and reviewers will also have a better time.
Best regards
Niklas
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://docs.openstack.org/tempest/latest/supported_version.html [3] https://devguide.python.org/versions/ [4] https://governance.openstack.org/tc/resolutions/20181024-python-update-proce... [5] https://governance.openstack.org/tc/reference/runtimes/2024.1.html [6] https://github.com/openstack/neutron-lib/blob/master/.pylintrc [7] https://peps.python.org/pep-0484/#rationale-and-goals [8] https://review.opendev.org/c/openstack/neutron-vpnaas/+/875745