On Tue, Feb 13, 2024, at 7:04 AM, Mohammed Naser wrote:
Hi Niklas,
These are all awesome ideas, I support them and I think we should take that initiative.
Also, I think the Python versions don't matter as much these days, I feel that almost most deploying tools are exploring some form of containerization so co-installability and all that aren't that much of an issue.
This is only true if we decide that we no longer care for Linux distros to package our software. One thing to be careful of with a switch like this is the lack of security patching for old branch's dependencies if relying on pypi.
Thanks Mohammed *From:* Niklas Schwarz <niklas.schwarz@inovex.de> *Sent:* February 13, 2024 7:27 AM *To:* openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> *Subject:* [all][tc] Modernize Python Stack
You don't often get email from niklas.schwarz@inovex.de. Learn why this is important <https://aka.ms/LearnAboutSenderIdentification> Hello,
I have just ran over the mailing listing post OpenStack 2024.2 testing runtime by Ganshyam Mann [1] on dropping support for python3.8 for the upcoming Openstack 2024.2 release. This post and some small contributions to some openstack components have lead me to write this post.
What I would like to suggest to modernize the Python-Stack used for OpenStack. Such a modernization includes:
- A current python version - A modern and up to date static code analysis - Introduce type annotations
Let me explain what I mean:
A current python version: The current supported Python versions are Python 3.8 to Python 3.11 [2]. As of writing, OpenStack Yoga was just moved to unmaintained. To install this release, e.g. on an Ubuntu Jammy operating system the packages are included in the Ubuntu-Packages. The default Python version on an Ubuntu 22.04 operating system is Python3.10. And looking at the support of the Python versions [3] Python 3.8 to 3.10 only receive security updates but the end is near for Python 3.8 to be out of support. The policy defined which Python version should be used for testing [4] will result that Python 3.8 is still used for the OpenStack 2024.1 release [5]. And this will also result in required support for Python3.8 in the upcoming releases of OpenStack. I my opinion the policy should be reworked and the OpenStack releases should be more aggressive on dropping support for older Python versions. Modern features of the language cannot be otherwise be used, therefor improvements in speed, security and general toolchain/ecosystem cannot be fully leveraged. By defining the lower bound of the Python Version we are limiting ourself to this version but we might also reduce the features we can use because of deprecations of the newer versions .
I would have this discussion on the thread from gmann that you referenced in [1]. It seems highly unlikely openstack will be willing to revert promises made for prior releases and their platforms, but changes can be made for future releases which is what that thread aims to discuss.
A modern and up to date static code analysis: Currently flake8 and pylint are used both of which are modern analysis tools. But their configuration is in my opinion not up to date. Lets have a look at the configuration of the neutron-lib [6]. As seen a lot of rules are disabled by the configuration and are needed at the current moment at the state of the project.
A comment in that file indicates those rule overrides should be addressed. I doubt this is controversial and merely needs volunteers to make it happen.
In addition some rules need to be adjusted such as the line length (80 characters is a very aggressive limit and on most modern displays a line length of 100 to 120 can fit without any problems, personal opinion).
Note 100 to 120 fitting on a display isn't the criteria I would look at. Instead you need to consider what side by side diffs look like with their additional overhead. Locally with vimdiff it looks like I might need at least 6 extra characters per file version (this might vary by file length due to line numbers). Three way diffs are reasonably common which comes out to (120 + 6) * 3 = 378 vs (80 + 6) * 3 = 258. I can fit 254 characters in a full screen terminal on my current desktop monitor.
The adjustment of the static code analysis besides opinionated rules will lead me to my third point to modernize the Python stack used
Introduce type annotations Python type annotations are well supported in the current Python versions and through the typing package additional types are added. The introduction of type annotations will improve the developer experience. This is also motivated by the Python foundation in pep-484 [7]. Because of the type annotation all modern IDEs can provide (more precise) type hints. The type hints can and in in my experience will reduce the introduction of bugs (accessing members or calling methods, no logical bugs) and will shorten the time to develop a new feature or fix a logical bug and do not run in runtime exceptions. The datatypes passed to a method are clear and we do not need to dig down a rabbithole to find out what is passed to a method. The name of a variable does not always tell the truth. As an example I want to look at a patch [8] for the neutron-vpnaas component. Christian Rohmann and I worked on this patch for the component together (this has also lead me to write this email) to improve the reconciling for VPNaaS. It took as along time to find out that a RouterInformation object is passed in to a method instead of a router but the variable was named router. We both agreed on that in the first place the naming of the variable was wrong but type annotations would have saved us a good amount of time. This does not mean that the naming is not important and that type annotations will save the world, the context which includes the name of variables is important and type annotations will add additional information to the context. And I have just touched the surface on what is possible with the introduction of type annotations, thinking of automatic document generation, test generation, even code generation,... As a reference to my point modernizing the static analysis this can then also include the introduction of mypy to check the types used. Of course such a change cannot be done in one big commit and needs time but I suggest to start adding type annotations and improve the overall code quality of the OpenStack services, starting with the oslo packages as they are used by mostly all OpenStack components.
It might be helpful for a project or two to act as test cases for adding this information and running mypy jobs (or adding mypy to the existing linter jobs). It has been a while since I tried to convert a project, but at that time there were a lot of rough edges. I suspect it would be good to get a current read on how painful it is to retrofit an existing codebase (vs starting with annotations in new projects). There are also alternatives to mypy. I'm not sure what the relative advantages between the tools are, but it may be worth considering whether a tool other than mypy is more appropriate?
Of course such changes cannot be done for such a huge project in one day. It took a long time to remove support for Python 2. So I do not expect such changes to happen immediately. But it would be nice to have a discussion about it and maybe create a road map and adjust some policy on supported Python versions and create new policies how to write Python code and what additional tools should be used. I can also understand if not everything is possible but the last two points need to be addressed in my opinion. These changes will lead overall to better code quality but most important the introduction to the big code base will be easier for new developers and reviewers will also have a better time.
Best regards
Niklas
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://docs.openstack.org/tempest/latest/supported_version.html [3] https://devguide.python.org/versions/ [4] https://governance.openstack.org/tc/resolutions/20181024-python-update-proce... [5] https://governance.openstack.org/tc/reference/runtimes/2024.1.html [6] https://github.com/openstack/neutron-lib/blob/master/.pylintrc [7] https://peps.python.org/pep-0484/#rationale-and-goals [8] https://review.opendev.org/c/openstack/neutron-vpnaas/+/875745