Thanks Niklas for the ideas and especially for the details, replying inline on each ideas. ---- On Tue, 13 Feb 2024 04:27:16 -0800 Niklas Schwarz wrote ---
Hello, I have just ran over the mailing listing post OpenStack 2024.2 testing runtime by Ganshyam Mann [1] on droppingsupport for python3.8 for the upcoming Openstack 2024.2 release. This post and some small contributionsto some openstack components have lead me to write this post.
What I would like to suggest to modernize the Python-Stack used for OpenStack.Such a modernization includes: - A current python version- A modern and up to date static code analysis- Introduce type annotations Let me explain what I mean: A current python version:The current supported Python versions are Python 3.8 to Python 3.11 [2]. As of writing, OpenStack Yoga was just moved to unmaintained. To install this release, e.g. on an UbuntuJammy operating system the packages are included in the Ubuntu-Packages. The default Python versionon an Ubuntu 22.04 operating system is Python3.10. And looking at the support of the Python versions [3]Python 3.8 to 3.10 only receive security updates but the end is near for Python 3.8 to be out of support.The policy defined which Python version should be used for testing [4] will result that Python 3.8 is stillused for the OpenStack 2024.1 release [5]. And this will also result in required support for Python3.8 in theupcoming releases of OpenStack. I my opinion the policy should be reworked and the OpenStack releasesshould be more aggressive on dropping support for older Python versions. Modern features of the languagecannot be otherwise be used, therefor improvements in speed, security and general toolchain/ecosystem cannotbe fully leveraged.By defining the lower bound of the Python Version we are limiting ourself to this version but we might also reducethe features we can use because of deprecations of the newer versions .
There are both side of argument on this. When we started dropping/untested the python min version 3.8 in 2023.2 it did not go well and we end up adding it back[1]. IMO, keeping support of older python version does not harm anything as long as we keep moving towards the latest version. If keeping older python version restrict us to move to latest version then this is good signals to bump our min version but until then it is always good to keep compatibility with more versions. It does not cost much as we only make sure code compatibilty on those min version (running unit test job enough for that). If we find that any newer feature present in latest python version and we want to use but cannot because we have to support some older version which does not have that feature then it is good reason to bump the min version. Other reason can be python version EOL, not able to install/test due to any reason or so.
A modern and up to date static code analysis:Currently flake8 and pylint are used both of which are modern analysis tools. But their configuration is in my opinionnot up to date. Lets have a look at the configuration of the neutron-lib [6]. As seen a lot of rules are disabledby the configuration and are needed at the current moment at the state of the project. In addition some rulesneed to be adjusted such as the line length (80 characters is a very aggressive limit and on most moderndisplays a line length of 100 to 120 can fit without any problems, personal opinion).The adjustment of the static code analysis besides opinionated rules will lead me to my third point to modernizethe Python stack used
On line length, I cannot say which lenght is better but what matters is ocnsistency. If we mixup the code with old length and new length then code become more unreadable. I will say let's continue with what we have in existing code, changing that now does not give us much benefits. On other modern code style or checking all rules is all good, runtime does not restrict those. it is up to project that how many rules they want to skip or run. Though I agree with your idea of testing it consistently across OpenStack.
Introduce type annotationsPython type annotations are well supported in the current Python versions and through the typing packageadditional types are added. The introduction of type annotations will improve the developer experience. This isalso motivated by the Python foundation in pep-484 [7]. Because of the type annotation all modern IDEs canprovide (more precise) type hints. The type hints can and in in my experience will reduce the introduction ofbugs (accessing members or calling methods, no logical bugs) and will shorten the time to develop a newfeature or fix a logical bug and do not run in runtime exceptions. The datatypes passed to a method are clearand we do not need to dig down a rabbithole to find out what is passed to a method.The name of a variable does not always tell the truth. As an example I want to look at a patch [8] for theneutron-vpnaas component. Christian Rohmann and I worked on this patch for the component together(this has also lead me to write this email) to improve the reconciling for VPNaaS. It took as along time to find outthat a RouterInformation object is passed in to a method instead of a router but the variable was named router.We both agreed on that in the first place the naming of the variable was wrong but type annotations would havesaved us a good amount of time. This does not mean that the naming is not important and that type annotationswill save the world, the context which includes the name of variables is important and type annotations willadd additional information to the context.And I have just touched the surface on what is possible with the introduction of type annotations, thinking ofautomatic document generation, test generation, even code generation,... As a reference to my point modernizing the static analysis this can then also include the introduction of mypyto check the types used.Of course such a change cannot be done in one big commit and needs time but I suggest to start addingtype annotations and improve the overall code quality of the OpenStack services, starting with the oslo packagesas they are used by mostly all OpenStack components.
++, again no restricution on this improvement also and I do not think anyone will reject such changes but if we ask project team to do all these then there might be some bandwidth issue as almost all projects lacks nunber of contributors even some of the key community-wide work are not completed there.
Of course such changes cannot be done for such a huge project in one day. It took a long time to removesupport for Python 2. So I do not expect such changes to happen immediately. But it would be nice to havea discussion about it and maybe create a road map and adjust some policy on supported Python versionsand create new policies how to write Python code and what additional tools should be used.I can also understand if not everything is possible but the last two points need to be addressed in my opinion.These changes will lead overall to better code quality but most important the introduction to the big code basewill be easier for new developers and reviewers will also have a better time.
Best regards Niklas
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/3NXTCUILYLMNI47UBOVZLMLXCPKDSXOJ/[2] https://docs.openstack.org/tempest/latest/supported_version.html[3] https://devguide.python.org/versions/[4] https://governance.openstack.org/tc/resolutions/20181024-python-update-process.html[5] https://governance.openstack.org/tc/reference/runtimes/2024.1.html[6] https://github.com/openstack/neutron-lib/blob/master/.pylintrc[7] https://peps.python.org/pep-0484/#rationale-and-goals[8] https://review.opendev.org/c/openstack/neutron-vpnaas/+/875745
[1] https://review.opendev.org/c/openstack/governance/+/882165 -gmann