On Tue, Feb 13, 2024, at 7:04 AM, Mohammed Naser wrote:

Hi Niklas,

These are all awesome ideas, I support them and I think we should take that initiative.

Also, I think the Python versions don't matter as much these days, I feel that almost most deploying tools are exploring some form of containerization so co-installability and all that aren't that much of an issue.

This is only true if we decide that we no longer care for Linux distros to package our software. One thing to be careful of with a switch like this is the lack of security patching for old branch's dependencies if relying on pypi.

Thanks Mohammed *From:* Niklas Schwarz niklas.schwarz@inovex.de *Sent:* February 13, 2024 7:27 AM *To:* openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org *Subject:* [all][tc] Modernize Python Stack

You don't often get email from niklas.schwarz@inovex.de. Learn why this is important https://aka.ms/LearnAboutSenderIdentification Hello,

I have just ran over the mailing listing post OpenStack 2024.2 testing runtime by Ganshyam Mann [1] on dropping support for python3.8 for the upcoming Openstack 2024.2 release. This post and some small contributions to some openstack components have lead me to write this post.

What I would like to suggest to modernize the Python-Stack used for OpenStack. Such a modernization includes:

• A current python version
• A modern and up to date static code analysis
• Introduce type annotations

Let me explain what I mean:

A current python version: The current supported Python versions are Python 3.8 to Python 3.11 [2]. As of writing, OpenStack Yoga was just moved to unmaintained. To install this release, e.g. on an Ubuntu Jammy operating system the packages are included in the Ubuntu-Packages. The default Python version on an Ubuntu 22.04 operating system is Python3.10. And looking at the support of the Python versions [3] Python 3.8 to 3.10 only receive security updates but the end is near for Python 3.8 to be out of support. The policy defined which Python version should be used for testing [4] will result that Python 3.8 is still used for the OpenStack 2024.1 release [5]. And this will also result in required support for Python3.8 in the upcoming releases of OpenStack. I my opinion the policy should be reworked and the OpenStack releases should be more aggressive on dropping support for older Python versions. Modern features of the language cannot be otherwise be used, therefor improvements in speed, security and general toolchain/ecosystem cannot be fully leveraged. By defining the lower bound of the Python Version we are limiting ourself to this version but we might also reduce the features we can use because of deprecations of the newer versions .

I would have this discussion on the thread from gmann that you referenced in [1]. It seems highly unlikely openstack will be willing to revert promises made for prior releases and their platforms, but changes can be made for future releases which is what that thread aims to discuss.

A modern and up to date static code analysis: Currently flake8 and pylint are used both of which are modern analysis tools. But their configuration is in my opinion not up to date. Lets have a look at the configuration of the neutron-lib [6]. As seen a lot of rules are disabled by the configuration and are needed at the current moment at the state of the project.

A comment in that file indicates those rule overrides should be addressed. I doubt this is controversial and merely needs volunteers to make it happen.

In addition some rules need to be adjusted such as the line length (80 characters is a very aggressive limit and on most modern displays a line length of 100 to 120 can fit without any problems, personal opinion).

Note 100 to 120 fitting on a display isn't the criteria I would look at. Instead you need to consider what side by side diffs look like with their additional overhead. Locally with vimdiff it looks like I might need at least 6 extra characters per file version (this might vary by file length due to line numbers). Three way diffs are reasonably common which comes out to (120 + 6) * 3 = 378 vs (80 + 6) * 3 = 258. I can fit 254 characters in a full screen terminal on my current desktop monitor.

The adjustment of the static code analysis besides opinionated rules will lead me to my third point to modernize the Python stack used

Introduce type annotations Python type annotations are well supported in the current Python versions and through the typing package additional types are added. The introduction of type annotations will improve the developer experience. This is also motivated by the Python foundation in pep-484 [7]. Because of the type annotation all modern IDEs can provide (more precise) type hints. The type hints can and in in my experience will reduce the introduction of bugs (accessing members or calling methods, no logical bugs) and will shorten the time to develop a new feature or fix a logical bug and do not run in runtime exceptions. The datatypes passed to a method are clear and we do not need to dig down a rabbithole to find out what is passed to a method. The name of a variable does not always tell the truth. As an example I want to look at a patch [8] for the neutron-vpnaas component. Christian Rohmann and I worked on this patch for the component together (this has also lead me to write this email) to improve the reconciling for VPNaaS. It took as along time to find out that a RouterInformation object is passed in to a method instead of a router but the variable was named router. We both agreed on that in the first place the naming of the variable was wrong but type annotations would have saved us a good amount of time. This does not mean that the naming is not important and that type annotations will save the world, the context which includes the name of variables is important and type annotations will add additional information to the context. And I have just touched the surface on what is possible with the introduction of type annotations, thinking of automatic document generation, test generation, even code generation,... As a reference to my point modernizing the static analysis this can then also include the introduction of mypy to check the types used. Of course such a change cannot be done in one big commit and needs time but I suggest to start adding type annotations and improve the overall code quality of the OpenStack services, starting with the oslo packages as they are used by mostly all OpenStack components.

It might be helpful for a project or two to act as test cases for adding this information and running mypy jobs (or adding mypy to the existing linter jobs). It has been a while since I tried to convert a project, but at that time there were a lot of rough edges. I suspect it would be good to get a current read on how painful it is to retrofit an existing codebase (vs starting with annotations in new projects).

There are also alternatives to mypy. I'm not sure what the relative advantages between the tools are, but it may be worth considering whether a tool other than mypy is more appropriate?

Of course such changes cannot be done for such a huge project in one day. It took a long time to remove support for Python 2. So I do not expect such changes to happen immediately. But it would be nice to have a discussion about it and maybe create a road map and adjust some policy on supported Python versions and create new policies how to write Python code and what additional tools should be used. I can also understand if not everything is possible but the last two points need to be addressed in my opinion. These changes will lead overall to better code quality but most important the introduction to the big code base will be easier for new developers and reviewers will also have a better time.

Best regards

Niklas

[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://docs.openstack.org/tempest/latest/supported_version.html [3] https://devguide.python.org/versions/ [4] https://governance.openstack.org/tc/resolutions/20181024-python-update-proce... [5] https://governance.openstack.org/tc/reference/runtimes/2024.1.html [6] https://github.com/openstack/neutron-lib/blob/master/.pylintrc [7] https://peps.python.org/pep-0484/#rationale-and-goals [8] https://review.opendev.org/c/openstack/neutron-vpnaas/+/875745

+1 here.

Python versions shipped by systems still matter, imo. Some python bindings, like python-libvirt, python-ceph, python-selinux are not provided by pip, so they must be built against python version needed manually, which is really not always a straightforward process.

Despite pyenv looks very appealing at first glance, it brings quite some challenges for deployments at the end.

And even if distro does provide higher version of python itself during lifetime, they do never provide all required bindings to it, which makes it very tough to use (or well, same as with pyenv basically).

On Wed, Feb 14, 2024, 10:14 Thomas Goirand zigo@debian.org wrote:

On 2/13/24 16:04, Mohammed Naser wrote:

...

Hi Niklas,

These are all awesome ideas, I support them and I think we should take that initiative.

Also, I think the Python versions don't matter as much these days, I feel that almost most deploying tools are exploring some form of containerization so co-installability and all that aren't that much of an issue.

I strongly do not agree with the above sentence. Co-installability is still a thing, it's not because *you* are using containers that everyone loves it. We're also moving toward using Kube for some part of the control plane, but we'll be continuing to use packages in the pods, and support bare-metal setup in the distro.

Cheers,

Thomas Goirand (zigo)

Le mer. 14 févr. 2024 à 11:26, Christian Rohmann < christian.rohmann@inovex.de> a écrit :

Thanks for starting a discussion about the (Python) stack as a whole and not just making this about using Python 3.9 vs. 3.12....

OpenStack is a well established cloud software stack and just like the Python interpreter and whole ecosystem is relies on, it should embrace new capabilities and not just move if there is an unavoidable breaking change.

Subjectively there has been quite some developer drain or let's say the amount of patches by new developers seems lower than years ago. Quite a few projects died with not replacement. The community has been very capable if an external change (Death of Python 2.7, SQLAlchemy 2.x, eventlet EoL, ....) forced some reaction or adaption. But does it really require some external force for some kind of modernization? Why is OpenStack is not pushing ahead in and out of itself? For the sake of one's own quality standards, but also to ensure the whole ecosystem remains vital and and attractive to new new developers?

On 13.02.24 13:27, Niklas Schwarz wrote:

...

A modern and up to date static code analysis: Currently flake8 and pylint are used both of which are modern analysis tools. But their configuration is in my opinion not up to date. Lets have a look at the configuration of the neutron-lib [6]. As seen a lot of rules are disabled by the configuration and are needed at the current moment at the state of the project. In addition some rules need to be adjusted such as the line length (80 characters is a very aggressive limit and on most modern displays a line length of 100 to 120 can fit without any problems, personal opinion). The adjustment of the static code analysis besides opinionated rules will lead me to my third point to modernize the Python stack used Introduce type annotations [...] The type hints can and in in my experience will reduce the introduction of bugs (accessing members or calling methods, no logical bugs) and will shorten the time to develop a new feature or fix a logical bug and do not run in runtime exceptions.

I can only second Niklas' reference to the PEP 484 ([2]) and its rationale:

...

This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and (perhaps, in some contexts) code generation utilizing type information. Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring.

an this is from 2014, so about 10 years ago.

While unit and functional tests allow people to move faster in writing their code, be it a bugfix or feature as they quickly see if their code breaks stuff. But having static code analysis as a first line of defense is a must. Continuing to allow more un-typed variables, references and interfaces limits the good those kind of tools can do.

If I may quote an reply to an old post to the ML ( [1]) about the introduction of "black" formatter:

On 18.4.19 19:04, Clint Byrum wrote: :

...

The main value is that using an automatic formatter will ultimately lead to less wasted test runs, less wasted developer time, and better diffs with less conflicts later.

So while I appreciate the desire to not rock the boat, it might be that the boat would go faster after some dead weight is dropped overboard, and that new energy is applied to forward progress instead of "darn it, now I have to think about how to fit this into 80 chars and parenthesis and ..."

Basically, weigh the one time cost vs. the ongoing savings, rather than considering them separately.

Agreeing on a set of rules then set in stone (or rather the config of formatter, linters and static code analysis tools) increases the quality of the code, makes it more approachable to new devs and allows reviewers to focus more on the logic and functional part of submitted code. And what's the downside of moving into this direction really?

• A big initial reformatting commit that makes it harder to then

backport changes to older releases?

• Introducing types to older code and identifying issues this way?
• Static code analysis throwing lots of warnings?

FWIW, those three reasons are the main reasons why at least every cycle during the PTG, I'm very conservative for new formatting features in the Nova project. That's it for me, we already discussed this in the nova project during the last Caracal PTG and we had no consensus.

Regards

Christian

[1]

https://lists.openstack.org/pipermail/openstack-discuss/2019-April/005364.ht... [2] https://peps.python.org/pep-0484/#rationale-and-goals

The community has been very capable if an external change (Death of Python 2.7, SQLAlchemy 2.x, eventlet EoL, ....) forced some reaction or adaption. But does it really require some external force for some kind of modernization?

I'm isolating this for a specific reason: many of the items listed here were driven by the same major contributors (human and corporate) to the community, and/or are in progress. For instance, the eventlet migration still does not have a finalized plan[0] and the primary drivers right now are developers from RedHat as well as a single GR-OSS developer we contributed to help improve eventlet in the short term. When describing this to my boss; I told him I'm incredibly proud that GR-OSS is participating in this work, because so few contributing companies are willing to do this invisible work -- but the sad thing is, eventlet eol is one of a large number of things that needs doing -- and there aren't many people to do it.

This email thread highlights many of those other issues which need addressing; but are missing a significant portion: a call to action and real work to make them happen.

There are several ways you help change this: 1) Don't like how the project is being run from a technical perspective? Run for the TC -- there certainly is a need for volunteers. Try to make change happen. Governance is not easy and there are hundreds of people with different opinions about what direction "forward" is -- and it's a lot more work than just adding and removing python versions from platform documents. 2) Do a proof of concept in a smaller project. I'd personally be willing to help you gain consensus on implementing (for instance) some type hints on Ironic. I can't promise the whole Ironic team would agree; but the point is the easiest way to influence change in OpenStack is to show willingness to do the work. For SQLA 2.0 migration, for instance, it was obvious there were some members of the community who worked towards it, with urgency, even pulling forward projects that are now inactive to function with SQLA 2.0. 3) Advocate inside your organization for more upstream contribution; either in the form of your time or additional upstream contributors. A recent paper[1] out of Harvard Business School says 96% of the demand-side value is created by only 5% of OSS developers. Everyone who has a leadership position at a company that uses or contributes to OSS has an obligation to advocate to help close these gaps.

I'm not going to break down the original email point by point, but generally: yes, there are things that OpenStack are behind on -- but emails to the mailing list and consensus aren't the solution. The solution is more people contributing to solve these problems -- especially in traditionally under-resourced teams such as QA, oslo libraries, and smaller projects on the verge of inactivity. Jeremy said (paraphrased) that we are OpenStack; rather than asking why OpenStack is not pushing forward, we should be asking ourselves why we aren't pushing forward. I agree with him. I also agree with the OP which asserts that we have clear technical improvement opportunities. What's stopping you from pushing them forward?

Thanks, Jay Faulkner OpenStack TC Chair & Ironic PTL Open Source Developer, G-Research OSS

0: https://review.opendev.org/c/openstack/governance/+/902585 1: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4693148

On Wed, Feb 14, 2024 at 2:21 AM Christian Rohmann < christian.rohmann@inovex.de> wrote:

Thanks for starting a discussion about the (Python) stack as a whole and not just making this about using Python 3.9 vs. 3.12....

OpenStack is a well established cloud software stack and just like the Python interpreter and whole ecosystem is relies on, it should embrace new capabilities and not just move if there is an unavoidable breaking change.

Subjectively there has been quite some developer drain or let's say the amount of patches by new developers seems lower than years ago. Quite a few projects died with not replacement. The community has been very capable if an external change (Death of Python 2.7, SQLAlchemy 2.x, eventlet EoL, ....) forced some reaction or adaption. But does it really require some external force for some kind of modernization? Why is OpenStack is not pushing ahead in and out of itself? For the sake of one's own quality standards, but also to ensure the whole ecosystem remains vital and and attractive to new new developers?

On 13.02.24 13:27, Niklas Schwarz wrote:

...

A modern and up to date static code analysis: Currently flake8 and pylint are used both of which are modern analysis tools. But their configuration is in my opinion not up to date. Lets have a look at the configuration of the neutron-lib [6]. As seen a lot of rules are disabled by the configuration and are needed at the current moment at the state of the project. In addition some rules need to be adjusted such as the line length (80 characters is a very aggressive limit and on most modern displays a line length of 100 to 120 can fit without any problems, personal opinion). The adjustment of the static code analysis besides opinionated rules will lead me to my third point to modernize the Python stack used Introduce type annotations [...] The type hints can and in in my experience will reduce the introduction of bugs (accessing members or calling methods, no logical bugs) and will shorten the time to develop a new feature or fix a logical bug and do not run in runtime exceptions.

I can only second Niklas' reference to the PEP 484 ([2]) and its rationale:

...

This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and (perhaps, in some contexts) code generation utilizing type information. Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring.

an this is from 2014, so about 10 years ago.

While unit and functional tests allow people to move faster in writing their code, be it a bugfix or feature as they quickly see if their code breaks stuff. But having static code analysis as a first line of defense is a must. Continuing to allow more un-typed variables, references and interfaces limits the good those kind of tools can do.

If I may quote an reply to an old post to the ML ( [1]) about the introduction of "black" formatter:

On 18.4.19 19:04, Clint Byrum wrote: :

...

The main value is that using an automatic formatter will ultimately lead to less wasted test runs, less wasted developer time, and better diffs with less conflicts later.

So while I appreciate the desire to not rock the boat, it might be that the boat would go faster after some dead weight is dropped overboard, and that new energy is applied to forward progress instead of "darn it, now I have to think about how to fit this into 80 chars and parenthesis and ..."

Basically, weigh the one time cost vs. the ongoing savings, rather than considering them separately.

Agreeing on a set of rules then set in stone (or rather the config of formatter, linters and static code analysis tools) increases the quality of the code, makes it more approachable to new devs and allows reviewers to focus more on the logic and functional part of submitted code. And what's the downside of moving into this direction really?

• A big initial reformatting commit that makes it harder to then

backport changes to older releases?

• Introducing types to older code and identifying issues this way?
• Static code analysis throwing lots of warnings?

Regards

Christian

[1]

https://lists.openstack.org/pipermail/openstack-discuss/2019-April/005364.ht... [2] https://peps.python.org/pep-0484/#rationale-and-goals

Hey Christian. Some comments inline.

On Tue, Feb 20, 2024 at 12:41 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:

Hello Jay, Jeremy, all,

thanks for your responses and the time to write up your thoughts.

On 14.02.24 14:53, Jeremy Stanley wrote:

Don't lose sight of the fact that this is an inherently existential question. OpenStack is all of us. You are OpenStack. I am OpenStack. Why are you not pushing ahead? Why am I not pushing ahead? This is the real question you're asking.

That is exactly on point to why a discussion as a starting point can be important to "push ahead". While I might be of some opinion that things are not as they could or should be, it might not be the consensus of the greater community, other important and hard-working members of this community. Does it really help if everybody forms their ideas right into a patch blackening all Python code of a project he once wrote a little patch for? Or does it help the generals idea if O start sprinkling type annotations all over any code that is OpenStack? Who would review or even accept that? Writing up a quick proposal requesting the use of type annotations and pylint for all OpenStack code from now on might sound nice, but how would that come across? I know we are using Python, which asks for forgiveness instead of permission, but really? What we need for any change, apart from any policy or format agreement is foundation is the intention to change, an in a certain direction.

A discussion to me is a good way to see if your idea is onto something and if others might even share it (seems to be the case in the matter of modernizing Python). It's also a good way to find others that might work together on the next step: A formal proposal if e.g. someone from TC even encourages to do that :-) But even if that proposal was then not even written by OP or other participants of the discussion - a heated discussion or the agreement by some established OpenStack contributors, might even be a good indication to TC that something is an important topic which could need some formal backing or follow up in one form or another. What else would be a good source for TCs agenda than the combined voices of the community?

Many of the technical leaders in the project have been significantly burned in times past from big ideas, in the form of goals, being backed with little/no technical effort to implement them. Even in the cases where those goals were given prioritized attention, there were always smaller projects that struggled to make the necessary lift. This to me has changed the nature of how this work is done.

There are a lot of ideas. The best idea is the one that actually has a chance of being implemented. I think an important part of technical leadership in a large project is ensuring goals set forward are achievable. At this point, I have extremely low amounts of trust that ideas proposed by people who aren't deeply involved in building OpenStack will get enough help to be implemented across all of OpenStack. I point at the RBAC project as a great example -- an amazing feature, something that's a major addition to any cloud it's put in, but there are still projects which don't have full support for it.

This is the heart of what I was trying to get at: the major constraint to the OpenStack agenda is having enough people to implement it. There's only so many folks doing active technical work on OpenStack, and fewer still who focus primarily on OpenStack-wide issues.

On 14.02.24 17:21, Jay Faulkner wrote:

...

The community has been very capable if an external change (Death of Python 2.7, SQLAlchemy 2.x, eventlet EoL, ....) forced some reaction or adaption. But does it really require some external force for some kind of modernization?

I'm isolating this for a specific reason: many of the items listed here were driven by the same major contributors (human and corporate) to the community, and/or are in progress. For instance, the eventlet migration still does not have a finalized plan[0] and the primary drivers right now are developers from RedHat as well as a single GR-OSS developer we contributed to help improve eventlet in the short term. When describing this to my boss; I told him I'm incredibly proud that GR-OSS is participating in this work, because so few contributing companies are willing to do this invisible work -- but the sad thing is, eventlet eol is one of a large number of things that needs doing -- and there aren't many people to do it.

I understand your frustration regarding the work on eventlet and yes, these kind of components and libraries are mostly taken for granted. This was seen quite a few times in recent years when e.g. the log4j issue happened.

I simply picked a few examples, where OpenStack, and I am referring to the community of projects and people, to my impression ran into an issue or a deprecation and there was lots of energy to make the required changes happen across many projects quickly to come out better on the other end, technically and socially. But more importantly it was the challenge and need for action that brought people together in accepting change. While you personally might rightly complain that the urgently needed work on eventlet was only done by a few, I also see the consensus for the necessary or even overdue change as another outcome.

To maybe try another example, Python 3.12, e.g. https://review.opendev.org/q/topic:%22python-3.12%22 While frustrating to see a simple version bump of Python causing lots of issues and CI fails. Put in the right perspective, any of those changes is part of that push forward and that keep the codebase alive.

Yep; that's the exact thing I'm talking about re: the python 3.12

movement. We do not support python 3.12, but despite that, Zigo has been driving work to help deal with python compatibility issues before those python versions are even officially supported. A great example of doing something rather than getting permission :). Things like this don't often make a lot of noise, but they are impactful and lower the gaps we have to cover when those versions come into our supported platform.

This email thread highlights many of those other issues which need

addressing; but are missing a significant portion: a call to action and real work to make them happen.

There are several ways you help change this:

1. Don't like how the project is being run from a technical perspective?

Run for the TC -- there certainly is a need for volunteers. Try to make change happen. Governance is not easy and there are hundreds of people with different opinions about what direction "forward" is -- and it's a lot more work than just adding and removing python versions from platform documents.

Not to repeat what I said above. But there might be hundreds of opinions voiced here and otherwise. What's important is a good mannered discussion leading to some kind of distillation of opinions into somewhat of an agreement, a common denominator. And isn't the open and public discussion actually helpful for TC identify an issue that is worth looking into? Why should one start with a nicely written up proposal holding all the answers, instead of starting a discussion around a topic first and then see if others share the impressions that the Python stack has some issue that really should be tackled?

The problem is the ratio of conversations/discussion to action. There's a large amount of people willing to help set an agenda; a much more limited set of people willing to engage it. That's why some of these comments have a tone of frustration-- the number of times this sort of thread shows up is much, much higher than the follow-through for ideas brought up in these threads. It's also why I am trying to be a pragmatist: what specific things can we do with the resources we have and/or that the community is willing to bring to bear.

2) Do a proof of concept in a smaller project. I'd personally be willing to

help you gain consensus on implementing (for instance) some type hints on Ironic. I can't promise the whole Ironic team would agree; but the point is the easiest way to influence change in OpenStack is to show willingness to do the work. For SQLA 2.0 migration, for instance, it was obvious there were some members of the community who worked towards it, with urgency, even pulling forward projects that are now inactive to function with SQLA 2.0.

Don't get me wrong, but I feel there is a little more of general frustration than actual reasoning here. Discrediting the (unknown) amount or value of contributed work by someone else, likely is not going to motivate them to help. Reading this, I was just about to vent on my own frustration of trying to get even some little patch reviewed and merged and the need drive even small changes home over multiple releases. This contributes, just like the technical state of the Python stack, to developer experience, their willingness to contribute and stay with the project. And I'd like to generalize this as much as I can, as it's quite universal for my experience with different OpenStack projects and not related to any person or team in particular. But having contributions (and also bug reports) that are not responded to in any way will scare people away from contributing and likely for good.

I gladly write up more detailed thoughts and experiences if someone would be using them to reflect on how things are organized.

This is actually a topic I, personally, am relatively passionate about, even if the direction of this conversation doesn't exactly reflect it :). Lowering the effort new developers need to get OpenStack contributions done would be a very valuable improvement. However, the kinds of changes that are needed require more than drive-by levels of contribution, so in a way I see this as a little tangential to the problem at hand.

3) Advocate inside your organization for more upstream contribution; either

in the form of your time or additional upstream contributors. A recent paper[1] out of Harvard Business School says 96% of the demand-side value is created by only 5% of OSS developers. Everyone who has a leadership position at a company that uses or contributes to OSS has an obligation to advocate to help close these gaps.

While I agree with this point in general, again, I don't think it's a good response to the topic of this thread. There are many issues, big and small, that contribute to the fact that OpenStack receives less contributions over time and most of them are by a small group of people, most of them on the payroll of the same group of companies. While I agree that institutions and companies should do their part, also asking why OpenStack and it's projects are not really attractive to contribute to as a sole developer from some smaller installation might also be helpful to discuss as it ensures the community remains healthy and broad. Yet another good topic for governance. Not to answer, but to bring to the agenda and actively address.

This is an honest question: do you think these problems are meaningfully

solvable by drive-by contributors? In the past, OpenStack-wide goals have taken years of committed work to get released. OpenStack has *hundreds* of repos of python software, even a small change across all of them is time consuming, even if there's no real technical difficulty and consensus is already reached.

I think the key problem is a perception that every problem is solvable by the TC, the Foundation Board, or project governance in general. We each have an individual and professional responsibility, as members of the OpenStack community, to help ensure things are properly resourced. I've done this before I was in the TC, while I've been on the TC, and once I am no longer on the TC -- it's not a governance function, it's an important form of community contribution.

I'm not going to break down the original email point by point, but generally: yes, there are things that OpenStack are behind on -- but emails to the mailing list and consensus aren't the solution. The solution is more people contributing to solve these problems -- especially in traditionally under-resourced teams such as QA, oslo libraries, and smaller projects on the verge of inactivity. Jeremy said (paraphrased) that we are OpenStack; rather than asking why OpenStack is not pushing forward, we should be asking ourselves why we aren't pushing forward. I agree with him. I also agree with the OP which asserts that we have clear technical improvement opportunities. What's stopping you from pushing them forward?

What made you assume the OP or myself are not actively pushing specs, code and ideas (like this one here)? Did you consider the urge to post to the ML and participate in a discussion on modernizing the Python stack could potentially be out of the experience doing so? Even if I hadn't done any such work, would the arguments about the Python stack be less valid?

If I may bluntly and selectively quote https://review.opendev.org/c/openstack/governance/+/902585/16/goals/proposed... ....

A community goal does not shape a new and personal vision of Openstack. A Community goal collects this vision from the scattered hopes and intentions of our community's past. [...] The TC is elected to provides technical leadership. [...] Without action we will soon face a brutal discontinuation.

With that, let me sum up my thoughts a little ...

While some tech debt in the Python stack is not an immediate existential crisis, it's vital to keep moving and to stay on top of things. Adopting new developments in the ecosystem is for the sake of quality (security, bugs, speed, efficiency, ...) and to avoid running into short term firefighting mode (deprecation, technical dead-ends, ...). But most importantly, if I read your responses, Jay and Jeremy, it's about lowering the barrier for "outside" contributions again. Code has to be approachable (e.g. is well structured, has type hinting, current syntax styles, ...) by part-time contributors to even have them try contributing and push a patch. The mentioned well established tooling such a formatters, linters or static code analysis and others should be used to assist senior and full-time devs with the QA and review processes to ultimately enable more contributions and by more individual contributors to be merged.

I agree with all of those ideas. I do not think that they are an endpoint to be targeted in whole. A smaller, achievable scope with clearly defined borders will be easier to scale across all of OpenStack and has a better chance of moving the project forward. The eventlet migration goal is a *great* example of this -- we broke off a large piece of an even larger problem (python 3.12 support) and are then able to take steps to resolve it.

I would advise if you want this to be a successful endeavor to pick one specific item to champion, show a proof of concept in a smaller OpenStack project, and move it forward. Giving a scope and a business case for why fixing a thing is important -- it's how you end up not doing this work alone. I'd define consensus in OpenStack as requiring not only general agreement on what should happen; but agreement to help make it happen. That's the only form of consensus I've seen that has an endpoint of real change and improvement.

Thanks again for your time,

Just noting -- I suspect you and I have many of the same values, but are just seeing things from different perspectives and on different time horizons. I strongly encourage you to run for a leadership position during this election cycle if governance and consensus building is something you care about.

Thanks, Jay Faulkner OpenStack TC Chair

On 03.04.24 7:55 PM, Jeremy Stanley wrote:

On 2024-04-03 19:06:25 +0200 (+0200), Sven Kieske wrote: [...]

...

bad coding practices like inscrutable variable names, no Types being used

[...]

It's hard to say lack of typing is a "bad coding practice" for code that was written (as much as a decade) before Python standardized on any sort of type annotation scheme. Not getting into judgement on whether typing is a real improvement in the language, but don't fault an antique radio for using tubes instead of transistors.

There is a line between not judging something for what it is, some years old, written with the knowledge of the time, and not allowing for new developments and improvements to be leveraged because one holds the "we have always done it this way"-paradigm too high.

And "modernizing" is the first word of the subject line, not "judging" or "discrediting". This is a discussion about which more recent developments in the Python ecosystem might make sense adopting, which conventions could be put in place. As in, "should we maybe switch to transistors instead of tubes to make radios cheaper, sound better, use less power". Nobody in the discussion so far seems to advocate burning down ratio stations, dismantling towers or to kill the presenters.

I have to agree with Sven that for (part time) contributors it's quite a learning curve diving into some of the code base which holdslots of traps to falls into, also due to the lack of types and sometime also easy to grasp naming. Also the syntax and general coding style has some influence. There are reasons other languages (e.g. Golang, Rust) have started with strict formatting and coding styles for the get go. It simply makes everything look similar. Lastly the compilers or interpreters don't care for these aspects. Common coding styles and tools are there to help us, the developers, to work more efficient, simply get more done without wasting time on getting confused within the code base, introduce accidental (type related) bugs or bluntly apply different code formatting rules, depending on which thing you are working on.

Regards

Christian

On 04.04.24 3:50 PM, Jeremy Stanley wrote:

("bad" by whose definition? how do you measure that empirically?) make for a weak argument. The points you want to make are best supported with evidence, not anecdote.

Let's start there and try to actually discuss the current good standards or practices that we either actively follow or should adopt. Provided you even agree that any change of the existing "way of doing things" could be remotely beneficial, please kindly share your thoughts and ideas of "good" coding practices then.

I'd personally determine the definitive lowest end of the quality scale is code simply runs and functionally does what it should do. Everything else is a gradient of "quality", be it  human or technical quality.

To give one or three examples which all already came up in this thread ...

1) Having the convention to break lines after 80 characters is a for-humans quality of the code. The interpreter simply does not care. But supposedly there are (were) reasons for this rule / convention to be in place? Since there were voices calling for this to be changed, are there metrics to determine how useful this still is?

2) I'd argue that PEP8 ([1]), e.g. via Flake8 or Ruff, which used for many projects' code is yet another (externalized) convention of coding practices. But there are different levels of adoption and exceptions lowering the advantages of unified coding styles. OpenStackSDK did switch to automatically formatting the code with black. While an initial big commit, there are no more variations introduced by different people contributing their code - the machine takes care of that. So does it make sense to adopt this for other projects and align "OpenStack" a bit more?

3) On the argument of adding type hints I'd say it's like memory safety which is arguably only advantageous if you make mistakes. Having types just helps to avoid a whole class of bugs (computer shouting at dev if I may quote Sven here) and also makes the code much more approachable. Also it allow to become quicker when refactoring things in a larger code base to now accidentally introduce bugs. Let me add two examples of such bugs that were found this way in [3] and [4] (both I quickly found searching for "mypy" and "bug" in Gerrit).

The status quo wins by default, always, and it is incumbent on those proposing change to make a compelling case for any needed investment on the part of others

Not to sound melodramatic, but I believe there is some amount of cry out to make contributing to the OpenStack code base more fun to contribute to again. Also please note that this cry out (partly) comes from folks who already contribute and would just like for it to be more fun and efficient. And in the end, there is the "investment" yes, but they might also be a benefit or return - technically, as in less runtime bugs and easier reviews. But also socially, via more contributions / contributors who don't have to keep their code compatible with many older Python versions and have turn off all their code quality assuring tools.

Regards

Christian

[1] https://peps.python.org/pep-0008 [2] https://github.com/openstack/openstacksdk/commit/c946294bddd0ad37b707c7f993c... [3] https://review.opendev.org/c/openstack/cinder/+/909689 [4] https://review.opendev.org/c/openstack/openstacksdk/+/900710

Please stop gatekeeping peoples' efforts because you don't agree with them.

I personally do not see this as gatekeeping or any other sort of stopping. Discussion clearly is getting opinionated, but I do not read any message as "stop, we are not interested". Instead people start discussing their interpretations of what others say - this is not productive.

What is stopping all of us is amount of companies interested in paying developers to start improving things (real maintenance) and not just adding features required by them (and this amount is now more or less 0). I see a very similar issue in the OpenAPI stuff - lot of people are interested, but no company is keen to dedicate people to implement that and so I do it in my private time.

I think what is required to have a progress here is to have people that "just start doing that". Pure discussion will not lead to anything except heated nerves or even everybody agreeing but nobody having time to do anything. You want "typing" - start adding it, I am pretty confident nobody will stop you and reject such changes. But do not expect somebody else will start doing it just because you think it would be useful. Every company is having their business and their interests which are about earning money and not about maintenance. This is sadly what OpenStack is now about, but understandable since nobody can just burn money (and I know it may sound abusive to some).

Cheers,

Artem

---

*From:* Christian Rohmann christian.rohmann@inovex.de *Sent:* April 4, 2024 11:31 AM *To:* Jeremy Stanley fungi@yuggoth.org; openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org *Subject:* Re: [all][tc] Modernize Python Stack On 04.04.24 3:50 PM, Jeremy Stanley wrote:

...

("bad" by whose definition? how do you measure that empirically?) make for a weak argument. The points you want to make are best supported with evidence, not anecdote.

Let's start there and try to actually discuss the current good standards or practices that we either actively follow or should adopt. Provided you even agree that any change of the existing "way of doing things" could be remotely beneficial, please kindly share your thoughts and ideas of "good" coding practices then.

I'd personally determine the definitive lowest end of the quality scale is code simply runs and functionally does what it should do. Everything else is a gradient of "quality", be it  human or technical quality.

To give one or three examples which all already came up in this thread ...

1. Having the convention to break lines after 80 characters is a

for-humans quality of the code. The interpreter simply does not care. But supposedly there are (were) reasons for this rule / convention to be in place? Since there were voices calling for this to be changed, are there metrics to determine how useful this still is?

2. I'd argue that PEP8 ([1]), e.g. via Flake8 or Ruff, which used for

many projects' code is yet another (externalized) convention of coding practices. But there are different levels of adoption and exceptions lowering the advantages of unified coding styles. OpenStackSDK did switch to automatically formatting the code with black. While an initial big commit, there are no more variations introduced by different people contributing their code - the machine takes care of that. So does it make sense to adopt this for other projects and align "OpenStack" a bit more?

3. On the argument of adding type hints I'd say it's like memory

safety which is arguably only advantageous if you make mistakes. Having types just helps to avoid a whole class of bugs (computer shouting at dev if I may quote Sven here) and also makes the code much more approachable. Also it allow to become quicker when refactoring things in a larger code base to now accidentally introduce bugs. Let me add two examples of such bugs that were found this way in [3] and [4] (both I quickly found searching for "mypy" and "bug" in Gerrit).

...

The status quo wins by default, always, and it is incumbent on those proposing change to make a compelling case for any needed investment on the part of others

Not to sound melodramatic, but I believe there is some amount of cry out to make contributing to the OpenStack code base more fun to contribute to again. Also please note that this cry out (partly) comes from folks who already contribute and would just like for it to be more fun and efficient. And in the end, there is the "investment" yes, but they might also be a benefit or return - technically, as in less runtime bugs and easier reviews. But also socially, via more contributions / contributors who don't have to keep their code compatible with many older Python versions and have turn off all their code quality assuring tools.

Regards

Christian

[1] https://peps.python.org/pep-0008 https://peps.python.org/pep-0008 [2] https://github.com/openstack/openstacksdk/commit/c946294bddd0ad37b707c7f993c... https://github.com/openstack/openstacksdk/commit/c946294bddd0ad37b707c7f993cdbc4706c00d00 [3] https://review.opendev.org/c/openstack/cinder/+/909689 https://review.opendev.org/c/openstack/cinder/+/909689 [4] https://review.opendev.org/c/openstack/openstacksdk/+/900710 https://review.opendev.org/c/openstack/openstacksdk/+/900710

On 2024-04-04 15:45:20 +0000 (+0000), Mohammed Naser wrote: [...]

Please stop gatekeeping peoples' efforts because you don't agree with them.

[...]

My goal was not to gate-keep, and I'm no more in favor of stagnation than anyone else. This is advice on how to sway the unconvinced parts of our community. "The status quo wins by default" is a basic law of society. If you can't convince other people to help with your proposed changes, then it's on you to do the work. If you can't do it by yourself, then nothing changes. QED.

Am Montag, dem 08.04.2024 um 14:54 +0100 schrieb smooney@redhat.com:

the tone that Sven has been taking in this thread has been often bordering on insuting if not mearly condesending.

@sven for people that care deeply about the quality of software we produce and has advocated for using tools like mypi, auto formnating, code modreisers liek pyupgrade ectra the way you are approching this is hurting the efforts to maintain and moderise our codebase more then its helping.

we have been having dissusion like this for years and we are making incremenal impormbet but any kind of browbeating with stats just reads as trying to bully people with data instead of takeing the time to have a calm, reasoned converstation that empathises with the others invovled.

Hi Sean,

I'm sorry that I apparently insulted you, that was really not what I wanted to do. I certainly have learned a lot about openstack from your insights posted to this mailing list in the past.

I know there have been previous discussions about these topics and that there are gradual improvements. I also think only incremental improvements are possible in a large project like openstack.

It was not my intend to "bully" anyone with data. But it was raised the question between the lines if there is any advantage to static typing at all. I just replied with a study that shows that at least under some circumstances, yes there are advantages.

I understand and think that it's good that people like me who demand change need of course put in the work and show that change is beneficial at all.

I just try to do that.

Sorry again if my tone went over the line, arguments around coding style and which tools to use somehow often can become very heated.

Kind regards

On Wed, Apr 10, 2024 at 10:09 AM smooney@redhat.com wrote:

On Wed, 2024-04-10 at 14:37 +0200, Sven Kieske wrote:

...

Am Montag, dem 08.04.2024 um 14:54 +0100 schrieb smooney@redhat.com:

...

the tone that Sven has been taking in this thread has been often bordering on insuting if not mearly condesending.

@sven for people that care deeply about the quality of software we produce and has advocated for using tools like mypi, auto formnating, code modreisers liek pyupgrade ectra the way you are approching this is hurting the efforts to maintain and moderise our codebase more then its helping.

we have been having dissusion like this for years and we are making incremenal impormbet but any kind of browbeating with stats just reads as trying to bully people with data instead of takeing the time to have a calm, reasoned converstation that empathises with the others invovled.

Hi Sean,

I'm sorry that I apparently insulted you, that was really not what I wanted to do. I certainly have learned a lot about openstack from your insights posted to this mailing list in the past.

I know there have been previous discussions about these topics and that there are gradual improvements. I also think only incremental improvements are possible in a large project like openstack.

It was not my intend to "bully" anyone with data. But it was raised the question between the lines if there is any advantage to static typing at all. I just replied with a study that shows that at least under some circumstances, yes there are advantages.

I understand and think that it's good that people like me who demand change need of course put in the work and show that change is beneficial at all.

I just try to do that.

Sorry again if my tone went over the line, arguments around coding style and which tools to use somehow often can become very heated.

no worreis i can get pretty pasionate about the things i care about too. software quality is one of them. i personally find value in type hints in python and use them for new code i write in general i konw others find less value in them but thats where indvigual projects can show leadership and experiment and then report back.

To piggyback on this, this topic was discussed yesterday during Neutron vPTG sessions, and the team decided to try the following:

- pick neutron-lib for typing adoption; reasons for neutron-lib are: it's our central python API component and consuming projects could benefit from typing both for analysis and as documentation, plus it's relatively simple; - Miro T will build infrastructure (tox, docs etc.) to start adoption; - we'll then proceed adopting typing there gradually and see if this is a success; - Eventually, we may want to step up for other repos, depending on results.

without opening the holy war of black vs pep8 that is another example of where some project have adopted it as a code formater and other have not both for vaild reasons.

if we were startign openstack from scratch today we would likely use many of these tools its harder to adopt them in existing projects.

i dont really agree with the main pushback (the impact on backports) but that does not mean indeivugal project cant adopt these tools

the sdk team felt the benifit outweighted any cost

https://github.com/openstack/openstacksdk/commit/a36f514295a4b4e6157ce69a210...

the imporant thing to rememeber is that the indivgual projects are empored to make some of these decisiosn by them seleve and dont need tc approval to do so.

we do have some rules around the project testing interfance that all offical project should follow

https://github.com/openstack/governance/blob/master/reference/pti/python.rst but that still leave a lot of flexablity for teams to evolve there workflow.

its take a few years but for nova we have impoved the workflow (IMO) by adotpting precommit as an optional way to run many of the simple/fast styple checks https://github.com/openstack/nova/blob/master/.pre-commit-config.yaml while also ensuring that those who dont want to use pre-commit can continue to use tox https://github.com/openstack/nova/blob/master/tox.ini#L101-L132

we have been more conservitive then i would personally like but we have at least been open to the discussion adn if someone was willing to do the work trying to make these improments.

if you approch it form that perspective i think you will much more eaislly which reminds me i need to rebase https://review.opendev.org/q/topic:%22sphinx-lint%22 to enable sphinx lint properly...

...

Kind regards