Don't lose sight of the fact that this is an inherently existential question. OpenStack is all of us. You are OpenStack. I am OpenStack. Why are you not pushing ahead? Why am I not pushing ahead? This is the real question you're asking.
> The community has been very capable if an external change (Death of> Python 2.7, SQLAlchemy 2.x, eventlet EoL, ....)
> forced some reaction or adaption. But does it really require some
> external force for some kind of modernization?
I'm isolating this for a specific reason: many of the items listed here were driven by the same major contributors (human and corporate) to the community, and/or are in progress. For instance, the eventlet migration still does not have a finalized plan[0] and the primary drivers right now are developers from RedHat as well as a single GR-OSS developer we contributed to help improve eventlet in the short term. When describing this to my boss; I told him I'm incredibly proud that GR-OSS is participating in this work, because so few contributing companies are willing to do this invisible work -- but the sad thing is, eventlet eol is one of a large number of things that needs doing -- and there aren't many people to do it.
I understand your frustration regarding the work on eventlet and yes, these kind of components and libraries are mostly taken for granted. This was seen quite a few times in recent years when e.g. the log4j issue happened.
I simply picked a few examples, where OpenStack, and I am referring to the community of projects and people, to my impression ran into an issue or a deprecation and there was lots of energy to make the required changes happen across many projects quickly to come out better on the other end, technically and socially. But more importantly it was the challenge and need for action that brought people together in accepting change. While you personally might rightly complain that the urgently needed work on eventlet was only done by a few, I also see the consensus for the necessary or even overdue change as another outcome.
To maybe try another example, Python 3.12, e.g.
https://review.opendev.org/q/topic:%22python-3.12%22
While frustrating to see a simple version bump of Python causing
lots of issues and CI fails. Put in the right perspective, any of
those changes is part of that push forward and that keep the
codebase alive.
This email thread highlights many of those other issues which need addressing; but are missing a significant portion: a call to action and real work to make them happen.
There are several ways you help change this:
1) Don't like how the project is being run from a technical perspective? Run for the TC -- there certainly is a need for volunteers. Try to make change happen. Governance is not easy and there are hundreds of people with different opinions about what direction "forward" is -- and it's a lot more work than just adding and removing python versions from platform documents.
Not to repeat what I said above. But there might be hundreds of
opinions voiced here and otherwise. What's important is a good
mannered discussion leading to some kind of distillation of
opinions into somewhat of an agreement, a common denominator. And
isn't the open and public discussion actually helpful for TC
identify an issue that is worth looking into? Why should one start
with a nicely written up proposal holding all the answers, instead
of starting a discussion around a topic first and then see if
others share the impressions that the Python stack has some issue
that really should be tackled?
2) Do a proof of concept in a smaller project. I'd personally be willing to help you gain consensus on implementing (for instance) some type hints on Ironic. I can't promise the whole Ironic team would agree; but the point is the easiest way to influence change in OpenStack is to show willingness to do the work. For SQLA 2.0 migration, for instance, it was obvious there were some members of the community who worked towards it, with urgency, even pulling forward projects that are now inactive to function with SQLA 2.0.
Don't get me wrong, but I feel there is a little more of general frustration than actual reasoning here. Discrediting the (unknown) amount or value of contributed work by someone else, likely is not going to motivate them to help. Reading this, I was just about to vent on my own frustration of trying to get even some little patch reviewed and merged and the need drive even small changes home over multiple releases. This contributes, just like the technical state of the Python stack, to developer experience, their willingness to contribute and stay with the project. And I'd like to generalize this as much as I can, as it's quite universal for my experience with different OpenStack projects and not related to any person or team in particular. But having contributions (and also bug reports) that are not responded to in any way will scare people away from contributing and likely for good.
I gladly write up more detailed thoughts and experiences if
someone would be using them to reflect on how things are
organized.
3) Advocate inside your organization for more upstream contribution; either in the form of your time or additional upstream contributors. A recent paper[1] out of Harvard Business School says 96% of the demand-side value is created by only 5% of OSS developers. Everyone who has a leadership position at a company that uses or contributes to OSS has an obligation to advocate to help close these gaps.
While I agree with this point in general, again, I don't think
it's a good response to the topic of this thread.
There are many issues, big and small, that contribute to the fact
that OpenStack receives less contributions over time and most of
them are by a small group of people, most of them on the payroll
of the same group of companies. While I agree that institutions
and companies should do their part, also asking why OpenStack and
it's projects are not really attractive to contribute to as a sole
developer from some smaller installation might also be helpful to
discuss as it ensures the community remains healthy and broad. Yet
another good topic for governance. Not to answer, but to bring to
the agenda and actively address.
I'm not going to break down the original email point by point, but generally: yes, there are things that OpenStack are behind on -- but emails to the mailing list and consensus aren't the solution. The solution is more people contributing to solve these problems -- especially in traditionally under-resourced teams such as QA, oslo libraries, and smaller projects on the verge of inactivity. Jeremy said (paraphrased) that we are OpenStack; rather than asking why OpenStack is not pushing forward, we should be asking ourselves why we aren't pushing forward. I agree with him. I also agree with the OP which asserts that we have clear technical improvement opportunities. What's stopping you from pushing them forward?
What made you assume the OP or myself are not actively pushing
specs, code and ideas (like this one here)? Did you consider the
urge to post to the ML and participate in a discussion on
modernizing the Python stack could potentially be out of the
experience doing so? Even if I hadn't done any such work, would
the arguments about the Python stack be less valid?
If I may bluntly and selectively quote
https://review.opendev.org/c/openstack/governance/+/902585/16/goals/proposed/migrate-eventlet-usages-to-asyncio.rst#1075
....
A community goal does not shape a new and personal vision of Openstack.
A Community goal collects this vision from the scattered hopes and intentions of our community's past.
[...]
The TC is elected to provides technical leadership.
[...]
Without action we will soon face a brutal discontinuation.
With that, let me sum up my thoughts a little ...
While some tech debt in the Python stack is not an immediate
existential crisis, it's vital to keep moving and to stay on top
of things.
Adopting new developments in the ecosystem is for the sake of
quality (security, bugs, speed, efficiency, ...) and to avoid
running into short term firefighting mode (deprecation, technical
dead-ends, ...). But most importantly, if I read your responses,
Jay and Jeremy, it's about lowering the barrier for "outside"
contributions again.
Code has to be approachable (e.g. is well structured, has type
hinting, current syntax styles, ...) by part-time contributors to
even have them try contributing and push a patch. The mentioned
well established tooling such a formatters, linters or static code
analysis and others should be used to assist senior and full-time
devs with the QA and review processes to ultimately enable more
contributions and by more individual contributors to be merged.
Thanks again for your time,