Hey Christian. Some comments inline.

On Tue, Feb 20, 2024 at 12:41 PM Christian Rohmann <christian.rohmann@inovex.de> wrote:

Hello Jay, Jeremy, all,

thanks for your responses and the time to write up your thoughts.
On 14.02.24 14:53, Jeremy Stanley wrote:
Don't lose sight of the fact that this is an inherently existential
question. OpenStack is all of us. You are OpenStack. I am OpenStack.
Why are you not pushing ahead? Why am I not pushing ahead? This is
the real question you're asking.
That is exactly on point to why a discussion as a starting point can be important to "push ahead".
While I might be of some opinion that things are not as they could or should be, it might not be the consensus of the greater community, other important and hard-working members of this community.
Does it really help if everybody forms their ideas right into a patch blackening all Python code of a project he once wrote a little patch for?
Or does it help the generals idea if O start sprinkling type annotations all over any code that is OpenStack?
Who would review or even accept that? Writing up a quick proposal requesting the use of type annotations and pylint for all OpenStack code from now on might sound nice, but how would that come across? I know we are using Python, which asks for forgiveness instead of permission, but really?
What we need for any change, apart from any policy or format agreement is foundation is the intention to change, an in a certain direction.

A discussion to me is a good way to see if your idea is onto something and if others might even share it (seems to be the case in the matter of modernizing Python).
It's also a good way to find others that might work together on the next step: A formal proposal if e.g. someone from TC even encourages to do that :-)
But even if that proposal was then not even written by OP or other participants of the discussion - a heated discussion or the agreement by some established OpenStack contributors, might even be a good indication to TC that something is an important topic which could need some formal backing or follow up in one form or another.

What else would be a good source for TCs agenda than the combined voices of the community?

Many of the technical leaders in the project have been significantly burned in times past from big ideas, in the form of goals, being backed with little/no technical effort to implement them. Even in the cases where those goals were given prioritized attention, there were always smaller projects that struggled to make the necessary lift. This to me has changed the nature of how this work is done.

There are a lot of ideas. The best idea is the one that actually has a chance of being implemented. I think an important part of technical leadership in a large project is ensuring goals set forward are achievable. At this point, I have extremely low amounts of trust that ideas proposed by people who aren't deeply involved in building OpenStack will get enough help to be implemented across all of OpenStack. I point at the RBAC project as a great example -- an amazing feature, something that's a major addition to any cloud it's put in, but there are still projects which don't have full support for it.

This is the heart of what I was trying to get at: the major constraint to the OpenStack agenda is having enough people to implement it. There's only so many folks doing active technical work on OpenStack, and fewer still who focus primarily on OpenStack-wide issues.

On 14.02.24 17:21, Jay Faulkner wrote:

> The community has been very capable if an external change (Death of
> Python 2.7, SQLAlchemy 2.x, eventlet EoL, ....)
> forced some reaction or adaption. But does it really require some
> external force for some kind of modernization?

I'm isolating this for a specific reason: many of the items listed here were driven by the same major contributors (human and corporate) to the community, and/or are in progress. For instance, the eventlet migration still does not have a finalized plan[0] and the primary drivers right now are developers from RedHat as well as a single GR-OSS developer we contributed to help improve eventlet in the short term. When describing this to my boss; I told him I'm incredibly proud that GR-OSS is participating in this work, because so few contributing companies are willing to do this invisible work -- but the sad thing is, eventlet eol is one of a large number of things that needs doing -- and there aren't many people to do it.

I understand your frustration regarding the work on eventlet and yes, these kind of components and libraries are mostly taken for granted. This was seen quite a few times in recent years when e.g. the log4j issue happened.

I simply picked a few examples, where OpenStack, and I am referring to the community of projects and people, to my impression ran into an issue or a deprecation and there was lots of energy to make the required changes happen across many projects quickly to come out better on the other end, technically and socially. But more importantly it was the challenge and need for action that brought people together in accepting change. While you personally might rightly complain that the urgently needed work on eventlet was only done by a few, I also see the consensus for the necessary or even overdue change as another outcome.

To maybe try another example, Python 3.12, e.g. https://review.opendev.org/q/topic:%22python-3.12%22
While frustrating to see a simple version bump of Python causing lots of issues and CI fails. Put in the right perspective, any of those changes is part of that push forward and that keep the codebase alive.

Yep; that's the exact thing I'm talking about re: the python 3.12 movement. We do not support python 3.12, but despite that, Zigo has been driving work to help deal with python compatibility issues before those python versions are even officially supported. A great example of doing something rather than getting permission :). Things like this don't often make a lot of noise, but they are impactful and lower the gaps we have to cover when those versions come into our supported platform.

This email thread highlights many of those other issues which need addressing; but are missing a significant portion: a call to action and real work to make them happen.

There are several ways you help change this:

1) Don't like how the project is being run from a technical perspective? Run for the TC -- there certainly is a need for volunteers. Try to make change happen. Governance is not easy and there are hundreds of people with different opinions about what direction "forward" is -- and it's a lot more work than just adding and removing python versions from platform documents.

Not to repeat what I said above. But there might be hundreds of opinions voiced here and otherwise. What's important is a good mannered discussion leading to some kind of distillation of opinions into somewhat of an agreement, a common denominator. And isn't the open and public discussion actually helpful for TC identify an issue that is worth looking into? Why should one start with a nicely written up proposal holding all the answers, instead of starting a discussion around a topic first and then see if others share the impressions that the Python stack has some issue that really should be tackled?

The problem is the ratio of conversations/discussion to action. There's a large amount of people willing to help set an agenda; a much more limited set of people willing to engage it. That's why some of these comments have a tone of frustration-- the number of times this sort of thread shows up is much, much higher than the follow-through for ideas brought up in these threads. It's also why I am trying to be a pragmatist: what specific things can we do with the resources we have and/or that the community is willing to bring to bear.

2) Do a proof of concept in a smaller project. I'd personally be willing to help you gain consensus on implementing (for instance) some type hints on Ironic. I can't promise the whole Ironic team would agree; but the point is the easiest way to influence change in OpenStack is to show willingness to do the work. For SQLA 2.0 migration, for instance, it was obvious there were some members of the community who worked towards it, with urgency, even pulling forward projects that are now inactive to function with SQLA 2.0.

Don't get me wrong, but I feel there is a little more of general frustration than actual reasoning here. Discrediting the (unknown) amount or value of contributed work by someone else, likely is not going to motivate them to help. Reading this, I was just about to vent on my own frustration of trying to get even some little patch reviewed and merged and the need drive even small changes home over multiple releases. This contributes, just like the technical state of the Python stack, to developer experience, their willingness to contribute and stay with the project. And I'd like to generalize this as much as I can, as it's quite universal for my experience with different OpenStack projects and not related to any person or team in particular. But having contributions (and also bug reports) that are not responded to in any way will scare people away from contributing and likely for good.

I gladly write up more detailed thoughts and experiences if someone would be using them to reflect on how things are organized.

This is actually a topic I, personally, am relatively passionate about, even if the direction of this conversation doesn't exactly reflect it :). Lowering the effort new developers need to get OpenStack contributions done would be a very valuable improvement. However, the kinds of changes that are needed require more than drive-by levels of contribution, so in a way I see this as a little tangential to the problem at hand.

3) Advocate inside your organization for more upstream contribution; either in the form of your time or additional upstream contributors. A recent paper[1] out of Harvard Business School says 96% of the demand-side value is created by only 5% of OSS developers. Everyone who has a leadership position at a company that uses or contributes to OSS has an obligation to advocate to help close these gaps.

While I agree with this point in general, again, I don't think it's a good response to the topic of this thread.
There are many issues, big and small, that contribute to the fact that OpenStack receives less contributions over time and most of them are by a small group of people, most of them on the payroll of the same group of companies. While I agree that institutions and companies should do their part, also asking why OpenStack and it's projects are not really attractive to contribute to as a sole developer from some smaller installation might also be helpful to discuss as it ensures the community remains healthy and broad. Yet another good topic for governance. Not to answer, but to bring to the agenda and actively address.

This is an honest question: do you think these problems are meaningfully solvable by drive-by contributors? In the past, OpenStack-wide goals have taken years of committed work to get released. OpenStack has *hundreds* of repos of python software, even a small change across all of them is time consuming, even if there's no real technical difficulty and consensus is already reached.

I think the key problem is a perception that every problem is solvable by the TC, the Foundation Board, or project governance in general. We each have an individual and professional responsibility, as members of the OpenStack community, to help ensure things are properly resourced. I've done this before I was in the TC, while I've been on the TC, and once I am no longer on the TC -- it's not a governance function, it's an important form of community contribution.

I'm not going to break down the original email point by point, but generally: yes, there are things that OpenStack are behind on -- but emails to the mailing list and consensus aren't the solution. The solution is more people contributing to solve these problems -- especially in traditionally under-resourced teams such as QA, oslo libraries, and smaller projects on the verge of inactivity. Jeremy said (paraphrased) that we are OpenStack; rather than asking why OpenStack is not pushing forward, we should be asking ourselves why we aren't pushing forward. I agree with him. I also agree with the OP which asserts that we have clear technical improvement opportunities. What's stopping you from pushing them forward?

What made you assume the OP or myself are not actively pushing specs, code and ideas (like this one here)? Did you consider the urge to post to the ML and participate in a discussion on modernizing the Python stack could potentially be out of the experience doing so? Even if I hadn't done any such work, would the arguments about the Python stack be less valid?

If I may bluntly and selectively quote https://review.opendev.org/c/openstack/governance/+/902585/16/goals/proposed/migrate-eventlet-usages-to-asyncio.rst#1075 ....

A community goal does not shape a new and personal vision of Openstack.
A Community goal collects this vision from the scattered hopes and intentions of our community's past.
[...]
The TC is elected to provides technical leadership.
[...]
Without action we will soon face a brutal discontinuation.

With that, let me sum up my thoughts a little ...

While some tech debt in the Python stack is not an immediate existential crisis, it's vital to keep moving and to stay on top of things.
Adopting new developments in the ecosystem is for the sake of quality (security, bugs, speed, efficiency, ...) and to avoid running into short term firefighting mode (deprecation, technical dead-ends, ...). But most importantly, if I read your responses, Jay and Jeremy, it's about lowering the barrier for "outside" contributions again.
Code has to be approachable (e.g. is well structured, has type hinting, current syntax styles, ...) by part-time contributors to even have them try contributing and push a patch. The mentioned well established tooling such a formatters, linters or static code analysis and others should be used to assist senior and full-time devs with the QA and review processes to ultimately enable more contributions and by more individual contributors to be merged.

I agree with all of those ideas. I do not think that they are an endpoint to be targeted in whole. A smaller, achievable scope with clearly defined borders will be easier to scale across all of OpenStack and has a better chance of moving the project forward. The eventlet migration goal is a *great* example of this -- we broke off a large piece of an even larger problem (python 3.12 support) and are then able to take steps to resolve it.

I would advise if you want this to be a successful endeavor to pick one specific item to champion, show a proof of concept in a smaller OpenStack project, and move it forward. Giving a scope and a business case for why fixing a thing is important -- it's how you end up not doing this work alone. I'd define consensus in OpenStack as requiring not only general agreement on what should happen; but agreement to help make it happen. That's the only form of consensus I've seen that has an endpoint of real change and improvement.

Thanks again for your time,

Just noting -- I suspect you and I have many of the same values, but are just seeing things from different perspectives and on different time horizons. I strongly encourage you to run for a leadership position during this election cycle if governance and consensus building is something you care about.

Thanks,

Jay Faulkner

OpenStack TC Chair