On 3/11/20 10:15 AM, Thierry Carrez wrote:

Hi all,

I'd like to issue a call for consolidation and simplification for OpenStack development.

In the early years of the project, we faced a lot of challenges. We had to spread the development load across manageable-size groups, so we encouraged the creation of a lot of project teams. We wanted to capture all the energy that was sent towards the project, so we passed project structure reforms (like the big tent) that would aggressively include new community groups in the "official" OpenStack community. We needed to remove bottlenecks, so we encouraged decentralized decision making. And we had to answer unique challenges, so we created software to match them (Zuul). In summary, we had a lot of people, and not enough systems to organize them, so we created those.

Fast-forward to 2020, and our challenges are different. The many systems that we created in the early days have created silos, with very small groups of people working in isolation, making cross-project work more difficult than it should be. The many systems that we created generate a lot of fragmentation. Like we have too many meetings (76, in case you were wondering), too much energy spent running them, too much frustration when nobody joins. Finally, the many systems that we created represent a lot of complexity for newcomers to handle. We have 180 IRC channels, most of them ghost towns where by the time someone answers, the person asking the question is long gone.

So I think it's time to generally think about simplifying and consolidating things. It's not as easy as it sounds. Our successful decentralization efforts make it difficult to make the centralized decision to regroup. It's hard to justify time and energy spent to /remove/ things, especially those that we spent time creating in the first place. But we now have too many systems and not enough people, so we need to consolidate and simplify.

Back around Havana, when we had around the same number of active contributors as today, we used to have 36 meetings and 20 teams. Do we really need 180 IRC channels, 76 meetings, 63 project teams (not even counting SIGs)?

Yes, we all specialized over time, so it's hard to merge for example Oslo + Requirements, or QA + Infrastructure, or Stable + Release Management, or Monasca + Telemetry. We are all overextended so it's hard to learn new tricks or codebases. And yet, while I'm not really sure what the best approach is, I think it's necessary.

We've often had a fair amount of overlap between Oslo and some of the other horizontal teams like releases and requirements, which makes a certain amount of sense since they're all cross-OpenStack efforts. Naturally they tend to attract the same people. That said, would it make sense to merge with any of them? I'm unsure. And that's not a passive-aggressive "unsure", I actually don't know ;-). I will say that I feel pretty good about where the Oslo team is right now. Our meetings are generally well-attended, I would say even better than they were a year ago, and there's good discussion that happens. Many weeks topics are brought up by someone who is not me, which seems like a good sign of engagement. I guess we'll see what happens in the upcoming PTL election, but I'm not feeling like we need to do anything drastic to ensure a positive future for the project. Maybe that's an argument that we should bring another smaller team under our umbrella. We kind of just did that with the docs team not so long ago. I don't know if anyone else has strong opinions about how that has gone - mostly it hasn't changed much for me as PTL other than having a few more projects to review and release from time to time. I'm not sure if there are other projects where a merger would go as smoothly, but I'm open to suggestions. I don't know if any of the above is helpful at all, but I think it's a good summary of my thoughts as I've considered this.

Comments, thoughts?

On Wed, Mar 11, 2020 at 8:18 AM Thierry Carrez <thierry@openstack.org> wrote:

Hi all,

I'd like to issue a call for consolidation and simplification for OpenStack development.

[trim]

So I think it's time to generally think about simplifying and consolidating things. It's not as easy as it sounds. Our successful decentralization efforts make it difficult to make the centralized decision to regroup. It's hard to justify time and energy spent to /remove/ things, especially those that we spent time creating in the first place. But we now have too many systems and not enough people, so we need to consolidate and simplify.

Back around Havana, when we had around the same number of active contributors as today, we used to have 36 meetings and 20 teams. Do we really need 180 IRC channels, 76 meetings, 63 project teams (not even counting SIGs)?

I highly doubt we need that much. Part of me wonders about how we record the uses and successes, and also how we broadcast that out. I feel like in us trying to do everything and continue doing everything possible as a community, we've never stopped to ask ourselves why, nor even leave ourselves time to breath and be able to share the story of our successes as well as why we are motivated.

Yes, we all specialized over time, so it's hard to merge for example Oslo + Requirements, or QA + Infrastructure, or Stable + Release Management, or Monasca + Telemetry. We are all overextended so it's hard to learn new tricks or codebases. And yet, while I'm not really sure what the best approach is, I think it's necessary.

Comments, thoughts?

-- Thierry Carrez (ttx)

I suspect some of the divisions in specific focus areas or projects is a place that we likely can't come back from, and that ?might? be a good thing. We have specific projects that have niche or focused use cases that those smaller communities will continue to focus on and support because they are working to solve or support a specific problem space. We also have lots of code, and not everyone can be generalists. Nor can we ask people to try and maintain code that we have no idea if anyone is using or finds that it fills one of their problem spaces. So I think it is important to focus on the projects that are actually in use and have maintainers. I distinctly note "and have maintainers" because we cannot ask people to stretch themselves any more than they already have to support yet another thing, another effort. And ultimately trying to force consolidation is going to be viewed as being asked to give more by contributors. So I do think we as a community need to scale back some of our complexity. Some of the disperse areas of information. But we also need to know where we should be focusing our time and energy, and if our work matters only to ourselves, to our employers, or to the community at large. And I'm sure the immediate thought is that the user survey has this data. I'm not convinced this is the case. The user survey is only a slice of a user-base which maintains a close communication with the community. It gets filled out by those that get the emails, those that see the social media, but users are out there that are not easily reachable via those avenues because they live in different circles. So a few thoughts off the top of my head: * If projects don't have maintainers nor anyone who wishes to be elected as a PTL, the TC should not try and continue to drive the project along by naming a leader of some sort. Those smaller communities can and should coalescence if they are still active through self organization to figure out their next step. * If projects can't coalescence, then we need to consider "maintenance" or some other similar word to indicate inactive. * All non-security releases should stop for the such projects, including no release for the cycle. No horizontal contributor should feel obligated to keep it maintained. * We need to accept that a piece of software in the community may stop working at some point in time for some set of users. This is okay and should actually help bring fixes in that we would not normally receive which becomes an indicator of use, and thus a means to revive! * We need to consolidate our sources of truth. i.e. Sunset and decommission wiki.openstack.org. * We need to learn to trust, and not keep power of the core reviewer. Keeping reviewers to a small select group just hurts projects. Does that mean mistakes may be made? absolutely! Can they be fixed? Most likely and at worst there is revert! Do core reviewers make mistakes too? Definitely! * We need to accept change, and move forward. * We need to abandon the desire for perfection for that blocks forward movement. * We need data to make decisions to move forward, so if we had a simple instrumentation service that allowed operators to post statistics, then that could be amazingly powerful! It would be even better to gamify, but I'm not sure that is entirely possible. For that data/statistics idea, I'm thinking a `<project>-survey` command, which includes a couple questions, and maybe polls some statistical data from the deployment at the time of submission. Phone home sounds worse, but operator invoked functionally that is what I'm thinking on some level. Anyway, I agree we are over extended. I think the only way we can even begin to move away from being over extended is to begin to scale certain things back, but I also feel strongly we have lots of hidden users out there based upon discussions I've had in hallway tracks. Regardless, I hope some of these thoughts and ideas resonate, and nobody decides to get out a flame thrower. Meanwhile, I'll dream of a day when I can +2 nova/virt/ironic code, and cinder folks can +2 ironic/common/cinder code.