[all][dev][ops][tc] Bridging the gap between community and contributing orgs
The tl;dr on this is that I'm opening a broad community discussion on how we can, collectively through improved communication, better bring together our established contributors with casual and prospective contributors who struggle to find successful patterns of contribution, so that we might all support each other in order to benefit OpenStack as a whole. Read on for details... OpenStack's TC has requested that foundation staff, when in conversation with representatives of member organizations, encourage participation in the project and collect feedback on any related challenges those organizations encounter in their attempts to do so. In order to better understand the feedback and brainstorm achievable recommendations for the broader community[*], Community Managers on the foundation staff started to work with a small focus group of established contributors with a solid understanding of what successful patterns of contribution look like. It also came to light that established contributors experience many of the same sorts of challenges, even more so if they don't have the luxury of focusing full-time upstream, and so success often comes down to knowing how to effectively navigate those challenges. The community's goal with this exercise is to improve everyone's experience, and your help is needed to do that! The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. Ideas so far to address these gaps include: * Review bandwidth - Ensure contributors and companies understand the importance and benefits of code review - Incentivize and credit meaningful "+1" reviews more - Understand that the volume of changes and limited reviewer bandwidth often means proposals aren't reviewed quickly (or at all), and their proponents may need some additional tenacity and engagement with the team to get their work noticed - Utilize "review dashboards" as a way to find stale changes which have fallen through the cracks * Review strategy and etiquette - Be clear about the meaning of votes on changes, especially negative ones, and set explicit timelines for things like "procedural -2" or WIP blocks when their approval needs to be delayed - Focus on comments and requests that affect whether or not the change can be merged after the fixes are submitted, rather than requesting trivial adjustments to an unsuitable change which has more fundamental problems - Reciprocate with reviews of changes from new reviewers you see leaving insubstantial reviews, demonstrating to them what deeper and more meaningful reviewing looks like * Mentoring newcomers - Established contributors with sufficient bandwidth can help mentoring newcomers, potential core reviewers and new leaders - Mentoring can happen as part of internship programs as well as by helping newcomers determined to become active contributors; both ways should be embraced and utilized What challenges are you facing? And, how would you improve them? Two weeks from now, we'll have a forum session at the OpenInfra Summit Asia in Suwon[**] where our community can refine these possible approaches and integrate others, as well as bring attention to them and promote their application. I've also proposed a similar forum session for OpenInfra Days North America in Indianapolis a month after that, where we can hopefully continue this conversation. The goal for this multi-stage effort is to improve the contribution experience for established participants, casual contributors and newcomers alike. While it's easy to jump to blaming in these situations, it will be far more productive if we can focus on the challenges and experiences that we would each like to remove or improve for ourselves and others in our community. As we find ways together to improve our efficiency, we can reduce the load on all contributors and lower the barriers for people to join and participate. [*] https://etherpad.opendev.org/p/r.2205024e55689bccb82c20c960853cb5 [**] https://2024.openinfraasia.org/a/schedule#view=calendar&title=Bridging%20the%20Gap -- Jeremy Stanley
On 2024-08-21 15:18:12 +0000 (+0000), Jeremy Stanley wrote: [...]
The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. [...]
Last week, Jens mentioned this article in IRC: https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/code-review-antipatte... Coincidentally published the same day I posted the first message in this thread, it's an amusing semi-satirical look at many of the same challenges our community has also identified. A quick summary of the author's points (though I strongly encourage you to check out the full article if you have time): * Batch up review feedback and try not to "move goalposts" by repeatedly coming up with new problems in subsequent reviews that you could have found the first time through (i.e. don't just stop reviewing on the first issue you find). * Don't demand a change author include unrelated fixes to nearby code (pointing it out as optional in case they want to tackle it if they have time is fine, just be clear it's not a requirement of approval). * Try to reach a consensus of opinion with other reviewers on the same changes/project, and avoid jerking the author around by changing your own opinion on things. * If a proposed implementation is unsuitable, be clear about what kind of alternative implementation would be suitable. * Lead with the most important problems in a change, not the trivial ones. * When reviewing individual changes that are part of a larger approved specification, try to focus on whether those changes implement the spec as written regardless of whether you agree with choices the project has made within the spec (if you disagree, get the spec changed or amended first). * Extremes of opinion can always be found to reject a change, so look for a position of compromise when possible. * Be cautious and deliberate about shifts of expectation over time, since authors are prone to replicate patterns from prior changes and existing code. I think articles like this help to confirm we're on the right track, and I find it a reassuring reminder that these are challenges pretty much all communities struggle with. We're not alone. If we can come up with viable approaches for improving the situation, we'll be helping not just our own community, but open source software projects everywhere. Also, a quick reminder: If you'll be in Suwon for OpenInfra Summit Asia or Indianapolis for OpenInfra Days North America, come to the forum session (Bridging the Gap: Community and Contributing Orgs). Whether or not you're available for the in-person sessions, replies to this mailing list thread can help refine the community's approach to contributor communications so we can start to get some of these things documented in relevant places (Contributor Guide, Project Team Guide, et cetera). -- Jeremy Stanley
Hi Everyone, Jeremy mentioned the Forum session on this topic, which was scheduled for Tuesday (September 3) week at the OpenInfra Summit Asia. I moderated the session on behalf of Jeremy, and reaching out here to share a recap. We had a great conversation with people who were able to join the discussion at the Forum session. Below is a brief summary of what the group talked about, which included sharing experiences and brainstorming about ideas to reduce current challenges that people are facing, both newcomers as well as casual and established contributors. Our conversation touched on three main areas, which were challenges, solution ideas and next steps. Challenges - We have less active developers and maintainers * Vendor companies have been the main group of organizations who invested in OpenStack development upstream, but those investments have reduced by today * Operators, who are around, have less capacity to work upstream, and their focus is more on bug fixing than adding new features - False assumptions * Maintainers have full-time allocation to work on OpenStack * Code review is the only way, outside of code contributions, for companies to provide meaningful help to the community - Project team priorities need to be more clear, and project management type of work is required to use the teams’ bandwidth more efficiently * Sometimes the process is in place, but not visible for people who are not engaged on IRC, etc - Even in case of established contributors, they often need to ping people on an ongoing basis to get their reviews to move forward, this method does not scale Solution ideas - Education on open source investment and forms of involvement needs to continue * Companies still spend big chunks of money on proprietary SW development, they need to understand the reasons and benefits to also invest in working on open source projects they rely on, such as OpenStack - Documented and visible processes * OpenStack project teams need to revisit their processes to make sure that the contributors are spending their time and efforts efficiently * When processes exist they also need to be discoverable by everyone, without the need to talk to an established contributor or maintainer - Code review is not the only area where maintainers need help. When an individual or comany, like an operator, has expertise in particular features or areas of the code, their input is needed and desired when changes are made to those parts of the code. - Tooling is not the solution to many of the challenges, so we need to focus on solving the social issues Next steps - Documentation and making existing processes visible, folks from the Nova team made a commitment to update their existing documentatoion to make it up to date and contain their current processes on how they do project management for the team * The Nova team also started to track bug fixes as part of their process to track changes and have the ability to prioritize, in recognition how that is an area where operators often get involved - Start small and replicate practices that works in the wider community * When project teams find ways that help their work, the community should consider exploring and adopting those techniques more widely For more detailed notes, please check the etherpad we used for the session: https://etherpad.opendev.org/p/r.914969755fedc5c6fa58d7e9881ffa66 Those who attended the Forum session, please chime in with any further thoughts and notes on anything that I missed to mention, and to continue the conversation. Those of you who did not have the opportunity to participate in the Forum session, what from the above summary and notes on the etherpad do you resonate with? Which experiences do you have as well and what solution ideas have you tried already? Best Regards, Ildikó ——— Ildikó Váncsa Director of Community Open Infrastructure Foundation
On Aug 29, 2024, at 4:18 AM, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2024-08-21 15:18:12 +0000 (+0000), Jeremy Stanley wrote: [...]
The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. [...]
Last week, Jens mentioned this article in IRC:
https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/code-review-antipatte...
Coincidentally published the same day I posted the first message in this thread, it's an amusing semi-satirical look at many of the same challenges our community has also identified. A quick summary of the author's points (though I strongly encourage you to check out the full article if you have time):
* Batch up review feedback and try not to "move goalposts" by repeatedly coming up with new problems in subsequent reviews that you could have found the first time through (i.e. don't just stop reviewing on the first issue you find).
* Don't demand a change author include unrelated fixes to nearby code (pointing it out as optional in case they want to tackle it if they have time is fine, just be clear it's not a requirement of approval).
* Try to reach a consensus of opinion with other reviewers on the same changes/project, and avoid jerking the author around by changing your own opinion on things.
* If a proposed implementation is unsuitable, be clear about what kind of alternative implementation would be suitable.
* Lead with the most important problems in a change, not the trivial ones.
* When reviewing individual changes that are part of a larger approved specification, try to focus on whether those changes implement the spec as written regardless of whether you agree with choices the project has made within the spec (if you disagree, get the spec changed or amended first).
* Extremes of opinion can always be found to reject a change, so look for a position of compromise when possible.
* Be cautious and deliberate about shifts of expectation over time, since authors are prone to replicate patterns from prior changes and existing code.
I think articles like this help to confirm we're on the right track, and I find it a reassuring reminder that these are challenges pretty much all communities struggle with. We're not alone. If we can come up with viable approaches for improving the situation, we'll be helping not just our own community, but open source software projects everywhere.
Also, a quick reminder: If you'll be in Suwon for OpenInfra Summit Asia or Indianapolis for OpenInfra Days North America, come to the forum session (Bridging the Gap: Community and Contributing Orgs). Whether or not you're available for the in-person sessions, replies to this mailing list thread can help refine the community's approach to contributor communications so we can start to get some of these things documented in relevant places (Contributor Guide, Project Team Guide, et cetera). -- Jeremy Stanley
A quick reminder: If you'll be in Indianapolis for OpenInfra Days North America next week, come to the "Bridging the gap between community and contributing organizations" Forum session at 11:20 AM EDT on Tuesday in room 305. If you can't make it, don't worry! Later this month, a cross-project PTG session is booked for 17:00 UTC on Tuesday (October 22) under a "bridging-the-gap" track name. Whether or not you're available for the above synchronous discussions, replies to this mailing list thread can help refine the community's approach to contributor communications so we can start to get some of these things documented in relevant places (Contributor Guide, Project Team Guide, et cetera). -- Jeremy Stanley
On 2024-10-10 00:37:30 +0000 (+0000), Jeremy Stanley wrote:
A quick reminder: If you'll be in Indianapolis for OpenInfra Days North America next week, come to the "Bridging the gap between community and contributing organizations" Forum session at 11:20 AM EDT on Tuesday in room 305.
The OID-NA session was a great success, with at least a dozen participants (I should have counted, sorry!). You can find the notes at https://etherpad.opendev.org/p/r.06f725e22f12d0e72639b5ac0c8bdac3 but with only 40 minutes we didn't get too deep into details. We talked a bit about how the list of findings and recommendations, and how they're categorized so far, resonated with people in the room. Some related personal experiences where they felt their proposed changes were not evaluated fairly or concerns taken seriously. There seemed to be consensus that perceived affiliation influences were likely to be merely statistical reality or at worst subconscious bias, and that communication challenges like those we've already highlighted are primarily emergent social behaviors, habits or basic human tendencies which are best addressed through education and documentation rather than technical means. There was a bit of debate about letting the perfect be the enemy of the good when it comes to the absence of suitable test environments for specialized hardware integration, and we talked some about the possibility or challenges of third-party testing vs including some research clouds into OpenDev's stable of available resources. Participants also saw value in having more clearly documented escalation paths for technical disagreements (core reviewer... team consensus... PTL... TC). These topics spilled over into an "OpenInfra leaders" session later, so https://etherpad.opendev.org/p/r.90cac8145c6e01bc19d7c02d4df6f8b3 is worth reviewing as well (around lines 16-23).
If you can't make it, don't worry! Later this month, a cross-project PTG session is booked for 17:00 UTC on Tuesday (October 22) under a "bridging-the-gap" track name. [...]
That's right, past me! I'm hoping we can use the session tomorrow to start divvying up writing tasks and identifying which documents might be the most appropriate locations for the amassed recommendations. Everyone's welcome, but people willing to volunteer to work on some of this are even more welcome. ;) In addition to tomorrow's PTG session on this topic, the TC devoted an hour to their role in the effort earlier today, notes begin around line 317 in https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b for those of you interested in an early recap. -- Jeremy Stanley
On 2024-10-21 23:57:09 +0000 (+0000), Jeremy Stanley wrote: [...]
I'm hoping we can use the session tomorrow to start divvying up writing tasks and identifying which documents might be the most appropriate locations for the amassed recommendations. Everyone's welcome, but people willing to volunteer to work on some of this are even more welcome. ;)
In addition to tomorrow's PTG session on this topic, the TC devoted an hour to their role in the effort earlier today [...]
Now that people are hopefully recovered and recharged from the PTG, and settling in for quieter times as the end of the year approaches, it's a good opportunity to take stock of where we are on this effort. First, for those who missed the TC session and haven't had a chance to peruse the notes yet, https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b#L321 is a fairly quick read. For an even quicker summary, we had a brief (re)introduction to the overarching topic and recap of past discussion series, as well as perceived next steps. Some specific ideas were also floated like better documentation on using Matrix to access OFTC IRC, expanding on the "escalation path" suggestion from OID-NA, scheduling project review days and sharing dashboards to reduce the number of changes going unnoticed, better gate stability as a means to improve contributor experience, revisiting how strict projects need to be about testing for hardware-specific features, and making reviewers/maintainers more visible so new contributors can find and talk to them. The dedicated working session used https://etherpad.opendev.org/p/bridging-the-gap-series-4 as a breakdown of previously discussed suggestions. Each item is templated out for additional notes as guidance for volunteers to elaborate on their intended meaning, and a best guess for which parts of what existing documents seem like the most appropriate locations to add or expand on these. We talked about the obvious documents (central contributor guide, per-project contributing docs, project team guide...), but also some non-traditional ideas like resurrecting the Gerrit "WelcomeBot" or canned welcome messages and direct outreach for reviewers. We only got through a couple of items in the series 4 pad, more or less to prove out the workflow, so this is where the rest of you come in! Please take a look at the pad and, if you see a suggestion which is of particular interest to you, please help expand on it, suggest a location (or multiple locations) where it can be documented for improved visibility, and volunteer to push changes! Also feel free to follow up here with any questions or ideas you have for helping move the effort forward to bridge these gaps. -- Jeremy Stanley
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... Anyone who has contributed changes to an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... Anyone who acted as a core reviewer for an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa... If you've done both, please fill in both surveys. If you contributed to or helped maintain deliverables in more than one OpenStack project team, please fill out the appropriate survey(s) once for each team (there is a drop-down selector for the team you're responding about/on behalf of). I know this is a lot to ask, but more data means more insightful analyses should be possible, so thanks for any time you can spare for it. Since not everyone pays close attention to openstack-discuss, it would be great if those of you who read this can bring it to the attention of your teammates and colleagues next week during PTG sessions as well. We're also planing to do some more targeted reminders to contributors as part of the 2025.1 Epoxy release communications, and to project maintainers either next week or immediately after the PTG, but if you've already submitted your responses then you can ignore those. The reason for the relatively short one-month deadline is that I and the other community managers are going to work on analyzing the results for publication to the community within the coming quarter, so hopefully we'll have good first impressions to share some time in June if enough of you help by submitting your experiences. Longer term, I expect we'll re-run these surveys immediately after each release cycle, so that we can start trying to identify trends in the data and determine whether the related Bridging the Gap efforts are making any noticeable impact. Thanks again to everyone for all your help in making OpenStack more welcoming and productive! -- Jeremy Stanley
On 2025-04-01 19:49:13 +0000 (+0000), Jeremy Stanley wrote:
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... [...]
Many thanks to Jens for pointing out in IRC that these are not entirely anonymous. The survey does require you to supply a name and E-mail address so that the foundation staff can reach out for clarification if needed on any free-form responses, and to help weed out bogus responses from bots or vandals. Your responses are treated confidentially, however, as only foundation staff will have access to the raw response data which will be thoroughly anonymized and aggregated before results are shared publicly with the community. The included contact information will only be used for purposes relating to the surveys, not for any other foundation activities. Please let me know if you have any questions, of course! -- Jeremy Stanley
Hi, I’m reaching out to you to give you an update on the surveys that the OpenInfra Foundation community managers have been running. Our goal has been to understand challenges that contributors and maintainers in OpenStack have in order to improve the experience for both. To protect the anonymity of the survey respondents we are sharing anonymized and aggregated survey results reflecting feedback we received throughout OpenStack. As a next step we will reach out to project teams who received more than one survey response, starting with the team(s) that received most, to share team-specific feedback and understand how we can help to address them. Below please find highlights of the two surveys: __MAINTAINER SURVEY__ We received 15 responses from 11 project teams, with 36% of teams receiving more than one response. We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Code review - Changes you propose are reviewed in a timely manner: 2.73 - Code review - You receive actionable feedback from other reviewers: 4.26 - Code review - Automated test failures quickly direct you to problems in your changes: 3.6 - Contributor docs - It is comprehensive to cover processes and good practices: 3.73 - Contributor docs - It is up to date: 3.53 53% of responders said they didn’t refer to their projects Contributor Guide during the Epoxy release cycle, while 20% referred to it 1-9 times and 13% 10-19 times as well as 13% 20+ times. The survey also asked respondents to mark which listed issues they face while trying to land their own changes, as well as trying to land other contributors’ changes. Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who faced each challenge: - Landing their own changes - Trouble getting review attention - 73% - Other - 33% - No changes pass CI - 13% - Need more assistance to proceed with review feedback - 6% - Expected to expand on review's scope - 6% - Disagreement or conflicting review comments - 6% - Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 66% - Change owner can't/won't add missing pieces to the change - 66% - Change owner's response is slow - 60% - Change is lower priority than others - 40% - Change is beyond expertise - 40% - Change broken, doesn't pass CI - 40% - Testing is broken for the project - 20% - Other - 13% - Change needs a BP/spec - 6% The below numbers are the percentages of project teams who faced each challenge: - Landing their own changes - Trouble getting review attention - 64% - Other - 36% - No changes pass CI - 18% - Need more assistance to proceed with review feedback - 9% - Expected to expand on review's scope - 9% - Disagreement or conflicting review comments - 9% - Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 82% - Change owner can't/won't add missing pieces to the change - 73% - Change owner's response is slow - 82% - Change is lower priority than others - 45% - Change is beyond expertise - 55% - Change broken, doesn't pass CI - 45% - Testing is broken for the project - 27% - Other - 18% - Change needs a BP/spec - 9% __CONTRIBUTOR SURVEY__ We received 28 responses from 15 project teams, with 53% of teams receiving more than one response. We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Contributor experience - Changes you propose are reviewed in a timely manner: 2.5 - Contributor experience - You receive actionable feedback from other reviewers: 4.25 - Contributor experience - Automated test failures quickly direct you to problems in your changes: 3.64 - Contributor documentation - You were able to find information about the processes the project team is using: 3.64 - Contributor documentation - It helped you to apply better practices throughout your contribution journey and achieve results faster: 3.57 - Contributor documentation - It is easy to discover: 3.32 - Contributor documentation - It is easy to navigate: 3.36 The survey also asked respondents to mark which listed issues they face while trying to land their own changes. Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who faced each challenge: - Trouble getting review attention - 68% - Other - 36% - Unable to determine priorities - 18% - Reviewers keep coming back with new requests - 11% - Need more clarification on feedback to move forward - 7% - Reviewer expectation or consensus shifts over time - 7% - Expected to expand scope to address other project issues - 7% - Reviewers disagree or give conflicting feedback - 4% - Asked to make changes deviating from past consensus - 4% - Reviewer forgets to un-block change - 4% - Unhelpful or incorrect reviews - 4% The below numbers are the percentages of project teams who faced each challenge: - Trouble getting review attention - 73% - Other - 47% - Unable to determine priorities - 20% - Reviewers keep coming back with new requests - 20% - Need more clarification on feedback to move forward - 13% - Reviewer expectation or consensus shifts over time - 13% - Expected to expand scope to address other project issues - 13% - Reviewers disagree or give conflicting feedback - 7% - Asked to make changes deviating from past consensus - 7% - Reviewer forgets to un-block change - 7% - Unhelpful or incorrect reviews - 7% While most challenges are affecting some or many projects, the biggest issues across the board seem to be getting attention from reviewers, and having delays on both the reviewer and change owner side. This is not necessarily surprising knowing that many or rather most contributors and maintainers are working on OpenStack part-time, or even outside of their day jobs. With the limited bandwidth that the community has, prioritizing work items and clear communication are more important than ever for the success and sustainability of the project. OpenInfra community managers will start reaching out to project teams with most responses and start working on addressing the challenges they face. As we progress with next steps we will also share practices with the OpenStack community at large to help improve contributor experience for everyone. While the official survey deadline has passed, we are still looking for feedback. If you missed filling out the survey(s), please provide your input now! ~ Please fill out this survey separately for every OpenStack project you contributed to during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... ~ Please fill out this survey separately for every OpenStack project you were a core reviewer of during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa... Thanks, Ildikó ——— Ildikó Váncsa Director of Community Open Infrastructure Foundation
On Apr 2, 2025, at 07:27, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2025-04-01 19:49:13 +0000 (+0000), Jeremy Stanley wrote:
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... [...]
Many thanks to Jens for pointing out in IRC that these are not entirely anonymous. The survey does require you to supply a name and E-mail address so that the foundation staff can reach out for clarification if needed on any free-form responses, and to help weed out bogus responses from bots or vandals.
Your responses are treated confidentially, however, as only foundation staff will have access to the raw response data which will be thoroughly anonymized and aggregated before results are shared publicly with the community. The included contact information will only be used for purposes relating to the surveys, not for any other foundation activities.
Please let me know if you have any questions, of course! -- Jeremy Stanley
Hi, I’m reaching out to you about the analysis that OpenInfra Foundation community managers have been running, to understand challenges that contributors and maintainers in OpenStack are facing in order to improve the experience for both. In addition to running surveys, which I shared the anonymized and aggregated results of in my last email[1], we also collected a selection of metrics to get an insight into the load and efficiency of OpenStack project teams. We pulled data from the Bitergia OpenStack dashboard[2] with a script[3]. We collected metrics for 13 project teams covering 5 release cycles from Antelope to Epoxy. As we’ve been collecting survey results and metrics at the same time, the project lists for the two efforts haven’t been synchronized yet. We picked the initial set of projects for the metrics analysis to get an overview from a combination of well-established and newer teams. As we progress further in the process we will match the data sets more closely. We collected the following metrics: - Review Efficiency Index (REI): Merged patches over Opened Patches - Median and Average Time to Review: These metrics focus on the time passed between the creation of a review and first time it gets reviewed by someone - Median and Average Time to Merge: These metrics show the time needed from the creation to the approval of a review - Median and Average Patchsets per Review: These metrics show the number of revisions a review has throughout its lifecycle - # of Changes Opened - # of Changes Closed - #/% of Changes Opened but not Closed in a Release Cycle - # of Maintainers: Anyone in a project team who voted with CR+2, CR-2 or W+1 in a release cycle in any of the project team’s repos. This may not be an exact match to people who are listed as core reviewers in the Gerrit core teams for the project team. - # of Reviewers: Anyone who commented or voted in any way on changes in a release cycle. The intention with building a baseline from a set of metrics is to be able to determine whether or not the current efforts are making any positive impact, or if we should adjust the approach we’ve been taking in any way. To avoid people making comparisons between project teams’ efficiency and performance, we only share anonymized and aggregated data, and provide a high-level picture to the community of what we’ve found so far. - REI: * Over the course of 5 release cycles, most teams had an REI slightly above 1, which means that more changes got closed (merged or abandoned) than opened. * Only the Bobcat release had an REI lower than one, the average across the 13 teams was 0.99. - Patches Opened and Closed * On average throughout the 5 release cycles teams got 412 new reviews, with the minimum of 64 and maximum of 873. * On average throughout the 5 release cycles teams closed 441 patches, with the minimum of 69 and maximum of 1012. * Collectively over the 5 release cycles, the 13 teams received a total of 5356 new changes and closed 5730 reviews. - Review backlog * On average throughout the 5 release cycles, teams had about 6% of new changes that didn’t get closed (merged or abandoned) throughout the release cycle, with the minimum of 1.17% and maximum of 15.8%. * Looking at release cycles, from Antelope to Dalmation the percentage of changes that didn’t close within the release cycle fluctuated between 5% and 8.2%, which jumped to 12.6% in Epoxy. * Of the 5356 new changes throughout the 5 release cycles, 331 took longer than a release cycle to close, or might still be open. - Merge Times * On average throughout the 5 release cycles it took teams 129 days to approve a review, with a median of 25 days. * The team with the lowest average needed 12 days, while their median settled at 5.7 days per cycle. * The team with the lowest median needed 5.2 days, while their average time window was 126 days. * The team with the highest average needed 247 days, while also grabbing the highest median number of 88 days. - Review Times * On average throughout the 5 release cycles it took teams 36 days to conduct the first review on new changes, with a median of 5 days. * The team with the lowest average needed 32 days, while their median settled at 0.9 days per cycle. * The team with the lowest median needed 0.7 days, while their average time window was 22 days. * The team with the highest average needed 88 days, while also grabbing the highest median number of 16 days. Throughout the survey responses challenges with review attention got the highest number of votes, and turned out to be a challenge in every project team who received responses. This is in line with the relatively long timeframe for first reviews shown by the above average numbers. Here I would like to note the difference between ‘average’ and ‘median’ numbers, where the latter shows the most common experience and is not influenced by the outlying datapoints the same way as 'average' is. The overall median review and merge times numbers suggest a relatively good overall experience for all teams across all 5 release cycles, however, the average numbers show areas for improvement. - # of Maintainers * On average teams had 11 active maintainers in the release cycles, with a team having as low as 5 and another team having as high as 21. - # of Reviewers * On average teams had 76 active reviewers in the release cycles, with a team with the lowest of 20 and another team as the highest of 148. There seems to be 7x as many reviewers as maintainers throughout the 13 project teams whose data we looked at. With accepting that not every reviewer is present throughout multiple release cycles, or even throughout a single one, there might be an opportunity to capture some of the more casual reviewers to further increase review bandwidth and nurture a group of new maintainers. The final analysis, with which we will be reaching out to project teams, is building on the combination of survey data and metrics. Please note that we do not have the bandwidth to reach out to every project team at once, as we would like to work with teams on strategy and next steps to address the challenges that we collectively uncover. In the following days, we will be starting to reach out to those teams who got the most survey responses and will expand our outreach as bandwidth allows. Thank you for your patience and understanding. Thanks, Ildikó [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://openstack.biterg.io/app/dashboards#/view/Overview [3] https://opendev.org/opendev/engagement/src/branch/master/tools/metrics/query...
On May 19, 2025, at 10:12, Ildiko Vancsa <ildiko.vancsa@gmail.com> wrote:
Hi,
I’m reaching out to you to give you an update on the surveys that the OpenInfra Foundation community managers have been running. Our goal has been to understand challenges that contributors and maintainers in OpenStack have in order to improve the experience for both.
To protect the anonymity of the survey respondents we are sharing anonymized and aggregated survey results reflecting feedback we received throughout OpenStack.
As a next step we will reach out to project teams who received more than one survey response, starting with the team(s) that received most, to share team-specific feedback and understand how we can help to address them.
Below please find highlights of the two surveys:
__MAINTAINER SURVEY__
We received 15 responses from 11 project teams, with 36% of teams receiving more than one response.
We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Code review - Changes you propose are reviewed in a timely manner: 2.73 - Code review - You receive actionable feedback from other reviewers: 4.26 - Code review - Automated test failures quickly direct you to problems in your changes: 3.6 - Contributor docs - It is comprehensive to cover processes and good practices: 3.73 - Contributor docs - It is up to date: 3.53
53% of responders said they didn’t refer to their projects Contributor Guide during the Epoxy release cycle, while 20% referred to it 1-9 times and 13% 10-19 times as well as 13% 20+ times.
The survey also asked respondents to mark which listed issues they face while trying to land their own changes, as well as trying to land other contributors’ changes. Please note that respondents were allowed to choose as many options as were applicable to them.
The below numbers are the percentages of respondents who faced each challenge:
- Landing their own changes - Trouble getting review attention - 73% - Other - 33% - No changes pass CI - 13% - Need more assistance to proceed with review feedback - 6% - Expected to expand on review's scope - 6% - Disagreement or conflicting review comments - 6%
- Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 66% - Change owner can't/won't add missing pieces to the change - 66% - Change owner's response is slow - 60% - Change is lower priority than others - 40% - Change is beyond expertise - 40% - Change broken, doesn't pass CI - 40% - Testing is broken for the project - 20% - Other - 13% - Change needs a BP/spec - 6%
The below numbers are the percentages of project teams who faced each challenge:
- Landing their own changes - Trouble getting review attention - 64% - Other - 36% - No changes pass CI - 18% - Need more assistance to proceed with review feedback - 9% - Expected to expand on review's scope - 9% - Disagreement or conflicting review comments - 9%
- Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 82% - Change owner can't/won't add missing pieces to the change - 73% - Change owner's response is slow - 82% - Change is lower priority than others - 45% - Change is beyond expertise - 55% - Change broken, doesn't pass CI - 45% - Testing is broken for the project - 27% - Other - 18% - Change needs a BP/spec - 9%
__CONTRIBUTOR SURVEY__
We received 28 responses from 15 project teams, with 53% of teams receiving more than one response.
We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Contributor experience - Changes you propose are reviewed in a timely manner: 2.5 - Contributor experience - You receive actionable feedback from other reviewers: 4.25 - Contributor experience - Automated test failures quickly direct you to problems in your changes: 3.64 - Contributor documentation - You were able to find information about the processes the project team is using: 3.64 - Contributor documentation - It helped you to apply better practices throughout your contribution journey and achieve results faster: 3.57 - Contributor documentation - It is easy to discover: 3.32 - Contributor documentation - It is easy to navigate: 3.36
The survey also asked respondents to mark which listed issues they face while trying to land their own changes. Please note that respondents were allowed to choose as many options as were applicable to them.
The below numbers are the percentages of respondents who faced each challenge: - Trouble getting review attention - 68% - Other - 36% - Unable to determine priorities - 18% - Reviewers keep coming back with new requests - 11% - Need more clarification on feedback to move forward - 7% - Reviewer expectation or consensus shifts over time - 7% - Expected to expand scope to address other project issues - 7% - Reviewers disagree or give conflicting feedback - 4% - Asked to make changes deviating from past consensus - 4% - Reviewer forgets to un-block change - 4% - Unhelpful or incorrect reviews - 4%
The below numbers are the percentages of project teams who faced each challenge: - Trouble getting review attention - 73% - Other - 47% - Unable to determine priorities - 20% - Reviewers keep coming back with new requests - 20% - Need more clarification on feedback to move forward - 13% - Reviewer expectation or consensus shifts over time - 13% - Expected to expand scope to address other project issues - 13% - Reviewers disagree or give conflicting feedback - 7% - Asked to make changes deviating from past consensus - 7% - Reviewer forgets to un-block change - 7% - Unhelpful or incorrect reviews - 7%
While most challenges are affecting some or many projects, the biggest issues across the board seem to be getting attention from reviewers, and having delays on both the reviewer and change owner side. This is not necessarily surprising knowing that many or rather most contributors and maintainers are working on OpenStack part-time, or even outside of their day jobs. With the limited bandwidth that the community has, prioritizing work items and clear communication are more important than ever for the success and sustainability of the project.
OpenInfra community managers will start reaching out to project teams with most responses and start working on addressing the challenges they face. As we progress with next steps we will also share practices with the OpenStack community at large to help improve contributor experience for everyone.
While the official survey deadline has passed, we are still looking for feedback. If you missed filling out the survey(s), please provide your input now!
~ Please fill out this survey separately for every OpenStack project you contributed to during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... ~ Please fill out this survey separately for every OpenStack project you were a core reviewer of during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa...
Thanks, Ildikó
———
Ildikó Váncsa Director of Community Open Infrastructure Foundation
On Apr 2, 2025, at 07:27, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2025-04-01 19:49:13 +0000 (+0000), Jeremy Stanley wrote:
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... [...]
Many thanks to Jens for pointing out in IRC that these are not entirely anonymous. The survey does require you to supply a name and E-mail address so that the foundation staff can reach out for clarification if needed on any free-form responses, and to help weed out bogus responses from bots or vandals.
Your responses are treated confidentially, however, as only foundation staff will have access to the raw response data which will be thoroughly anonymized and aggregated before results are shared publicly with the community. The included contact information will only be used for purposes relating to the surveys, not for any other foundation activities.
Please let me know if you have any questions, of course! -- Jeremy Stanley
Just a quick reminder that the 2025.1/Epoxy cycle contributor and maintainer surveys are still open through April 30 (one week). If you haven't taken them yet, please complete the ones applicable to your participation! To reiterate... Anyone who has contributed changes to an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... Anyone who acted as a core reviewer for an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa... If you've done both, please fill in both surveys. If you contributed to or helped maintain deliverables in more than one OpenStack project team, please fill out the appropriate survey(s) once for each team (there is a drop-down selector for the team you're responding about/on behalf of). I know this is a lot to ask, but more data means more insightful analyses should be possible, so thanks for any time you can spare for it. Since not everyone pays close attention to openstack-discuss, it would be great if those of you who read this can bring it to the attention of your teammates and colleagues during meetings or other interactions. Thanks as always for your continued help in making OpenStack a more effective and welcoming community! -- Jeremy Stanley
participants (2)
-
Ildiko Vancsa
-
Jeremy Stanley