[all][dev][ops][tc] Bridging the gap between community and contributing orgs
The tl;dr on this is that I'm opening a broad community discussion on how we can, collectively through improved communication, better bring together our established contributors with casual and prospective contributors who struggle to find successful patterns of contribution, so that we might all support each other in order to benefit OpenStack as a whole. Read on for details... OpenStack's TC has requested that foundation staff, when in conversation with representatives of member organizations, encourage participation in the project and collect feedback on any related challenges those organizations encounter in their attempts to do so. In order to better understand the feedback and brainstorm achievable recommendations for the broader community[*], Community Managers on the foundation staff started to work with a small focus group of established contributors with a solid understanding of what successful patterns of contribution look like. It also came to light that established contributors experience many of the same sorts of challenges, even more so if they don't have the luxury of focusing full-time upstream, and so success often comes down to knowing how to effectively navigate those challenges. The community's goal with this exercise is to improve everyone's experience, and your help is needed to do that! The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. Ideas so far to address these gaps include: * Review bandwidth - Ensure contributors and companies understand the importance and benefits of code review - Incentivize and credit meaningful "+1" reviews more - Understand that the volume of changes and limited reviewer bandwidth often means proposals aren't reviewed quickly (or at all), and their proponents may need some additional tenacity and engagement with the team to get their work noticed - Utilize "review dashboards" as a way to find stale changes which have fallen through the cracks * Review strategy and etiquette - Be clear about the meaning of votes on changes, especially negative ones, and set explicit timelines for things like "procedural -2" or WIP blocks when their approval needs to be delayed - Focus on comments and requests that affect whether or not the change can be merged after the fixes are submitted, rather than requesting trivial adjustments to an unsuitable change which has more fundamental problems - Reciprocate with reviews of changes from new reviewers you see leaving insubstantial reviews, demonstrating to them what deeper and more meaningful reviewing looks like * Mentoring newcomers - Established contributors with sufficient bandwidth can help mentoring newcomers, potential core reviewers and new leaders - Mentoring can happen as part of internship programs as well as by helping newcomers determined to become active contributors; both ways should be embraced and utilized What challenges are you facing? And, how would you improve them? Two weeks from now, we'll have a forum session at the OpenInfra Summit Asia in Suwon[**] where our community can refine these possible approaches and integrate others, as well as bring attention to them and promote their application. I've also proposed a similar forum session for OpenInfra Days North America in Indianapolis a month after that, where we can hopefully continue this conversation. The goal for this multi-stage effort is to improve the contribution experience for established participants, casual contributors and newcomers alike. While it's easy to jump to blaming in these situations, it will be far more productive if we can focus on the challenges and experiences that we would each like to remove or improve for ourselves and others in our community. As we find ways together to improve our efficiency, we can reduce the load on all contributors and lower the barriers for people to join and participate. [*] https://etherpad.opendev.org/p/r.2205024e55689bccb82c20c960853cb5 [**] https://2024.openinfraasia.org/a/schedule#view=calendar&title=Bridging%20the... -- Jeremy Stanley
On 2024-08-21 15:18:12 +0000 (+0000), Jeremy Stanley wrote: [...]
The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. [...]
Last week, Jens mentioned this article in IRC: https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/code-review-antipatte... Coincidentally published the same day I posted the first message in this thread, it's an amusing semi-satirical look at many of the same challenges our community has also identified. A quick summary of the author's points (though I strongly encourage you to check out the full article if you have time): * Batch up review feedback and try not to "move goalposts" by repeatedly coming up with new problems in subsequent reviews that you could have found the first time through (i.e. don't just stop reviewing on the first issue you find). * Don't demand a change author include unrelated fixes to nearby code (pointing it out as optional in case they want to tackle it if they have time is fine, just be clear it's not a requirement of approval). * Try to reach a consensus of opinion with other reviewers on the same changes/project, and avoid jerking the author around by changing your own opinion on things. * If a proposed implementation is unsuitable, be clear about what kind of alternative implementation would be suitable. * Lead with the most important problems in a change, not the trivial ones. * When reviewing individual changes that are part of a larger approved specification, try to focus on whether those changes implement the spec as written regardless of whether you agree with choices the project has made within the spec (if you disagree, get the spec changed or amended first). * Extremes of opinion can always be found to reject a change, so look for a position of compromise when possible. * Be cautious and deliberate about shifts of expectation over time, since authors are prone to replicate patterns from prior changes and existing code. I think articles like this help to confirm we're on the right track, and I find it a reassuring reminder that these are challenges pretty much all communities struggle with. We're not alone. If we can come up with viable approaches for improving the situation, we'll be helping not just our own community, but open source software projects everywhere. Also, a quick reminder: If you'll be in Suwon for OpenInfra Summit Asia or Indianapolis for OpenInfra Days North America, come to the forum session (Bridging the Gap: Community and Contributing Orgs). Whether or not you're available for the in-person sessions, replies to this mailing list thread can help refine the community's approach to contributor communications so we can start to get some of these things documented in relevant places (Contributor Guide, Project Team Guide, et cetera). -- Jeremy Stanley
Hi Everyone, Jeremy mentioned the Forum session on this topic, which was scheduled for Tuesday (September 3) week at the OpenInfra Summit Asia. I moderated the session on behalf of Jeremy, and reaching out here to share a recap. We had a great conversation with people who were able to join the discussion at the Forum session. Below is a brief summary of what the group talked about, which included sharing experiences and brainstorming about ideas to reduce current challenges that people are facing, both newcomers as well as casual and established contributors. Our conversation touched on three main areas, which were challenges, solution ideas and next steps. Challenges - We have less active developers and maintainers * Vendor companies have been the main group of organizations who invested in OpenStack development upstream, but those investments have reduced by today * Operators, who are around, have less capacity to work upstream, and their focus is more on bug fixing than adding new features - False assumptions * Maintainers have full-time allocation to work on OpenStack * Code review is the only way, outside of code contributions, for companies to provide meaningful help to the community - Project team priorities need to be more clear, and project management type of work is required to use the teams’ bandwidth more efficiently * Sometimes the process is in place, but not visible for people who are not engaged on IRC, etc - Even in case of established contributors, they often need to ping people on an ongoing basis to get their reviews to move forward, this method does not scale Solution ideas - Education on open source investment and forms of involvement needs to continue * Companies still spend big chunks of money on proprietary SW development, they need to understand the reasons and benefits to also invest in working on open source projects they rely on, such as OpenStack - Documented and visible processes * OpenStack project teams need to revisit their processes to make sure that the contributors are spending their time and efforts efficiently * When processes exist they also need to be discoverable by everyone, without the need to talk to an established contributor or maintainer - Code review is not the only area where maintainers need help. When an individual or comany, like an operator, has expertise in particular features or areas of the code, their input is needed and desired when changes are made to those parts of the code. - Tooling is not the solution to many of the challenges, so we need to focus on solving the social issues Next steps - Documentation and making existing processes visible, folks from the Nova team made a commitment to update their existing documentatoion to make it up to date and contain their current processes on how they do project management for the team * The Nova team also started to track bug fixes as part of their process to track changes and have the ability to prioritize, in recognition how that is an area where operators often get involved - Start small and replicate practices that works in the wider community * When project teams find ways that help their work, the community should consider exploring and adopting those techniques more widely For more detailed notes, please check the etherpad we used for the session: https://etherpad.opendev.org/p/r.914969755fedc5c6fa58d7e9881ffa66 Those who attended the Forum session, please chime in with any further thoughts and notes on anything that I missed to mention, and to continue the conversation. Those of you who did not have the opportunity to participate in the Forum session, what from the above summary and notes on the etherpad do you resonate with? Which experiences do you have as well and what solution ideas have you tried already? Best Regards, Ildikó ——— Ildikó Váncsa Director of Community Open Infrastructure Foundation
On Aug 29, 2024, at 4:18 AM, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2024-08-21 15:18:12 +0000 (+0000), Jeremy Stanley wrote: [...]
The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. [...]
Last week, Jens mentioned this article in IRC:
https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/code-review-antipatte...
Coincidentally published the same day I posted the first message in this thread, it's an amusing semi-satirical look at many of the same challenges our community has also identified. A quick summary of the author's points (though I strongly encourage you to check out the full article if you have time):
* Batch up review feedback and try not to "move goalposts" by repeatedly coming up with new problems in subsequent reviews that you could have found the first time through (i.e. don't just stop reviewing on the first issue you find).
* Don't demand a change author include unrelated fixes to nearby code (pointing it out as optional in case they want to tackle it if they have time is fine, just be clear it's not a requirement of approval).
* Try to reach a consensus of opinion with other reviewers on the same changes/project, and avoid jerking the author around by changing your own opinion on things.
* If a proposed implementation is unsuitable, be clear about what kind of alternative implementation would be suitable.
* Lead with the most important problems in a change, not the trivial ones.
* When reviewing individual changes that are part of a larger approved specification, try to focus on whether those changes implement the spec as written regardless of whether you agree with choices the project has made within the spec (if you disagree, get the spec changed or amended first).
* Extremes of opinion can always be found to reject a change, so look for a position of compromise when possible.
* Be cautious and deliberate about shifts of expectation over time, since authors are prone to replicate patterns from prior changes and existing code.
I think articles like this help to confirm we're on the right track, and I find it a reassuring reminder that these are challenges pretty much all communities struggle with. We're not alone. If we can come up with viable approaches for improving the situation, we'll be helping not just our own community, but open source software projects everywhere.
Also, a quick reminder: If you'll be in Suwon for OpenInfra Summit Asia or Indianapolis for OpenInfra Days North America, come to the forum session (Bridging the Gap: Community and Contributing Orgs). Whether or not you're available for the in-person sessions, replies to this mailing list thread can help refine the community's approach to contributor communications so we can start to get some of these things documented in relevant places (Contributor Guide, Project Team Guide, et cetera). -- Jeremy Stanley
A quick reminder: If you'll be in Indianapolis for OpenInfra Days North America next week, come to the "Bridging the gap between community and contributing organizations" Forum session at 11:20 AM EDT on Tuesday in room 305. If you can't make it, don't worry! Later this month, a cross-project PTG session is booked for 17:00 UTC on Tuesday (October 22) under a "bridging-the-gap" track name. Whether or not you're available for the above synchronous discussions, replies to this mailing list thread can help refine the community's approach to contributor communications so we can start to get some of these things documented in relevant places (Contributor Guide, Project Team Guide, et cetera). -- Jeremy Stanley
On 2024-10-10 00:37:30 +0000 (+0000), Jeremy Stanley wrote:
A quick reminder: If you'll be in Indianapolis for OpenInfra Days North America next week, come to the "Bridging the gap between community and contributing organizations" Forum session at 11:20 AM EDT on Tuesday in room 305.
The OID-NA session was a great success, with at least a dozen participants (I should have counted, sorry!). You can find the notes at https://etherpad.opendev.org/p/r.06f725e22f12d0e72639b5ac0c8bdac3 but with only 40 minutes we didn't get too deep into details. We talked a bit about how the list of findings and recommendations, and how they're categorized so far, resonated with people in the room. Some related personal experiences where they felt their proposed changes were not evaluated fairly or concerns taken seriously. There seemed to be consensus that perceived affiliation influences were likely to be merely statistical reality or at worst subconscious bias, and that communication challenges like those we've already highlighted are primarily emergent social behaviors, habits or basic human tendencies which are best addressed through education and documentation rather than technical means. There was a bit of debate about letting the perfect be the enemy of the good when it comes to the absence of suitable test environments for specialized hardware integration, and we talked some about the possibility or challenges of third-party testing vs including some research clouds into OpenDev's stable of available resources. Participants also saw value in having more clearly documented escalation paths for technical disagreements (core reviewer... team consensus... PTL... TC). These topics spilled over into an "OpenInfra leaders" session later, so https://etherpad.opendev.org/p/r.90cac8145c6e01bc19d7c02d4df6f8b3 is worth reviewing as well (around lines 16-23).
If you can't make it, don't worry! Later this month, a cross-project PTG session is booked for 17:00 UTC on Tuesday (October 22) under a "bridging-the-gap" track name. [...]
That's right, past me! I'm hoping we can use the session tomorrow to start divvying up writing tasks and identifying which documents might be the most appropriate locations for the amassed recommendations. Everyone's welcome, but people willing to volunteer to work on some of this are even more welcome. ;) In addition to tomorrow's PTG session on this topic, the TC devoted an hour to their role in the effort earlier today, notes begin around line 317 in https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b for those of you interested in an early recap. -- Jeremy Stanley
On 2024-10-21 23:57:09 +0000 (+0000), Jeremy Stanley wrote: [...]
I'm hoping we can use the session tomorrow to start divvying up writing tasks and identifying which documents might be the most appropriate locations for the amassed recommendations. Everyone's welcome, but people willing to volunteer to work on some of this are even more welcome. ;)
In addition to tomorrow's PTG session on this topic, the TC devoted an hour to their role in the effort earlier today [...]
Now that people are hopefully recovered and recharged from the PTG, and settling in for quieter times as the end of the year approaches, it's a good opportunity to take stock of where we are on this effort. First, for those who missed the TC session and haven't had a chance to peruse the notes yet, https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b#L321 is a fairly quick read. For an even quicker summary, we had a brief (re)introduction to the overarching topic and recap of past discussion series, as well as perceived next steps. Some specific ideas were also floated like better documentation on using Matrix to access OFTC IRC, expanding on the "escalation path" suggestion from OID-NA, scheduling project review days and sharing dashboards to reduce the number of changes going unnoticed, better gate stability as a means to improve contributor experience, revisiting how strict projects need to be about testing for hardware-specific features, and making reviewers/maintainers more visible so new contributors can find and talk to them. The dedicated working session used https://etherpad.opendev.org/p/bridging-the-gap-series-4 as a breakdown of previously discussed suggestions. Each item is templated out for additional notes as guidance for volunteers to elaborate on their intended meaning, and a best guess for which parts of what existing documents seem like the most appropriate locations to add or expand on these. We talked about the obvious documents (central contributor guide, per-project contributing docs, project team guide...), but also some non-traditional ideas like resurrecting the Gerrit "WelcomeBot" or canned welcome messages and direct outreach for reviewers. We only got through a couple of items in the series 4 pad, more or less to prove out the workflow, so this is where the rest of you come in! Please take a look at the pad and, if you see a suggestion which is of particular interest to you, please help expand on it, suggest a location (or multiple locations) where it can be documented for improved visibility, and volunteer to push changes! Also feel free to follow up here with any questions or ideas you have for helping move the effort forward to bridge these gaps. -- Jeremy Stanley
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... Anyone who has contributed changes to an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... Anyone who acted as a core reviewer for an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa... If you've done both, please fill in both surveys. If you contributed to or helped maintain deliverables in more than one OpenStack project team, please fill out the appropriate survey(s) once for each team (there is a drop-down selector for the team you're responding about/on behalf of). I know this is a lot to ask, but more data means more insightful analyses should be possible, so thanks for any time you can spare for it. Since not everyone pays close attention to openstack-discuss, it would be great if those of you who read this can bring it to the attention of your teammates and colleagues next week during PTG sessions as well. We're also planing to do some more targeted reminders to contributors as part of the 2025.1 Epoxy release communications, and to project maintainers either next week or immediately after the PTG, but if you've already submitted your responses then you can ignore those. The reason for the relatively short one-month deadline is that I and the other community managers are going to work on analyzing the results for publication to the community within the coming quarter, so hopefully we'll have good first impressions to share some time in June if enough of you help by submitting your experiences. Longer term, I expect we'll re-run these surveys immediately after each release cycle, so that we can start trying to identify trends in the data and determine whether the related Bridging the Gap efforts are making any noticeable impact. Thanks again to everyone for all your help in making OpenStack more welcoming and productive! -- Jeremy Stanley
On 2025-04-01 19:49:13 +0000 (+0000), Jeremy Stanley wrote:
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... [...]
Many thanks to Jens for pointing out in IRC that these are not entirely anonymous. The survey does require you to supply a name and E-mail address so that the foundation staff can reach out for clarification if needed on any free-form responses, and to help weed out bogus responses from bots or vandals. Your responses are treated confidentially, however, as only foundation staff will have access to the raw response data which will be thoroughly anonymized and aggregated before results are shared publicly with the community. The included contact information will only be used for purposes relating to the surveys, not for any other foundation activities. Please let me know if you have any questions, of course! -- Jeremy Stanley
Hi, I’m reaching out to you to give you an update on the surveys that the OpenInfra Foundation community managers have been running. Our goal has been to understand challenges that contributors and maintainers in OpenStack have in order to improve the experience for both. To protect the anonymity of the survey respondents we are sharing anonymized and aggregated survey results reflecting feedback we received throughout OpenStack. As a next step we will reach out to project teams who received more than one survey response, starting with the team(s) that received most, to share team-specific feedback and understand how we can help to address them. Below please find highlights of the two surveys: __MAINTAINER SURVEY__ We received 15 responses from 11 project teams, with 36% of teams receiving more than one response. We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Code review - Changes you propose are reviewed in a timely manner: 2.73 - Code review - You receive actionable feedback from other reviewers: 4.26 - Code review - Automated test failures quickly direct you to problems in your changes: 3.6 - Contributor docs - It is comprehensive to cover processes and good practices: 3.73 - Contributor docs - It is up to date: 3.53 53% of responders said they didn’t refer to their projects Contributor Guide during the Epoxy release cycle, while 20% referred to it 1-9 times and 13% 10-19 times as well as 13% 20+ times. The survey also asked respondents to mark which listed issues they face while trying to land their own changes, as well as trying to land other contributors’ changes. Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who faced each challenge: - Landing their own changes - Trouble getting review attention - 73% - Other - 33% - No changes pass CI - 13% - Need more assistance to proceed with review feedback - 6% - Expected to expand on review's scope - 6% - Disagreement or conflicting review comments - 6% - Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 66% - Change owner can't/won't add missing pieces to the change - 66% - Change owner's response is slow - 60% - Change is lower priority than others - 40% - Change is beyond expertise - 40% - Change broken, doesn't pass CI - 40% - Testing is broken for the project - 20% - Other - 13% - Change needs a BP/spec - 6% The below numbers are the percentages of project teams who faced each challenge: - Landing their own changes - Trouble getting review attention - 64% - Other - 36% - No changes pass CI - 18% - Need more assistance to proceed with review feedback - 9% - Expected to expand on review's scope - 9% - Disagreement or conflicting review comments - 9% - Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 82% - Change owner can't/won't add missing pieces to the change - 73% - Change owner's response is slow - 82% - Change is lower priority than others - 45% - Change is beyond expertise - 55% - Change broken, doesn't pass CI - 45% - Testing is broken for the project - 27% - Other - 18% - Change needs a BP/spec - 9% __CONTRIBUTOR SURVEY__ We received 28 responses from 15 project teams, with 53% of teams receiving more than one response. We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Contributor experience - Changes you propose are reviewed in a timely manner: 2.5 - Contributor experience - You receive actionable feedback from other reviewers: 4.25 - Contributor experience - Automated test failures quickly direct you to problems in your changes: 3.64 - Contributor documentation - You were able to find information about the processes the project team is using: 3.64 - Contributor documentation - It helped you to apply better practices throughout your contribution journey and achieve results faster: 3.57 - Contributor documentation - It is easy to discover: 3.32 - Contributor documentation - It is easy to navigate: 3.36 The survey also asked respondents to mark which listed issues they face while trying to land their own changes. Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who faced each challenge: - Trouble getting review attention - 68% - Other - 36% - Unable to determine priorities - 18% - Reviewers keep coming back with new requests - 11% - Need more clarification on feedback to move forward - 7% - Reviewer expectation or consensus shifts over time - 7% - Expected to expand scope to address other project issues - 7% - Reviewers disagree or give conflicting feedback - 4% - Asked to make changes deviating from past consensus - 4% - Reviewer forgets to un-block change - 4% - Unhelpful or incorrect reviews - 4% The below numbers are the percentages of project teams who faced each challenge: - Trouble getting review attention - 73% - Other - 47% - Unable to determine priorities - 20% - Reviewers keep coming back with new requests - 20% - Need more clarification on feedback to move forward - 13% - Reviewer expectation or consensus shifts over time - 13% - Expected to expand scope to address other project issues - 13% - Reviewers disagree or give conflicting feedback - 7% - Asked to make changes deviating from past consensus - 7% - Reviewer forgets to un-block change - 7% - Unhelpful or incorrect reviews - 7% While most challenges are affecting some or many projects, the biggest issues across the board seem to be getting attention from reviewers, and having delays on both the reviewer and change owner side. This is not necessarily surprising knowing that many or rather most contributors and maintainers are working on OpenStack part-time, or even outside of their day jobs. With the limited bandwidth that the community has, prioritizing work items and clear communication are more important than ever for the success and sustainability of the project. OpenInfra community managers will start reaching out to project teams with most responses and start working on addressing the challenges they face. As we progress with next steps we will also share practices with the OpenStack community at large to help improve contributor experience for everyone. While the official survey deadline has passed, we are still looking for feedback. If you missed filling out the survey(s), please provide your input now! ~ Please fill out this survey separately for every OpenStack project you contributed to during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... ~ Please fill out this survey separately for every OpenStack project you were a core reviewer of during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa... Thanks, Ildikó ——— Ildikó Váncsa Director of Community Open Infrastructure Foundation
On Apr 2, 2025, at 07:27, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2025-04-01 19:49:13 +0000 (+0000), Jeremy Stanley wrote:
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... [...]
Many thanks to Jens for pointing out in IRC that these are not entirely anonymous. The survey does require you to supply a name and E-mail address so that the foundation staff can reach out for clarification if needed on any free-form responses, and to help weed out bogus responses from bots or vandals.
Your responses are treated confidentially, however, as only foundation staff will have access to the raw response data which will be thoroughly anonymized and aggregated before results are shared publicly with the community. The included contact information will only be used for purposes relating to the surveys, not for any other foundation activities.
Please let me know if you have any questions, of course! -- Jeremy Stanley
Hi, I’m reaching out to you about the analysis that OpenInfra Foundation community managers have been running, to understand challenges that contributors and maintainers in OpenStack are facing in order to improve the experience for both. In addition to running surveys, which I shared the anonymized and aggregated results of in my last email[1], we also collected a selection of metrics to get an insight into the load and efficiency of OpenStack project teams. We pulled data from the Bitergia OpenStack dashboard[2] with a script[3]. We collected metrics for 13 project teams covering 5 release cycles from Antelope to Epoxy. As we’ve been collecting survey results and metrics at the same time, the project lists for the two efforts haven’t been synchronized yet. We picked the initial set of projects for the metrics analysis to get an overview from a combination of well-established and newer teams. As we progress further in the process we will match the data sets more closely. We collected the following metrics: - Review Efficiency Index (REI): Merged patches over Opened Patches - Median and Average Time to Review: These metrics focus on the time passed between the creation of a review and first time it gets reviewed by someone - Median and Average Time to Merge: These metrics show the time needed from the creation to the approval of a review - Median and Average Patchsets per Review: These metrics show the number of revisions a review has throughout its lifecycle - # of Changes Opened - # of Changes Closed - #/% of Changes Opened but not Closed in a Release Cycle - # of Maintainers: Anyone in a project team who voted with CR+2, CR-2 or W+1 in a release cycle in any of the project team’s repos. This may not be an exact match to people who are listed as core reviewers in the Gerrit core teams for the project team. - # of Reviewers: Anyone who commented or voted in any way on changes in a release cycle. The intention with building a baseline from a set of metrics is to be able to determine whether or not the current efforts are making any positive impact, or if we should adjust the approach we’ve been taking in any way. To avoid people making comparisons between project teams’ efficiency and performance, we only share anonymized and aggregated data, and provide a high-level picture to the community of what we’ve found so far. - REI: * Over the course of 5 release cycles, most teams had an REI slightly above 1, which means that more changes got closed (merged or abandoned) than opened. * Only the Bobcat release had an REI lower than one, the average across the 13 teams was 0.99. - Patches Opened and Closed * On average throughout the 5 release cycles teams got 412 new reviews, with the minimum of 64 and maximum of 873. * On average throughout the 5 release cycles teams closed 441 patches, with the minimum of 69 and maximum of 1012. * Collectively over the 5 release cycles, the 13 teams received a total of 5356 new changes and closed 5730 reviews. - Review backlog * On average throughout the 5 release cycles, teams had about 6% of new changes that didn’t get closed (merged or abandoned) throughout the release cycle, with the minimum of 1.17% and maximum of 15.8%. * Looking at release cycles, from Antelope to Dalmation the percentage of changes that didn’t close within the release cycle fluctuated between 5% and 8.2%, which jumped to 12.6% in Epoxy. * Of the 5356 new changes throughout the 5 release cycles, 331 took longer than a release cycle to close, or might still be open. - Merge Times * On average throughout the 5 release cycles it took teams 129 days to approve a review, with a median of 25 days. * The team with the lowest average needed 12 days, while their median settled at 5.7 days per cycle. * The team with the lowest median needed 5.2 days, while their average time window was 126 days. * The team with the highest average needed 247 days, while also grabbing the highest median number of 88 days. - Review Times * On average throughout the 5 release cycles it took teams 36 days to conduct the first review on new changes, with a median of 5 days. * The team with the lowest average needed 32 days, while their median settled at 0.9 days per cycle. * The team with the lowest median needed 0.7 days, while their average time window was 22 days. * The team with the highest average needed 88 days, while also grabbing the highest median number of 16 days. Throughout the survey responses challenges with review attention got the highest number of votes, and turned out to be a challenge in every project team who received responses. This is in line with the relatively long timeframe for first reviews shown by the above average numbers. Here I would like to note the difference between ‘average’ and ‘median’ numbers, where the latter shows the most common experience and is not influenced by the outlying datapoints the same way as 'average' is. The overall median review and merge times numbers suggest a relatively good overall experience for all teams across all 5 release cycles, however, the average numbers show areas for improvement. - # of Maintainers * On average teams had 11 active maintainers in the release cycles, with a team having as low as 5 and another team having as high as 21. - # of Reviewers * On average teams had 76 active reviewers in the release cycles, with a team with the lowest of 20 and another team as the highest of 148. There seems to be 7x as many reviewers as maintainers throughout the 13 project teams whose data we looked at. With accepting that not every reviewer is present throughout multiple release cycles, or even throughout a single one, there might be an opportunity to capture some of the more casual reviewers to further increase review bandwidth and nurture a group of new maintainers. The final analysis, with which we will be reaching out to project teams, is building on the combination of survey data and metrics. Please note that we do not have the bandwidth to reach out to every project team at once, as we would like to work with teams on strategy and next steps to address the challenges that we collectively uncover. In the following days, we will be starting to reach out to those teams who got the most survey responses and will expand our outreach as bandwidth allows. Thank you for your patience and understanding. Thanks, Ildikó [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://openstack.biterg.io/app/dashboards#/view/Overview [3] https://opendev.org/opendev/engagement/src/branch/master/tools/metrics/query...
On May 19, 2025, at 10:12, Ildiko Vancsa <ildiko.vancsa@gmail.com> wrote:
Hi,
I’m reaching out to you to give you an update on the surveys that the OpenInfra Foundation community managers have been running. Our goal has been to understand challenges that contributors and maintainers in OpenStack have in order to improve the experience for both.
To protect the anonymity of the survey respondents we are sharing anonymized and aggregated survey results reflecting feedback we received throughout OpenStack.
As a next step we will reach out to project teams who received more than one survey response, starting with the team(s) that received most, to share team-specific feedback and understand how we can help to address them.
Below please find highlights of the two surveys:
__MAINTAINER SURVEY__
We received 15 responses from 11 project teams, with 36% of teams receiving more than one response.
We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Code review - Changes you propose are reviewed in a timely manner: 2.73 - Code review - You receive actionable feedback from other reviewers: 4.26 - Code review - Automated test failures quickly direct you to problems in your changes: 3.6 - Contributor docs - It is comprehensive to cover processes and good practices: 3.73 - Contributor docs - It is up to date: 3.53
53% of responders said they didn’t refer to their projects Contributor Guide during the Epoxy release cycle, while 20% referred to it 1-9 times and 13% 10-19 times as well as 13% 20+ times.
The survey also asked respondents to mark which listed issues they face while trying to land their own changes, as well as trying to land other contributors’ changes. Please note that respondents were allowed to choose as many options as were applicable to them.
The below numbers are the percentages of respondents who faced each challenge:
- Landing their own changes - Trouble getting review attention - 73% - Other - 33% - No changes pass CI - 13% - Need more assistance to proceed with review feedback - 6% - Expected to expand on review's scope - 6% - Disagreement or conflicting review comments - 6%
- Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 66% - Change owner can't/won't add missing pieces to the change - 66% - Change owner's response is slow - 60% - Change is lower priority than others - 40% - Change is beyond expertise - 40% - Change broken, doesn't pass CI - 40% - Testing is broken for the project - 20% - Other - 13% - Change needs a BP/spec - 6%
The below numbers are the percentages of project teams who faced each challenge:
- Landing their own changes - Trouble getting review attention - 64% - Other - 36% - No changes pass CI - 18% - Need more assistance to proceed with review feedback - 9% - Expected to expand on review's scope - 9% - Disagreement or conflicting review comments - 9%
- Landing other people’s changes - Change owner/stakeholders are unreachable to discuss change - 82% - Change owner can't/won't add missing pieces to the change - 73% - Change owner's response is slow - 82% - Change is lower priority than others - 45% - Change is beyond expertise - 55% - Change broken, doesn't pass CI - 45% - Testing is broken for the project - 27% - Other - 18% - Change needs a BP/spec - 9%
__CONTRIBUTOR SURVEY__
We received 28 responses from 15 project teams, with 53% of teams receiving more than one response.
We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at: - Contributor experience - Changes you propose are reviewed in a timely manner: 2.5 - Contributor experience - You receive actionable feedback from other reviewers: 4.25 - Contributor experience - Automated test failures quickly direct you to problems in your changes: 3.64 - Contributor documentation - You were able to find information about the processes the project team is using: 3.64 - Contributor documentation - It helped you to apply better practices throughout your contribution journey and achieve results faster: 3.57 - Contributor documentation - It is easy to discover: 3.32 - Contributor documentation - It is easy to navigate: 3.36
The survey also asked respondents to mark which listed issues they face while trying to land their own changes. Please note that respondents were allowed to choose as many options as were applicable to them.
The below numbers are the percentages of respondents who faced each challenge: - Trouble getting review attention - 68% - Other - 36% - Unable to determine priorities - 18% - Reviewers keep coming back with new requests - 11% - Need more clarification on feedback to move forward - 7% - Reviewer expectation or consensus shifts over time - 7% - Expected to expand scope to address other project issues - 7% - Reviewers disagree or give conflicting feedback - 4% - Asked to make changes deviating from past consensus - 4% - Reviewer forgets to un-block change - 4% - Unhelpful or incorrect reviews - 4%
The below numbers are the percentages of project teams who faced each challenge: - Trouble getting review attention - 73% - Other - 47% - Unable to determine priorities - 20% - Reviewers keep coming back with new requests - 20% - Need more clarification on feedback to move forward - 13% - Reviewer expectation or consensus shifts over time - 13% - Expected to expand scope to address other project issues - 13% - Reviewers disagree or give conflicting feedback - 7% - Asked to make changes deviating from past consensus - 7% - Reviewer forgets to un-block change - 7% - Unhelpful or incorrect reviews - 7%
While most challenges are affecting some or many projects, the biggest issues across the board seem to be getting attention from reviewers, and having delays on both the reviewer and change owner side. This is not necessarily surprising knowing that many or rather most contributors and maintainers are working on OpenStack part-time, or even outside of their day jobs. With the limited bandwidth that the community has, prioritizing work items and clear communication are more important than ever for the success and sustainability of the project.
OpenInfra community managers will start reaching out to project teams with most responses and start working on addressing the challenges they face. As we progress with next steps we will also share practices with the OpenStack community at large to help improve contributor experience for everyone.
While the official survey deadline has passed, we are still looking for feedback. If you missed filling out the survey(s), please provide your input now!
~ Please fill out this survey separately for every OpenStack project you contributed to during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... ~ Please fill out this survey separately for every OpenStack project you were a core reviewer of during the Epoxy release: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa...
Thanks, Ildikó
———
Ildikó Váncsa Director of Community Open Infrastructure Foundation
On Apr 2, 2025, at 07:27, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2025-04-01 19:49:13 +0000 (+0000), Jeremy Stanley wrote:
As the next step in this ongoing effort to improve the contributor and maintainer experience, community managers on the OpenInfra Foundation staff worked together to convert the pain points identified in prior phases into a pair of very quick (2-3 minute) anonymous surveys... [...]
Many thanks to Jens for pointing out in IRC that these are not entirely anonymous. The survey does require you to supply a name and E-mail address so that the foundation staff can reach out for clarification if needed on any free-form responses, and to help weed out bogus responses from bots or vandals.
Your responses are treated confidentially, however, as only foundation staff will have access to the raw response data which will be thoroughly anonymized and aggregated before results are shared publicly with the community. The included contact information will only be used for purposes relating to the surveys, not for any other foundation activities.
Please let me know if you have any questions, of course! -- Jeremy Stanley
Just a quick reminder that the 2025.1/Epoxy cycle contributor and maintainer surveys are still open through April 30 (one week). If you haven't taken them yet, please complete the ones applicable to your participation! To reiterate... Anyone who has contributed changes to an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_contributor_satisf... Anyone who acted as a core reviewer for an OpenStack project during the 2025.1/Epoxy cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_satisfa... If you've done both, please fill in both surveys. If you contributed to or helped maintain deliverables in more than one OpenStack project team, please fill out the appropriate survey(s) once for each team (there is a drop-down selector for the team you're responding about/on behalf of). I know this is a lot to ask, but more data means more insightful analyses should be possible, so thanks for any time you can spare for it. Since not everyone pays close attention to openstack-discuss, it would be great if those of you who read this can bring it to the attention of your teammates and colleagues during meetings or other interactions. Thanks as always for your continued help in making OpenStack a more effective and welcoming community! -- Jeremy Stanley
It's that time again! The PTG is just around the corner, and a lot of teams I talked to over the course of the 2025.2/Flamingo cycle are planning to include improving the day-to-day experience of maintaining and contributing to OpenStack as a topic. Whether you're going to brainstorm new solutions, talk about what you've done, or work on implementing fixes you've already identified, let me know so I can try to join and help out. In addition to team-specific discussions, I and the other community managers on the OpenInfra Foundation staff are holding an office hour at 14:00 UTC on Thursday of the PTG (in the maint-contrib-xp track) where we can as a community collectively talk through pains and possible ways to address them. I've also injected a placeholder for this topic in the Technical Committee's PTG agenda, so that we can cover some OpenStack-wide commonalities and future tasks. On that note, the PTG is a prime opportunity to remind your colleagues and teammates about the Maintainer and Contributor Surveys (take note of their new URLs): https://openinfrafoundation.formstack.com/forms/openstack_maintainer_experie... https://openinfrafoundation.formstack.com/forms/openstack_contributor_experi... These have been updated for the 2025.2/Flamingo cycle retrospective, including new questions about review priorities. While we gleaned some useful insights from the 2025.1/Epoxy retrospective responses, there were far too few to draw deeper conclusions, so we'd love it if everyone would try to find some time for these as well as encouraging others in the community to do the same. For those of you coming to the OpenInfra Summit in France in just a couple of weeks, we've got a related Forum session booked too: "Improving Contributor Experience in the OpenStack Community" at 2:45 PM that Sunday in the Alfred Sauvy room. I hope to see some of you there, and for the rest we can catch up during the PTG! -- Jeremy Stanley
As we all finalize PTG agendas for the coming week, please take a moment to think about making some time for topics around improving the community experience for our maintainers and contributors. Whether that's prioritizing mentoring of prospective new maintainers, improving contributor documentation, or looking at tactics other teams have tried that made a positive impact, this is the ideal time for integrating these efforts into the direction we'll collectively set for the coming development cycle. In addition to team-specific discussions, this is a quick reminder that I and the other community managers on the OpenInfra Foundation staff are holding an office hour at 14:00 UTC on Thursday of the PTG (in the maint-contrib-xp track) where we can as a community collectively talk through pains and possible ways to address them. I've also injected a placeholder for this topic in the Technical Committee's PTG agenda, so that we can cover some OpenStack-wide commonalities and future tasks. And please don't forget to encourage your colleagues and teammates to take the 2025.2/Flamingo retrospective Maintainer and Contributor Surveys (take note of their new URLs): https://openinfrafoundation.formstack.com/forms/openstack_maintainer_experie... https://openinfrafoundation.formstack.com/forms/openstack_contributor_experi... Also, for those of you who couldn't attend the related Forum session we held at OpenInfra Summit Europe 2025, this is the read-only link for notes we took there: https://etherpad.opendev.org/p/r.3bbbf7c04b2697846f107775a592a082 Have a good weekend, and a productive PTG week! -- Jeremy Stanley
A quick recap for those who were interested in the PTG sessions but couldn't make it to all of them... We held an "office hours" session on improving the maintainer and contributor experience, notes here: https://etherpad.opendev.org/p/r.a23c66c4b9324895c8dc808a4ed1530b The Technical Committee made time to discuss these topics as well, and their notes can be found under "Bridging the gap between community and contributing orgs" in: https://etherpad.opendev.org/p/r.5dfdc57d7bc31731ccf9d51cb80de6c5 https://www.youtube.com/watch?v=p-Aq39DEHWk (skip to 20:00) Part of the Ops Radio Hour during the PTG was dedicated to discussing aspects of this too, under "Improving contributors experience - Getting more Ops involved edition" in: https://etherpad.opendev.org/p/r.89fd2d8f343c8edb136d8a8b2f2efdc9 https://www.youtube.com/watch?v=NlUnAde6Xi0 (skip to 23:00) And the Horizon team spent a while talking about "Contributor/maintainer surveys and metrics analysis" mentioned in: https://etherpad.opendev.org/p/r.bf13ee1b33a14fcfee5ee216d56f7f14 https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... If this or related topics came up in other PTG sessions, please feel free to reply with links! -- Jeremy Stanley
Just a quick reminder that the 2025.2/Flamingo cycle contributor and maintainer surveys are still open through November 16 (this weekend). If you haven't taken them yet, please complete the ones applicable to your participation! To reiterate... Anyone who acted as a core reviewer for an OpenStack project during the 2025.2/Flamingo cycle is encouraged to complete this survey no later than November 16: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_experie... Anyone who has contributed changes to an OpenStack project during the 2025.2/Flamingo cycle is encouraged to complete this survey no later than November 16: https://openinfrafoundation.formstack.com/forms/openstack_contributor_experi... Since not everyone pays close attention to openstack-discuss, it would be great if those of you who read this can bring it to the attention of your teammates and colleagues during meetings or other interactions. Thanks as always for your continued help in making OpenStack a more effective and welcoming community! -- Jeremy Stanley
Just a quick reminder that the 2026.1/Gazpacho cycle contributor and maintainer surveys are still open through April 30 (one week). If you haven't taken them yet, please complete the ones applicable to your participation! To reiterate... Anyone who has contributed changes to an OpenStack project during the 2026.1/Gazpacho cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_contributor_experi... Anyone who acted as a core reviewer for an OpenStack project during the 2026.1/Gazpacho cycle is encouraged to complete this survey no later than April 30: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_experie... For the projects where you are a maintainer, you only need to fill out the maintainer survey for that project team. Since not everyone pays close attention to openstack-discuss, it would be great if those of you who read this can bring it to the attention of your teammates and colleagues during meetings or other interactions (like next week's PTG sessions). Thanks as always for your continued help in making OpenStack a more effective and welcoming community! -- Jeremy Stanley
Hi, I’m reaching out to you to give you an update on the second round of surveys that the OpenInfra Foundation community managers have been running. Our goal has been to understand challenges that contributors and maintainers in OpenStack have in order to improve the experience for both. To protect the anonymity of the survey respondents we are sharing anonymized and aggregated survey results reflecting feedback we received throughout OpenStack. As a next step we will reach out to project teams who received more than one survey response, starting with the team(s) that received most, to share team-specific feedback and understand how we can help to address them. Highlights in trends comparing the first [1] and second rounds of survey analysis: * Maintainer survey: + In the second round, we got 40% fewer responses, with 36% fewer teams participating. + 63% of the Flamingo OpenStack Maintainer Experience survey respondents filled out the Epoxy maintainer survey as well. + Overall, Flamingo maintainer survey respondents rated their experience more positive than the overall experience shared for the Epoxy release cycle. + Getting review attention is still the biggest challenge + In the second round we got responses from maintainers who don’t contribute code themselves + Maintainers had growing challenges with finding time to review non-priority changes, changes that were submitted without blueprints or specs and finding time for in-depth reviews + While maintainers still had challenges with change owners becoming unresponsive, it was much less common than it was reported for Epoxy * Contributor survey: + Results to the Flamingo OpenStack Contributor Experience survey are mostly comparable with results from the Epoxy contributor survey. + The biggest difference was the mention of broken CI. + While review attention got fewer mentions, it’s still the biggest issue contributors face. + Issues with review feedback quality, shallow initial reviews as well as overly nit picky feedback, also came up more in recent survey results compared to the previous round. Below please find the broader analysis of the two surveys: __MAINTAINER SURVEY__ We received 9 responses from 7 project teams, with only 1 team receiving more than one response. We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at the following: - Code review - Changes you propose are reviewed in a timely manner: 4 - Code review - You receive actionable feedback from other reviewers: 4.44 - Code review - Automated test failures quickly direct you to problems in your changes: 4.33 - Contributor docs - It is comprehensive to cover processes and good practices: 4.33 - Contributor docs - It is up to date: 4 The survey question that related to how much maintainers are relying on pointing to the contributor documentation didn’t display options correctly in this round, and therefore we excluded it from the analysis. We added a new question to find out what project teams are using to track their priorities.Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who relied on each option: - Gerrit change topics - 100% - Reporting in team meetings - 100% - Reporting in IRC or on mailing lists - 89% - Etherpad or similar collaborative document - 67% - Custom review dashboards or queries - 67% - Specification documents - 56% - Review-Priority or similar Gerrit votes - 44% - Launchpad Blueprints - 44% - Gerrit hashtags - 33% - Kanban lists - 11% The below numbers are the percentages of project teams who relied on each option: - Gerrit change topics - 100% - Reporting in team meetings - 100% - Reporting in IRC or on mailing lists - 86% - Etherpad or similar collaborative document - 86% - Custom review dashboards or queries - 86% - Specification documents - 57% - Review-Priority or similar Gerrit votes - 57% - Launchpad Blueprints - 43% - Gerrit hashtags - 29% - Kanban lists - 14% There is a small difference between the tools that were listed by maintainers and the overall reported tool usage for project teams. This can imply that maintainers of the same team are unaware of tools being used by other maintainers, and a lack of team consensus on tools and methods to use. The survey also asked respondents to mark which listed issues they face while trying to land their own changes, as well as trying to land other contributors’ changes. Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who faced each challenge: - Landing their own changes - Trouble getting review attention - 56% - Other - 22%: + Maintainers didn’t contribute code: 11% + No issues: 11% People listed 4 additional challenges in the Epoxy round. - Landing other people’s changes - Change is lower priority than others - 56% - Change owner/stakeholders are unreachable to discuss change - 56% - Change is beyond expertise - 44% - Change owner can't/won't add missing pieces to the change - 44% - Change broken, doesn't pass CI - 44% - Change owner's response is slow - 33% - Change needs a BP/spec - 22% - Other (Lack of time to review changes in depth) - 22% - The change isn't interesting to me - 11% The below numbers are the percentages of project teams who faced each challenge: - Landing their own changes - I have trouble getting the attention of reviewers for my change - 57% - Other - 28% + Maintainer did not contribute code or docs - 14% + No issues - 14% - Landing other people’s changes: - Change is lower priority than others - 71% - Change owner/stakeholders are unreachable to discuss change - 57% - Change is beyond expertise - 57% - Change owner can't/won't add missing pieces to the change - 43% - Change broken, doesn't pass CI - 43% - Change owner's response is slow - 43% - Change needs a BP/spec - 29% - Other (Lack of time to review changes in depth) - 29% - The change isn't interesting to me - 14% The main challenge for maintainers in the Flamingo cycle was still getting review attention when contributing themselves, but this time around they didn’t report other issues. When trying to land other people’s changes, individuals faced the biggest challenge with review priorities and change owners becoming unreachable, while on the team level broken CI, slow change owner responses and owners not being able to fix issues in their changes were also among the most common issues. Issues which were mentioned more often in the Flamingo survey responses included changes being low priority, maintainers not having the expertise and changes needing a blueprint or spec. __CONTRIBUTOR SURVEY__ We received 31 responses from 17 project teams, with 29% of these teams receiving more than one response. 52% of respondents were new contributors in the Flamingo release cycle, which is twice as many as in the Epoxy survey responses. We asked respondents to rate the below statements between 1 (very bad) and 5 (excellent), which averaged at the following: - Contributor experience - Changes you propose are reviewed in a timely manner: 3.52 - Contributor experience - You receive actionable feedback from other reviewers: 4.39 - Contributor experience - Automated test failures quickly direct you to problems in your changes: 3.9 - Contributor documentation - You were able to find information about the processes the project team is using: 3.97 - Contributor documentation - It helped you to apply better practices throughout your contribution journey and achieve results faster: 4.16 - Contributor documentation - It is easy to discover: 3.68 - Contributor documentation - It is easy to navigate: 3.68 We added a new question to find out what project teams are using to track their priorities. Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who relied on each option: - Checking in IRC or on mailing lists - 54.84% - Checking Launchpad Blueprints - 48.39% - Checking team meeting discussions or logs - 45.16% - Checking Gerrit change topics - 41.94% - Checking specification documents - 32.26% - Checking Review-Priority or similar Gerrit votes - 25.81% - Checking Gerrit hashtags - 22.58% - I don't know how the team is tracking priorities - 16.13% - Checking StoryBoard Stories - 12.90% - Checking team's tracking etherpad or similar collaborative document - 12.90% - I'm not sure if the team is tracking priorities - 9.68% - Checking Custom review dashboards or queries - 9.68% - Other - 9.68% - Checking Kanban lists - 0% The below numbers are the percentages of project teams who relied on each option: - Checking in IRC or on mailing lists - 64.71% - Checking Launchpad Blueprints - 47.06% - Checking team meeting discussions or logs - 58.82% - Checking Gerrit change topics - 47.06% - Checking specification documents - 35.29% - Checking Review-Priority or similar Gerrit votes - 47.06% - Checking Gerrit hashtags - 35.29% - I don't know how the team is tracking priorities - 23.53% - Checking StoryBoard Stories - 23.53% - Checking team's tracking etherpad or similar collaborative document - 17.65% - I'm not sure if the team is tracking priorities - 17.65% - Checking Custom review dashboards or queries - 17.65% - Other - 17.65% - Checking Kanban lists - 0% Inconsistencies in team priority tracking can indicate a disconnect between where maintainers define, highlight and follow team priorities versus where contributors look for them. The biggest disconnect seems to be Gerrit change topics and team meetings, which all maintainers seem to rely on, while only half of the contributors use or follow. The survey also asked respondents to mark which listed issues they face while trying to land their own changes. Please note that respondents were allowed to choose as many options as were applicable to them. The below numbers are the percentages of respondents who faced each challenge: - Trouble getting review attention - 48% - Other - 45% - Unable to determine priorities - 13% - Reviewers keep coming back with new requests - 11% - Need more clarification on feedback to move forward - 10% - Reviewer expectation or consensus shifts over time - 6% - Expected to expand scope to address other project issues - 13% - Reviewers disagree or give conflicting feedback - 3% - Asked to make changes deviating from past consensus - 6% - Change reviewed too thoroughly - 3% - Shallow initial reviews - 6% - No changes pass CI - 19% The below numbers are the percentages of project teams who faced each challenge: - Trouble getting review attention - 64% - Other - 59% - Unable to determine priorities - 18% - Reviewers keep coming back with new requests - 11% - Need more clarification on feedback to move forward - 6% - Reviewer expectation or consensus shifts over time - 6% - Expected to expand scope to address other project issues - 18% - Reviewers disagree or give conflicting feedback - 6% - Asked to make changes deviating from past consensus - 6% - Change reviewed too thoroughly - 6% - Shallow initial reviews - 12% - No changes pas CI - 29% While the official survey deadline has passed, we are still looking for feedback. If you missed filling out the survey(s), please provide your input now! ~ Please fill out this survey separately for every OpenStack project you contributed to during the Flamingo release: https://openinfrafoundation.formstack.com/forms/openstack_contributor_experi... ~ Please fill out this survey separately for every OpenStack project you were a core reviewer of during the Flamingo release: https://openinfrafoundation.formstack.com/forms/openstack_maintainer_experie... Thanks, Ildikó [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... ——— Ildikó Váncsa Director of Community OpenInfra Foundation
Hi, I’m reaching out to you about the analysis that OpenInfra Foundation community managers have been running, to understand challenges that contributors and maintainers in OpenStack are facing in order to improve the experience for both. In addition to running surveys, which I shared the anonymized and aggregated results of in my last email[1], similarly to the previous round[2], we also collected a selection of metrics to get an insight into the load and efficiency of OpenStack project teams. We collected metrics, with the same method as previously, for 23 project teams, compared to the 13 teams that served as an initial set in the first round, covering 5 release cycles from Bobcat to Flamingo. This time around we included all the teams who received survey responses in either this or the previous round. We collected the following metrics: - Review Efficiency Index (REI): A ratio of Merged patches over Opened Patches - Median and Average Time to Review: These metrics focus on the time passed between the creation of a review and first time it gets reviewed by someone - Median and Average Time to Merge: These metrics show the time needed from the creation to the approval and merging of a review - Median and Average Patchsets per Review: These metrics show the number of revisions a review has throughout its lifecycle - # of Changes Opened - # of Changes Closed - # of Active Maintainers: Anyone in a project team who voted with CR+2, CR-2 or W+1 in a release cycle in any of the project team’s repos. This may not be an exact match to people who are listed as core reviewers in the Gerrit core teams for the project team. - # of Reviewers: Anyone who commented or voted in any way on changes in a release cycle. We excluded the ‘#/% of Changes Opened but not Closed in a Release Cycle’ metric in this round. Highlights in trends comparing the two rounds of metrics analysis: * The average number of incoming changes per team dropped by 23% + With more teams included in the data set, the minimum number of incoming changes dropped to half, while the maximum number remained close to the same. This is due to adding teams to the data set who are less active. + Teams were still able to close changes by a similar, slightly higher rate * The average number of reviewers per team dropped by 24% + The average number of maintainers also dropped by 19% + The ratio between the number of maintainers and reviewers dropped by 9% * The review and merge time metrics are comparable with the previous round’s results To avoid divisive comparisons between project teams' efficiency and performance, we only share anonymized and aggregated data, and provide a high-level picture to the community of what we’ve found so far. - REI: * Over the course of 5 release cycles, most teams had an REI slightly above 1, which means that more changes got closed (merged or abandoned) than opened. * Still only the Bobcat release had an REI lower than one, the average across the 23 teams was 1.09 - Patches Opened and Closed * On average throughout the 5 release cycles teams got 319 new changes, with the minimum of 33 and maximum of 902. * On average throughout the 5 release cycles teams closed 343 changes, with the minimum of 32 and maximum of 952. * Collectively over the 5 release cycles, the 23 teams received a total of 7352 new changes and closed 7889. - Merge Times * On average throughout the 5 release cycles it took teams 125 days to approve a review, with a median of 24 days. * The team with the lowest average needed 14 days, while their median settled at 1.6 days per cycle, which was also the lowest across all teams. * The team with the highest average needed 240 days, while also grabbing the highest median number of 84 days. - Review Times * On average throughout the 5 release cycles it took teams 38 days to conduct the first review on new changes, with a median of 6 days. * The team with the lowest average needed 2.5 days, while their median settled at 0.45 days per cycle, which was also the lowest across all teams. * The team with the highest average needed 96 days, while their median settled at 9 days. * The team with the highest median needed 35 days, while their average settled at 85 days. Throughout the survey responses, challenges with review attention still got the highest number of votes, and turned out to be a challenge in most project teams who received responses. This was the tendency shown in our analysis following the Epoxy release cycle. The overall median and average numbers between the two rounds are almost the same, even though the per averages show more significant differences. Here I would like to note the difference between ‘average’ (a.k.a. ‘mean’) and ‘median’ numbers, where the latter shows the most common experience and is not influenced by the outlying data points the same way as average is. In that regard, averages may provide a better indication of perceived performance while medians are closer to actual day-to-day reality. The overall median review and merge times numbers suggest a relatively good overall experience for all teams across all 5 release cycles, however, the average numbers show areas for improvement. - # of Active Maintainers * On average teams had 9 active maintainers in the release cycles, with a team having as low as 3 and another team having as high as 20 - # of Reviewers * On average teams had 58 active reviewers in the release cycles, with a team with the lowest of 17 and another team as the highest of 146 There is a slight decrease in the number of reviewers on average compared to the previous round of metrics analysis. There still seems to be 6.4x as many reviewers as maintainers throughout the 23 project teams whose data we looked at. With accepting that not every reviewer is active throughout multiple release cycles, or even throughout a single one, the opportunity still seems to be present to capture some of the more casual reviewers to further increase review bandwidth and nurture a group of new maintainers. The final analysis, with which we will be reaching out to project teams, is building on the combination of survey data and metrics. Please note that we do not have the bandwidth to reach out to every project team at once, as we would like to work with teams on a strategy and next steps to address the challenges that we collectively uncover. In the following days, we will be starting to reach out to those teams who got the most survey responses and will expand our outreach as bandwidth allows. Thank you for your patience and understanding. Thanks, Ildikó [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... ——— Ildikó Váncsa Director of Community OpenInfra Foundation
participants (2)
-
Ildiko Vancsa
-
Jeremy Stanley