[tc][stable] Changing stable branch policy
Hi everyone, At the PTG, the TC discussed what we can do about our stable branch policy and there was a few different ideas put on the table, however, something that I felt made a lot of sense was revisiting the way that we currently apply it. We all know that we're definitely a lot more resource limited as a community and it's important for us to start breaking down some of those ideas which made sense when the velocity of the project was very high. One of the things that used to make sense is maintaining a dedicated stable core team across all projects. At the current time: 1. Some projects seem to have some sort of power of their stable branches through historical reasons 2. Some projects don't have access to merging things into stable branches and need to rely on the stable maintenance team to do that 3. We are *really* thankful for our current stable team, but it seems that there is a lot of work that does bottleneck other teams (and really, stable reviews is a difficult task). The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects). I'd like to invite the community to comment on this change, the approach that we can take to do this (or other ideas) -- being mindful with the limited set of resources that we have inside the community. Thanks, Mohammed
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core. [1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members -- Thanks, Matt
On 11/18/19 4:08 PM, Matt Riedemann wrote:
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core.
I believe the proposal is to allow the Nova team to manage nova-stable-maint in the same way they do nova-core, not to force anyone to drop their stable-maint team entirely.
[1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members
On 18/11/19 5:35 pm, Ben Nemec wrote:
On 11/18/19 4:08 PM, Matt Riedemann wrote:
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core.
I believe the proposal is to allow the Nova team to manage nova-stable-maint in the same way they do nova-core, not to force anyone to drop their stable-maint team entirely.
I think the proposal was actually for each *-stable-maint team to manage itself. This would avoid the situation where e.g. the TC appoints a brand-new PTL and suddenly they get to make themselves a stable core, as in that case the team would still have to be bootstrapped by the stable-maint team. But it would allow those who are both closest to the project and confirmed to be familiar with the stable guidelines to make decisions about who else is ready to join that group. - ZB
[1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members
---- On Mon, 18 Nov 2019 19:17:04 -0600 Zane Bitter <zbitter@redhat.com> wrote ----
On 18/11/19 5:35 pm, Ben Nemec wrote:
On 11/18/19 4:08 PM, Matt Riedemann wrote:
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core.
I believe the proposal is to allow the Nova team to manage nova-stable-maint in the same way they do nova-core, not to force anyone to drop their stable-maint team entirely.
I think the proposal was actually for each *-stable-maint team to manage itself. This would avoid the situation where e.g. the TC appoints a brand-new PTL and suddenly they get to make themselves a stable core, as in that case the team would still have to be bootstrapped by the stable-maint team. But it would allow those who are both closest to the project and confirmed to be familiar with the stable guidelines to make decisions about who else is ready to join that group.
I am still finding difficult to understand the change and how it will solve the current problem. The current problem is: * Fewer contributors in the stable-maintenance team (core stable team and project side stable team) which is nothing but we have fewer contributors who understand the stable policies. * The stable policies are not the problem so we will stick with current stable policies across all the projects. Stable policies have to be maintained at single place for consistency in backports across projects. If we are moving the stable maintenance team ownership from current stable-maintenance team to project side then, how it will solve the issue, does it enable more contributors to understand the stable policy and extend the team? if yes, then why it cannot happen with current model? If the project team or PTL making its core member get more familiar with the stable policy and add as a stable core team then why it cannot happen with the current model. For example, if I am PTL or core of any project and finding hard to get my backport merged then I or my project team core should review more stable branch patches and propose them in stable team core. If we move the stable team ownership to the projects side then I think PTL is going to do the same. Ask the team members to understand the stable policies and do more review and then add them in stable core team. If any member know the stable policies then directly add. I feel that the current problem cannot be solved by moving the ownership of the team, we need to encourage more and more developers to become stable core in existing model especially from projects who find difficulties in merging their backport. One more thing, do we have data that how much time as avg it take to merge the backport and what all projects facing the backport merge issue ? -gmann
- ZB
[1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members
Ghanshyam Mann wrote:
[...] I am still finding difficult to understand the change and how it will solve the current problem.
The current problem is: * Fewer contributors in the stable-maintenance team (core stable team and project side stable team) which is nothing but we have fewer contributors who understand the stable policies.
* The stable policies are not the problem so we will stick with current stable policies across all the projects. Stable policies have to be maintained at single place for consistency in backports across projects. [...] I don't think that this the problem this change wants to solve.
Currently the stable-core team is perceived as a bottleneck to getting more people into project-specific stable teams, or keeping those teams membership up to date. As a result stable maintenance is still seen in some teams as an alien thing, rather than an integral team duty. I suspect that by getting out of the badge-granting game, stable-core could focus more on stable policy definition and education, and review how well or bad each team does on the stable front. Because reviewing backports for stable branch suitability is just one part of doing stable branch right -- the other is to actively backport relevant patches. Personally, the main reason I support this change is that we have too much "ask for permission" things in OpenStack today, something that was driven by a code-review-for-everything culture. So the more we can remove the need to ask for permission to do some work, the better. -- Thierry Carrez (ttx)
On Tue, Nov 19, 2019 at 6:16 AM Thierry Carrez <thierry@openstack.org> wrote:
Ghanshyam Mann wrote:
[...] I am still finding difficult to understand the change and how it will solve the current problem.
The current problem is: * Fewer contributors in the stable-maintenance team (core stable team and project side stable team) which is nothing but we have fewer contributors who understand the stable policies.
* The stable policies are not the problem so we will stick with current stable policies across all the projects. Stable policies have to be maintained at single place for consistency in backports across projects. [...] I don't think that this the problem this change wants to solve.
Currently the stable-core team is perceived as a bottleneck to getting more people into project-specific stable teams, or keeping those teams membership up to date. As a result stable maintenance is still seen in some teams as an alien thing, rather than an integral team duty.
I suspect that by getting out of the badge-granting game, stable-core could focus more on stable policy definition and education, and review how well or bad each team does on the stable front. Because reviewing backports for stable branch suitability is just one part of doing stable branch right -- the other is to actively backport relevant patches.
Personally, the main reason I support this change is that we have too much "ask for permission" things in OpenStack today, something that was driven by a code-review-for-everything culture. So the more we can remove the need to ask for permission to do some work, the better.
For context, I thought I'd gather my thoughts to explain the idea best and woke up to this well summarized email by Thierry. I agree with this and the intention is indeed what Thierry is mentioning here.
-- Thierry Carrez (ttx)
As the person who has asked for stable branch merge permission before, I felt the pain and I totally agree with mnaser's proposal. - Best regards, Lingxian Kong Catalyst Cloud On Wed, Nov 20, 2019 at 7:03 AM Mohammed Naser <mnaser@vexxhost.com> wrote:
On Tue, Nov 19, 2019 at 6:16 AM Thierry Carrez <thierry@openstack.org> wrote:
Ghanshyam Mann wrote:
[...] I am still finding difficult to understand the change and how it will
solve the current problem.
The current problem is: * Fewer contributors in the stable-maintenance team (core stable team
and project side stable team)
which is nothing but we have fewer contributors who understand the stable policies.
* The stable policies are not the problem so we will stick with current stable policies across all the projects. Stable policies have to be maintained at single place for consistency in backports across projects. [...] I don't think that this the problem this change wants to solve.
Currently the stable-core team is perceived as a bottleneck to getting more people into project-specific stable teams, or keeping those teams membership up to date. As a result stable maintenance is still seen in some teams as an alien thing, rather than an integral team duty.
I suspect that by getting out of the badge-granting game, stable-core could focus more on stable policy definition and education, and review how well or bad each team does on the stable front. Because reviewing backports for stable branch suitability is just one part of doing stable branch right -- the other is to actively backport relevant patches.
Personally, the main reason I support this change is that we have too much "ask for permission" things in OpenStack today, something that was driven by a code-review-for-everything culture. So the more we can remove the need to ask for permission to do some work, the better.
For context, I thought I'd gather my thoughts to explain the idea best and woke up to this well summarized email by Thierry. I agree with this and the intention is indeed what Thierry is mentioning here.
-- Thierry Carrez (ttx)
---- On Tue, 19 Nov 2019 11:58:12 -0600 Mohammed Naser <mnaser@vexxhost.com> wrote ----
On Tue, Nov 19, 2019 at 6:16 AM Thierry Carrez <thierry@openstack.org> wrote:
Ghanshyam Mann wrote:
[...] I am still finding difficult to understand the change and how it will solve the current problem.
The current problem is: * Fewer contributors in the stable-maintenance team (core stable team and project side stable team) which is nothing but we have fewer contributors who understand the stable policies.
* The stable policies are not the problem so we will stick with current stable policies across all the projects. Stable policies have to be maintained at single place for consistency in backports across projects. [...] I don't think that this the problem this change wants to solve.
Currently the stable-core team is perceived as a bottleneck to getting more people into project-specific stable teams, or keeping those teams membership up to date. As a result stable maintenance is still seen in some teams as an alien thing, rather than an integral team duty.
I suspect that by getting out of the badge-granting game, stable-core could focus more on stable policy definition and education, and review how well or bad each team does on the stable front. Because reviewing backports for stable branch suitability is just one part of doing stable branch right -- the other is to actively backport relevant patches.
Personally, the main reason I support this change is that we have too much "ask for permission" things in OpenStack today, something that was driven by a code-review-for-everything culture. So the more we can remove the need to ask for permission to do some work, the better.
For context, I thought I'd gather my thoughts to explain the idea best and woke up to this well summarized email by Thierry. I agree with this and the intention is indeed what Thierry is mentioning here.
I can understand your point and for some area I agree but not for stable policies case :). IMO, "ask for permission" is good when the stability of the software come into pic. For any proprietary software development we have a lot of "ask for permission" from the various team like QA, Requirement verification, Audit, patching up the fixes on production. Those are mainly to maintain the quality and stability of the software. If we give all the power of deciding all these things to a developer then you can imagine the situation. I know we have to trust the developer in Open Source to maintain all these areas in their code but that could be easy if OpenStack could have been a single software project. Because OpenStack is ~50 projects, our main challenge is to maintain the consistency among them via centric team to enforce and verify the key area like stability etc. It is not easy to maintain a centric team especially when OpenStack facing the less contributors issue. but as long as stable policies things are completely broken I think we should not change it. -gmann
-- Thierry Carrez (ttx)
On 18/11/19 9:18 pm, Ghanshyam Mann wrote:
---- On Mon, 18 Nov 2019 19:17:04 -0600 Zane Bitter <zbitter@redhat.com> wrote ----
On 18/11/19 5:35 pm, Ben Nemec wrote:
On 11/18/19 4:08 PM, Matt Riedemann wrote:
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core.
I believe the proposal is to allow the Nova team to manage nova-stable-maint in the same way they do nova-core, not to force anyone to drop their stable-maint team entirely.
I think the proposal was actually for each *-stable-maint team to manage itself. This would avoid the situation where e.g. the TC appoints a brand-new PTL and suddenly they get to make themselves a stable core, as in that case the team would still have to be bootstrapped by the stable-maint team. But it would allow those who are both closest to the project and confirmed to be familiar with the stable guidelines to make decisions about who else is ready to join that group.
I am still finding difficult to understand the change and how it will solve the current problem.
The current problem is: * Fewer contributors in the stable-maintenance team (core stable team and project side stable team) which is nothing but we have fewer contributors who understand the stable policies.
* The stable policies are not the problem so we will stick with current stable policies across all the projects. Stable policies have to be maintained at single place for consistency in backports across projects.
If we are moving the stable maintenance team ownership from current stable-maintenance team to project side then, how it will solve the issue, does it enable more contributors to understand the stable policy and extend the team?
Yes.
if yes, then why it cannot happen with current model?
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
If the project team or PTL making its core member get more familiar with the stable policy and add as a stable core team then why it cannot happen with the current model.
For example, if I am PTL or core of any project and finding hard to get my backport merged then I or my project team core should review more stable branch patches and propose them in stable team core.
I have tried that with only very limited success.
If we move the stable team ownership to the projects side then I think PTL is going to do the same. Ask the team members to understand the stable policies and do more review and then add them in stable core team. If any member know the stable policies then directly add.
You make it sound like that's not a good thing?
I feel that the current problem cannot be solved by moving the ownership of the team, we need to encourage more and more developers to become stable core in existing model especially from projects who find difficulties in merging their backport.
In my experience at least there's no shortage of people willing to do the work, but there is a severe shortage of people willing to do the work of climbing over the bar to get permission to do the work. The position espoused by Tony and Matt at least is that those shouldn't be different things, and in principle that's correct, but in practice they are. Humans are weird.
One more thing, do we have data that how much time as avg it take to merge the backport and what all projects facing the backport merge issue ?
-gmann
- ZB
[1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before". With Tony more removed these days and I myself not wanting to vet every one of these "add me to the stable core team" requests, I'm more or less OK with the proposal so that it removes me as a bottleneck. That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it. If it's a problem hopefully they'll hear about it from their consumers, but if the project is in such maintenance mode anyway that they can break the stable guidelines, then they might not have many external consumers to complain anyway. Either way I don't need to be involved. So sure, +1 from me on the proposal given nova can still do what it's already been doing with a specific stable maint core team ACL in Gerrit. -- Thanks, Matt
Hi Matt, For your kind information, I never asked to be to the core of stable-maintainer. Someone has recommended my name for it. So if you find it hard ir against the stable core policy, kindly remove me from this list. I will earn it with my efforts. Thank you for your kind support. Abhishek Kekane On Wed, 20 Nov 2019 at 8:00 PM, Matt Riedemann <mriedemos@gmail.com> wrote:
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
With Tony more removed these days and I myself not wanting to vet every one of these "add me to the stable core team" requests, I'm more or less OK with the proposal so that it removes me as a bottleneck. That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it. If it's a problem hopefully they'll hear about it from their consumers, but if the project is in such maintenance mode anyway that they can break the stable guidelines, then they might not have many external consumers to complain anyway. Either way I don't need to be involved.
So sure, +1 from me on the proposal given nova can still do what it's already been doing with a specific stable maint core team ACL in Gerrit.
--
Thanks,
Matt
-- Thanks & Best Regards,
Abhishek Kekane
On Wed, Nov 20, 2019 at 9:43 AM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Matt,
For your kind information, I never asked to be to the core of stable-maintainer. Someone has recommended my name for it. So if you find it hard ir against the stable core policy, kindly remove me from this list. I will earn it with my efforts.
Just for context, I don't think Matt was actually targeting you, it's actually something that has been raised a few times in the past with many other projects (i.e. Trove is one another recent memory). I'm happy to see you on the stable team for Glance. . :)
Thank you for your kind support.
Abhishek Kekane
On Wed, 20 Nov 2019 at 8:00 PM, Matt Riedemann <mriedemos@gmail.com> wrote:
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
With Tony more removed these days and I myself not wanting to vet every one of these "add me to the stable core team" requests, I'm more or less OK with the proposal so that it removes me as a bottleneck. That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it. If it's a problem hopefully they'll hear about it from their consumers, but if the project is in such maintenance mode anyway that they can break the stable guidelines, then they might not have many external consumers to complain anyway. Either way I don't need to be involved.
So sure, +1 from me on the proposal given nova can still do what it's already been doing with a specific stable maint core team ACL in Gerrit.
--
Thanks,
Matt
-- Thanks & Best Regards,
Abhishek Kekane
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. https://vexxhost.com
Hi Mohammed, My sincere apologies, but if you are certain that it is against policy and not able to buy it then why one should be added to the team and even if its done then why bothering about it. Once again, sincere apologies. Abhishek On Wed, 20 Nov 2019 at 8:16 PM, Mohammed Naser <mnaser@vexxhost.com> wrote:
On Wed, Nov 20, 2019 at 9:43 AM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Matt,
For your kind information, I never asked to be to the core of
stable-maintainer. Someone has recommended my name for it. So if you find it hard ir against the stable core policy, kindly remove me from this list. I will earn it with my efforts.
Just for context, I don't think Matt was actually targeting you, it's actually something that has been raised a few times in the past with many other projects (i.e. Trove is one another recent memory). I'm happy to see you on the stable team for Glance. . :)
Thank you for your kind support.
Abhishek Kekane
On Wed, 20 Nov 2019 at 8:00 PM, Matt Riedemann <mriedemos@gmail.com> wrote:
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
With Tony more removed these days and I myself not wanting to vet every one of these "add me to the stable core team" requests, I'm more or less OK with the proposal so that it removes me as a bottleneck. That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it. If it's a problem hopefully they'll hear about it from their consumers, but if the project is in such maintenance mode anyway that they can break the stable guidelines, then they might not have many external consumers to complain anyway. Either way I don't need to be involved.
So sure, +1 from me on the proposal given nova can still do what it's already been doing with a specific stable maint core team ACL in Gerrit.
--
Thanks,
Matt
-- Thanks & Best Regards,
Abhishek Kekane
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. https://vexxhost.com
-- Thanks & Best Regards,
Abhishek Kekane
On 11/20/19 8:21 AM, Matt Riedemann wrote:
That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it.
Another interesting point that was made in Shanghai was that the initial review is only the first level where bad backports can be caught. Even if something merges, it still has to be proposed for release, at which point the release liaison or PTL should be looking at it, and then once the release is proposed the release team is going to look at the changes included. So there is a safety net if a reviewer makes a mistake.
On 2019-11-20 08:49:23 -0600 (-0600), Ben Nemec wrote: [...]
Even if something merges, it still has to be proposed for release, at which point the release liaison or PTL should be looking at it, and then once the release is proposed the release team is going to look at the changes included. So there is a safety net if a reviewer makes a mistake.
True in principle, but we've basically always treated stable branches as a place from which downstream consumers can consume patches, and the stable point releases on them are more of a formality. I may simply not be connected with the right segments of our community, but I haven't heard anyone say they specifically wait to consume stable branch point releases vs just taking the branch content at a random point in time or selectively picking relevant patches out of it to incorporate into a packaged version... and even the theoretical stable point release reviewer safety net vaporizes for branches which pass into extended maintenance mode. However, the above should not be taken as an objection on my part for the plan. I agree the real safety net here is the users, and the lessons a reviewer learns after helping a panicked user of their software work around a regression or behavior change which should never have been allowed to merge on a stable branch in the first place. Failure is the best teacher. -- Jeremy Stanley
---- On Wed, 20 Nov 2019 08:21:29 -0600 Matt Riedemann <mriedemos@gmail.com> wrote ----
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
With Tony more removed these days and I myself not wanting to vet every one of these "add me to the stable core team" requests, I'm more or less OK with the proposal so that it removes me as a bottleneck. That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it. If it's a problem hopefully they'll hear about it from their consumers, but if the project is in such maintenance mode anyway that they can break the stable guidelines, then they might not have many external consumers to complain anyway. Either way I don't need to be involved.
This is the main problem going to be and I am worried about it. We had the great stable policies with a dedicated team maintaining it well. 'great' and 'well' I am writing from the user perspective where they get applicable backport which does not break their within-in-release upgrade. If backport is delayed it is fine for them as compared to breaking backport. OpenStack has a very well known problem of inconsistent APIs. Inconsistency is from usage, interop, debug perspective. All projects define its own definition of APIs (new API, changes, discoverability etc) which we say it is fine because that project wants to do that way but the user faces the problem. We have not solved this problem yet due to various reasons. IMO, this problem could have solved or minimized if we had "Single mandatory way to write and maintain your API and central team to very that" from *starting* instead of project decide its own way or recommended guidelines only. The same case is for stable backports, we have a "Single mandatory way to backport the changes and central team to verify that". And due to that, OpenStack is more stable on the backports business. Why we are changing that? Stable backport is more of maintaining the quality and no-break things than merging a large amount of backports and *fast*. -gmann
So sure, +1 from me on the proposal given nova can still do what it's already been doing with a specific stable maint core team ACL in Gerrit.
--
Thanks,
Matt
On 11/20/19 11:08 AM, Ghanshyam Mann wrote:
---- On Wed, 20 Nov 2019 08:21:29 -0600 Matt Riedemann <mriedemos@gmail.com> wrote ----
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
With Tony more removed these days and I myself not wanting to vet every one of these "add me to the stable core team" requests, I'm more or less OK with the proposal so that it removes me as a bottleneck. That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it. If it's a problem hopefully they'll hear about it from their consumers, but if the project is in such maintenance mode anyway that they can break the stable guidelines, then they might not have many external consumers to complain anyway. Either way I don't need to be involved.
This is the main problem going to be and I am worried about it. We had the great stable policies with a dedicated team maintaining it well. 'great' and 'well' I am writing from the user perspective where they get applicable backport which does not break their within-in-release upgrade. If backport is delayed it is fine for them as compared to breaking backport.
Is it? Important backports being delayed due to lack of stable reviewers still leaves consumers with a broken piece of software. I also think this is a bit of a false dichotomy. Nobody is suggesting that we start approving backports willy-nilly, just that we lower the barrier to entry for people who want to help with stable branch reviews. I would hope we can trust our teams to be responsible with their stable-maint list and not just start handing out +2 to anyone with a pulse. If not, I think we have a bigger problem, but let's cross that bridge if and when we come to it.
OpenStack has a very well known problem of inconsistent APIs. Inconsistency is from usage, interop, debug perspective. All projects define its own definition of APIs (new API, changes, discoverability etc) which we say it is fine because that project wants to do that way but the user faces the problem. We have not solved this problem yet due to various reasons.
IMO, this problem could have solved or minimized if we had "Single mandatory way to write and maintain your API and central team to very that" from *starting* instead of project decide its own way or recommended guidelines only.
The same case is for stable backports, we have a "Single mandatory way to backport the changes and central team to verify that". And due to that, OpenStack is more stable on the backports business. Why we are changing that?
This does not change the stable policy, and the existence of project-specific stable-maint teams means there was never a single central team reviewing all stable backports. Even if there had been, it's pretty clear that model doesn't work given the staffing constraints we are facing in almost every area, and which are only going to get worse after this cycle. The proposal removes some inflexible process that currently prevents contributors from becoming stable maintainers, it does _not_ mean we stop expecting stable maintainers to understand the stable policy.
Stable backport is more of maintaining the quality and no-break things than merging a large amount of backports and *fast*.
-gmann
So sure, +1 from me on the proposal given nova can still do what it's already been doing with a specific stable maint core team ACL in Gerrit.
--
Thanks,
Matt
On Wed, Nov 20, 2019 at 7:50 PM Ben Nemec <openstack@nemebean.com> wrote:
---- On Wed, 20 Nov 2019 08:21:29 -0600 Matt Riedemann < mriedemos@gmail.com> wrote ----
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with
review/backport history of contributors in every project as the individual project stable team is with contributors in each
This is assuming that each project has a stable core team already,
which
a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
With Tony more removed these days and I myself not wanting to vet every one of these "add me to the stable core team" requests, I'm more or less OK with the proposal so that it removes me as a bottleneck. That might mean people merge things on stable branches for their projects that don't follow the guidelines but so be it. If it's a problem hopefully they'll hear about it from their consumers, but if the project is in such maintenance mode anyway that they can break the stable guidelines, then they might not have many external consumers to complain anyway. Either way I don't need to be involved.
This is the main problem going to be and I am worried about it. We had
On 11/20/19 11:08 AM, Ghanshyam Mann wrote: the project. the great stable policies
with a dedicated team maintaining it well. 'great' and 'well' I am writing from the user perspective where they get applicable backport which does not break their within-in-release upgrade. If backport is delayed it is fine for them as compared to breaking backport.
Is it? Important backports being delayed due to lack of stable reviewers still leaves consumers with a broken piece of software.
I also think this is a bit of a false dichotomy. Nobody is suggesting that we start approving backports willy-nilly, just that we lower the barrier to entry for people who want to help with stable branch reviews. I would hope we can trust our teams to be responsible with their stable-maint list and not just start handing out +2 to anyone with a pulse. If not, I think we have a bigger problem, but let's cross that bridge if and when we come to it.
OpenStack has a very well known problem of inconsistent APIs.
usage, interop, debug perspective. All projects define its own definition of APIs (new API, changes, discoverability etc) which we say it is fine because that
Inconsistency is from project wants to do that way
but the user faces the problem. We have not solved this problem yet due to various reasons.
IMO, this problem could have solved or minimized if we had "Single mandatory way to write and maintain your API and central team to very that" from *starting* instead of project decide its own way or recommended guidelines only.
The same case is for stable backports, we have a "Single mandatory way to backport the changes and central team to verify that". And due to that, OpenStack is more stable on the backports business. Why we are changing that?
This does not change the stable policy, and the existence of project-specific stable-maint teams means there was never a single central team reviewing all stable backports. Even if there had been, it's pretty clear that model doesn't work given the staffing constraints we are facing in almost every area, and which are only going to get worse after this cycle.
The proposal removes some inflexible process that currently prevents contributors from becoming stable maintainers, it does _not_ mean we stop expecting stable maintainers to understand the stable policy.
Stable backport is more of maintaining the quality and no-break things
than merging a large amount
of backports and *fast*.
-gmann
So sure, +1 from me on the proposal given nova can still do what it's already been doing with a specific stable maint core team ACL in
Gerrit.
--
Thanks,
Matt
Sounds like we're back in a spot where Stable Maintenance team probably shouldn't be a thing then and I think this time around I do agree. Perhaps we should just discontinue the Stable Maintenance group all together (not just as part of stable reviewers and moderators of in project stable maintenance groups) and subject our Stable Branch policies and guidelines directly under governance. Obviously unless we're planning to put the bolt on TCs head as well being unscalable group most don't care about. (We might even get some folks who care about stable branches and releases to be interested about TC.) If nothing else, the good part of this discussion is the message that we have amazing flood of people being interested to join the stable maintenance to overwhelm the current Stable Maint team with requests. So lets remove the clearly toxic vetting process and get these people to work! - jokke_
On 11/21/19 7:46 AM, Erno Kuvaja wrote:
On Wed, Nov 20, 2019 at 7:50 PM Ben Nemec <openstack@nemebean.com <mailto:openstack@nemebean.com>> wrote:
On 11/20/19 11:08 AM, Ghanshyam Mann wrote: > ---- On Wed, 20 Nov 2019 08:21:29 -0600 Matt Riedemann <mriedemos@gmail.com <mailto:mriedemos@gmail.com>> wrote ---- > > On 11/20/2019 1:18 AM, Zane Bitter wrote: > > > Because the core stable team is necessarily not as familiar with the > > > review/backport history of contributors in every project as the > > > individual project stable team is with contributors in each project. > > > > This is assuming that each project has a stable core team already, which > > a lot don't, that's why we get a lot of "hi I'm the PTL du jour on > > project X now please make me stable core even though I've never reviewed > > any stable branch changes before". > > > > With Tony more removed these days and I myself not wanting to vet every > > one of these "add me to the stable core team" requests, I'm more or less > > OK with the proposal so that it removes me as a bottleneck. That might > > mean people merge things on stable branches for their projects that > > don't follow the guidelines but so be it. If it's a problem hopefully > > they'll hear about it from their consumers, but if the project is in > > such maintenance mode anyway that they can break the stable guidelines, > > then they might not have many external consumers to complain anyway. > > Either way I don't need to be involved. > > This is the main problem going to be and I am worried about it. We had the great stable policies > with a dedicated team maintaining it well. 'great' and 'well' I am writing from the user perspective > where they get applicable backport which does not break their within-in-release upgrade. > If backport is delayed it is fine for them as compared to breaking backport.
Is it? Important backports being delayed due to lack of stable reviewers still leaves consumers with a broken piece of software.
I also think this is a bit of a false dichotomy. Nobody is suggesting that we start approving backports willy-nilly, just that we lower the barrier to entry for people who want to help with stable branch reviews. I would hope we can trust our teams to be responsible with their stable-maint list and not just start handing out +2 to anyone with a pulse. If not, I think we have a bigger problem, but let's cross that bridge if and when we come to it.
> > > OpenStack has a very well known problem of inconsistent APIs. Inconsistency is from > usage, interop, debug perspective. All projects define its own definition of APIs (new API, > changes, discoverability etc) which we say it is fine because that project wants to do that way > but the user faces the problem. We have not solved this problem yet due to various reasons. > > IMO, this problem could have solved or minimized if we had "Single mandatory way to write > and maintain your API and central team to very that" from *starting* instead of project > decide its own way or recommended guidelines only. > > The same case is for stable backports, we have a "Single mandatory way to backport the changes > and central team to verify that". And due to that, OpenStack is more stable on the backports business. > Why we are changing that?
This does not change the stable policy, and the existence of project-specific stable-maint teams means there was never a single central team reviewing all stable backports. Even if there had been, it's pretty clear that model doesn't work given the staffing constraints we are facing in almost every area, and which are only going to get worse after this cycle.
The proposal removes some inflexible process that currently prevents contributors from becoming stable maintainers, it does _not_ mean we stop expecting stable maintainers to understand the stable policy.
> > Stable backport is more of maintaining the quality and no-break things than merging a large amount > of backports and *fast*. > > -gmann > > > > > So sure, +1 from me on the proposal given nova can still do what it's > > already been doing with a specific stable maint core team ACL in Gerrit. > > > > -- > > > > Thanks, > > > > Matt > > > > > >
Sounds like we're back in a spot where Stable Maintenance team probably shouldn't be a thing then and I think this time around I do agree.
Perhaps we should just discontinue the Stable Maintenance group all together (not just as part of stable reviewers and moderators of in project stable maintenance groups) and subject our Stable Branch policies and guidelines directly under governance. Obviously unless we're planning to put the bolt on TCs head as well being unscalable group most don't care about. (We might even get some folks who care about stable branches and releases to be interested about TC.)
I don't have a strong opinion about this, other than that it's important to have someone people can go to for questions about stable backports. If that's a stable team or some other entity is irrelevant to me, but unusual situations do come up and the stable team was extremely helpful with the one I asked about recently[0]. 0: http://lists.openstack.org/pipermail/openstack-discuss/2019-October/009984.h...
If nothing else, the good part of this discussion is the message that we have amazing flood of people being interested to join the stable maintenance to overwhelm the current Stable Maint team with requests. So lets remove the clearly toxic vetting process and get these people to work!
- jokke_
On 2019-11-21 13:46:06 +0000 (+0000), Erno Kuvaja wrote: [...]
Sounds like we're back in a spot where Stable Maintenance team probably shouldn't be a thing then and I think this time around I do agree. [...]
Conveniently, it hasn't been a thing for 1.5 years now, ever since https://review.openstack.org/584206 merged to officially disband and replace it with a SIG. -- Jeremy Stanley
On 20/11/19 9:21 am, Matt Riedemann wrote:
On 11/20/2019 1:18 AM, Zane Bitter wrote:
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
Correct, what I'm suggesting is a middle-ground position so that in the cases where there is no project-specific stable team, that team has to be bootstrapped by the global stable-maint team in the same way that they do already. This avoids the situation you mention, where e.g. the TC appoints a PTL who does not even qualify to run for election (no commits to the project) and suddenly they're able to approve stable backports with no training or oversight. We're obliged to appoint a PTL for every project, whether they're qualified or not, but we should not be obliged to add unqualified people to the project stable core team. For the cases where there *is* already a project stable team, it allows folks who have already been vetted and who are closest to the data to have input on the decision, and it relieves the burden of 3 already overworked people who are currently required to do all of the vetting of new stable reviewers. cheers, Zane.
Zane Bitter wrote:
On 20/11/19 9:21 am, Matt Riedemann wrote:
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
Correct, what I'm suggesting is a middle-ground position so that in the cases where there is no project-specific stable team, that team has to be bootstrapped by the global stable-maint team in the same way that they do already.
That sounds like a reasonable compromise. -- Thierry Carrez (ttx)
On 26/11/2019 09:47, Thierry Carrez wrote:
Zane Bitter wrote:
On 20/11/19 9:21 am, Matt Riedemann wrote:
This is assuming that each project has a stable core team already, which a lot don't, that's why we get a lot of "hi I'm the PTL du jour on project X now please make me stable core even though I've never reviewed any stable branch changes before".
Correct, what I'm suggesting is a middle-ground position so that in the cases where there is no project-specific stable team, that team has to be bootstrapped by the global stable-maint team in the same way that they do already.
That sounds like a reasonable compromise.
Yeah, I like this idea - it allows more flexibility while not chaining the door wide open.
---- On Wed, 20 Nov 2019 01:18:39 -0600 Zane Bitter <zbitter@redhat.com> wrote ----
On 18/11/19 9:18 pm, Ghanshyam Mann wrote:
---- On Mon, 18 Nov 2019 19:17:04 -0600 Zane Bitter <zbitter@redhat.com> wrote ----
On 18/11/19 5:35 pm, Ben Nemec wrote:
On 11/18/19 4:08 PM, Matt Riedemann wrote:
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core.
I believe the proposal is to allow the Nova team to manage nova-stable-maint in the same way they do nova-core, not to force anyone to drop their stable-maint team entirely.
I think the proposal was actually for each *-stable-maint team to manage itself. This would avoid the situation where e.g. the TC appoints a brand-new PTL and suddenly they get to make themselves a stable core, as in that case the team would still have to be bootstrapped by the stable-maint team. But it would allow those who are both closest to the project and confirmed to be familiar with the stable guidelines to make decisions about who else is ready to join that group.
I am still finding difficult to understand the change and how it will solve the current problem.
The current problem is: * Fewer contributors in the stable-maintenance team (core stable team and project side stable team) which is nothing but we have fewer contributors who understand the stable policies.
* The stable policies are not the problem so we will stick with current stable policies across all the projects. Stable policies have to be maintained at single place for consistency in backports across projects.
If we are moving the stable maintenance team ownership from current stable-maintenance team to project side then, how it will solve the issue, does it enable more contributors to understand the stable policy and extend the team?
Yes.
if yes, then why it cannot happen with current model?
Because the core stable team is necessarily not as familiar with the review/backport history of contributors in every project as the individual project stable team is with contributors in each project.
If the project team or PTL making its core member get more familiar with the stable policy and add as a stable core team then why it cannot happen with the current model.
For example, if I am PTL or core of any project and finding hard to get my backport merged then I or my project team core should review more stable branch patches and propose them in stable team core.
I have tried that with only very limited success.
If we move the stable team ownership to the projects side then I think PTL is going to do the same. Ask the team members to understand the stable policies and do more review and then add them in stable core team. If any member know the stable policies then directly add.
You make it sound like that's not a good thing?
This is a good thing but I am saying why it cannot happen in the current model? If I am PTL of x project then and find hard to merge my backport I can ask for stable team volunteer from my core/active contributors and then they learn/onboard to stable team.
I feel that the current problem cannot be solved by moving the ownership of the team, we need to encourage more and more developers to become stable core in existing model especially from projects who find difficulties in merging their backport.
In my experience at least there's no shortage of people willing to do the work, but there is a severe shortage of people willing to do the work of climbing over the bar to get permission to do the work.
The position espoused by Tony and Matt at least is that those shouldn't be different things, and in principle that's correct, but in practice they are. Humans are weird.
humm, I feel same can exist in new model also. Not all existing core going to become the stable core for that project. PTL or someone has to filter out the list and add the new members to list where same issue can happen. I feel if stable policies are going to be the same then we should not change the model which is working fine from user perspective. -gmann
One more thing, do we have data that how much time as avg it take to merge the backport and what all projects facing the backport merge issue ?
-gmann
- ZB
[1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members
On Mon, Nov 18, 2019 at 5:13 PM Matt Riedemann <mriedemos@gmail.com> wrote:
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core.
Thanks for bringing this up, I think we'll slowly iron those out. I think this can be a team-specific decision, we should have $project-stable-maint for every single project anyways, and the team could decide to put all of $project-core inside of it, or a select group of people.
[1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members
--
Thanks,
Matt
On Mon, Nov 18, 2019 at 04:08:24PM -0600, Matt Riedemann wrote:
On 11/18/2019 3:40 PM, Mohammed Naser wrote:
The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects).
So for a project like nova that has a separate nova-core [1] and nova-stable-maint team [2] where some from [2] aren't in [1], what does this mean? Drop [2] and just rely on [1]? That won't work for those in nova-core that aren't familiar enough with the stable branch guidelines or simply don't care to review stable branch changes, and won't work for those that are in nova-stable-maint but not nova-core.
I wouldn't think that anything would need to change about how Nova does things. If the Nova team wants to manage Nova stable branches using nova-stable-maint then this proposal absolutely supports that. The main change is removing stable-maint-core [3] from nove-stable-maint as stable-maint-core would presumably be dissolving as part of this change. Many teams already have a stable team [4]. For the ones that don't seem to (for example packaging-rpm, telemetry, monasca, or kuryr) it would make sense to make a $PROJECT-stable-maint and then leave it up to that project to either add $PROJECT-core to it or designate specific members to manage the stable branches. So in the end all the teams have the option to work like Nova does. Nate
[1] https://review.opendev.org/#/admin/groups/25,members [2] https://review.opendev.org/#/admin/groups/540,members [3] https://review.opendev.org/#/admin/groups/530,members [4] https://review.opendev.org/#/admin/groups/?filter=stable
participants (12)
-
Abhishek Kekane
-
Ben Nemec
-
Erno Kuvaja
-
Ghanshyam Mann
-
Graham Hayes
-
Jeremy Stanley
-
Lingxian Kong
-
Matt Riedemann
-
Mohammed Naser
-
Nate Johnston
-
Thierry Carrez
-
Zane Bitter