Hi, First of all, let's just start that Matt's resolution about Extended Maintenance [1] is a good and detailed description about what and how the community decided to handle the extended support of old stable branches. Even the testing difficulties part [2] is well described and actually explains the very same situation what Gmann's mail is about. According to that section it is possible that branches uses reduced sets of testing, even without tempest jobs. I think we understood that it is more risky to accept fixes that might not be fully tested, but that is a risk that we accepted when Extended Maintenance process was proposed. Furthermore, I could imagine that some patches are not accepted to get merged in such branches because stable cores consider it too risky without testing it properly, while other patches can be easily merged as those are not considered dangerous. On the other hand of course if a project's stable maintainers do not have the capacity to deal with an old branch, then it is a possibility to propose End of Life for that branch on this mailing list (according to process [3]). If no one volunteers to help them with the reviews and backports, then it's time to EOL. And again, according to the resolution, it is imagined a per-project EOL, not an overall community-wide EOL as was before. What I see though is that there are not that high number of incoming patches against Ocata in the last months so it shouldn't take that much extra work compared to newer stable branches, still it needs work of course and dedicated reviewers, dedicated time. So who are interested in keeping old branches alive should step up and spend time on reviewing. We shouldn't forget that this process is to help developers, vendors and operators. But we need them to show interest. What Sean wrote about requirement team's burden is much more real as that is the biggest pain point of the old branches. (Especially now, when not even py27 is dropped, but random modules decide to drop also py35, py36, py37, too - not to mention that sometimes they do it improperly, causing extra chaos). This could really push us toward EOL'ing a complete branch. So maybe this team needs more help from interested parties. Finally, my last comment: it's a great success that two years after the originally planned EOL date, Ocata could still accept fixes. That was/is the goal of Extended Maintenance. TL;DR: If it's not feasible to fix a general issue of a job, then drop that job. And I think we should not EOL Ocata in general, rather let projects EOL their ocata branch if they cannot invest more time on fixing them. [1] https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.h... [2] https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.h... [3] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-li... Előd On 2020. 05. 28. 20:30, Sean McGinnis wrote:
On 5/28/20 12:03 PM, Ghanshyam Mann wrote:
Hello Everyone,
Ocata is in the 'Extended Maintenance' phase[1] but its gate is broken and difficult to fix now.
The main issue started for gate is about Tempest and its deps constraint is correctly used on Ocata or not. We fixed most of those issues when things started failing during py2 drop[2].
But now things are broken due to newer stestr is installed in Ocata jobs[2]. As Sean also mentioned in this backport[4] that stestr is not constraint in Ocata[5] as it was not in g-r that time so cannot be added in u-c.
We actually did just recently merge a patch to add it:
https://review.opendev.org/#/c/725213/
There were some other breakages that had to be fixed before we could get it to land, so that took a little longer than expected. That said...
Tempest old tag cannot be fixed to have that with cap on the tempest side.
The only option left to unblock the gate is to remove the integration testing from Ocata gate. But it will untested after that as we cannot say unit testing is enough to mark it tested. At least from QA perspective, I would not say it is tested if there is no integration job running there.
Keeping it untested (no integration testing) and saying it is Maintained in the community can be very confusing things. I think its time to move it to the 'Unmaintained' phase and then EOL. Thought?
I do think it is getting to the point where it is becoming increasingly difficult to keep stable/ocata happy, and time is better spent on more recent stable branches.
When we discussed changing the stable policy to allow for Extended Maintenance, part of that was that we would support EM as long as someone was able to keep things working and tested. I think we've gotten to the point where that is no longer the case with ocata.
This is the first branch that has made it this long before going EOL, so it has been useful to see what other issues came up that were not discussed. One thing that it's made me realize is there is a lot more codependency than we maybe realized in those early discussions. The policy and discussions that I remember were really about whether each project would continue to put in the effort to keep things running. But it really requires a lot more than just an individual project team to do that. If any one of the "core" projects starts to have an issue, that really means all of the projects have an issue. So it would not be possible for, just as an example, Cinder to decide to transition its stable/ocata branch to EOL, but Nova to keep maintaining their stable/ocata branch.
There is also the cross-cutting concerns of requirements management and tempest. I think these two areas have been where the most issues have cropped up, and therefore requiring the most work to keep things running. Kudos to gmann and the tempest team for getting things to work after we've dropped py27. But requirements is a very small team (as is tempest/QA), and things are breaking that require some thought and time to work through the right way to handle issues in a stable branch, so a lot of the effort is falling to these smaller groups to try to prop things up.
I know I personally don't have extra time to be working on these, and no real motivation other than I like to see Zuul comments be green. Now that we've gotten the stestr issue hopefully resolved, I don't think I can spend any more time on stable/ocata issues. Others are free to pitch in, but I really think it is in the community's best interest if we just decide we've done enough (Ocata would have gone EOL over 2 years ago under the old policy) and it's time to call it and mark those branches EOL.
Sean