Open Stack

Hi,

First of all, let's just start that Matt's resolution about Extended 
Maintenance [1] is a good and detailed description about what and how 
the community decided to handle the extended support of old stable 
branches. Even the testing difficulties part [2] is well described and 
actually explains the very same situation what Gmann's mail is about. 
According to that section it is possible that branches uses reduced sets 
of testing, even without tempest jobs. I think we understood that it is 
more risky to accept fixes that might not be fully tested, but that is a 
risk that we accepted when Extended Maintenance process was proposed. 
Furthermore, I could imagine that some patches are not accepted to get 
merged in such branches because stable cores consider it too risky 
without testing it properly, while other patches can be easily merged as 
those are not considered dangerous.

On the other hand of course if a project's stable maintainers do not 
have the capacity to deal with an old branch, then it is a possibility 
to propose End of Life for that branch on this mailing list (according 
to process [3]). If no one volunteers to help them with the reviews and 
backports, then it's time to EOL. And again, according to the 
resolution, it is imagined a per-project EOL, not an overall 
community-wide EOL as was before.

What I see though is that there are not that high number of incoming 
patches against Ocata in the last months so it shouldn't take that much 
extra work compared to newer stable branches, still it needs work of 
course and dedicated reviewers, dedicated time. So who are interested in 
keeping old branches alive should step up and spend time on reviewing. 
We shouldn't forget that this process is to help developers, vendors and 
operators. But we need them to show interest.

What Sean wrote about requirement team's burden is much more real as 
that is the biggest pain point of the old branches. (Especially now, 
when not even py27 is dropped, but random modules decide to drop also 
py35, py36, py37, too - not to mention that sometimes they do it 
improperly, causing extra chaos). This could really push us toward 
EOL'ing a complete branch. So maybe this team needs more help from 
interested parties.

Finally, my last comment: it's a great success that two years after the 
originally planned EOL date, Ocata could still accept fixes. That was/is 
the goal of Extended Maintenance.

TL;DR: If it's not feasible to fix a general issue of a job, then drop 
that job. And I think we should not EOL Ocata in general, rather let 
projects EOL their ocata branch if they cannot invest more time on 
fixing them.

[1] 
https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.html
[2] 
https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.html#testing
[3] 
https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-life

Előd


On 2020. 05. 28. 20:30, Sean McGinnis wrote:
> On 5/28/20 12:03 PM, Ghanshyam Mann wrote:
>> Hello Everyone,
>>
>> Ocata is in the 'Extended Maintenance' phase[1] but its gate is 
>> broken and difficult to fix now.
>>
>> The main issue started for gate is about Tempest and its deps 
>> constraint is correctly used on Ocata or not.
>> We fixed most of those issues when things started failing during py2 
>> drop[2].
>>
>> But now things are broken due to newer stestr is installed in Ocata 
>> jobs[2]. As Sean also mentioned in this
>> backport[4] that stestr is not constraint in Ocata[5] as it was not 
>> in g-r that time so cannot be added in u-c.
>
> We actually did just recently merge a patch to add it:
>
> https://review.opendev.org/#/c/725213/
>
> There were some other breakages that had to be fixed before we could get
> it to land, so that took a little longer than expected. That said...
>
>>
>> Tempest old tag cannot be fixed to have that with cap on the tempest 
>> side.
>>
>> The only option left to unblock the gate is to remove the integration 
>> testing from Ocata gate. But it will untested after that
>> as we cannot say unit testing is enough to mark it tested.  At least 
>> from QA perspective, I would not say it is tested
>> if there is no integration job running there.
>>
>> Keeping it untested (no integration testing) and saying it is 
>> Maintained in the community can be very confusing things.
>> I think its time to move it to the 'Unmaintained' phase and then EOL. 
>> Thought?
>
> I do think it is getting to the point where it is becoming increasingly
> difficult to keep stable/ocata happy, and time is better spent on more
> recent stable branches.
>
> When we discussed changing the stable policy to allow for Extended
> Maintenance, part of that was that we would support EM as long as
> someone was able to keep things working and tested. I think we've gotten
> to the point where that is no longer the case with ocata.
>
> This is the first branch that has made it this long before going EOL, so
> it has been useful to see what other issues came up that were not
> discussed. One thing that it's made me realize is there is a lot more
> codependency than we maybe realized in those early discussions. The
> policy and discussions that I remember were really about whether each
> project would continue to put in the effort to keep things running. But
> it really requires a lot more than just an individual project team to do
> that. If any one of the "core" projects starts to have an issue, that
> really means all of the projects have an issue. So it would not be
> possible for, just as an example, Cinder to decide to transition its
> stable/ocata branch to EOL, but Nova to keep maintaining their
> stable/ocata branch.
>
> There is also the cross-cutting concerns of requirements management and
> tempest. I think these two areas have been where the most issues have
> cropped up, and therefore requiring the most work to keep things
> running. Kudos to gmann and the tempest team for getting things to work
> after we've dropped py27. But requirements is a very small team (as is
> tempest/QA), and things are breaking that require some thought and time
> to work through the right way to handle issues in a stable branch, so a
> lot of the effort is falling to these smaller groups to try to prop
> things up.
>
> I know I personally don't have extra time to be working on these, and no
> real motivation other than I like to see Zuul comments be green. Now
> that we've gotten the stestr issue hopefully resolved, I don't think I
> can spend any more time on stable/ocata issues. Others are free to pitch
> in, but I really think it is in the community's best interest if we just
> decide we've done enough (Ocata would have gone EOL over 2 years ago
> under the old policy) and it's time to call it and mark those branches 
> EOL.
>
> Sean
>
>