Hi,

On Thu, May 28, 2020 at 02:06:08PM -0400, Brian Rosmaita wrote:

On 5/28/20 1:03 PM, Ghanshyam Mann wrote:

...

Hello Everyone,

Ocata is in the 'Extended Maintenance' phase[1] but its gate is broken and difficult to fix now.

The main issue started for gate is about Tempest and its deps constraint is correctly used on Ocata or not. We fixed most of those issues when things started failing during py2 drop[2].

But now things are broken due to newer stestr is installed in Ocata jobs[2]. As Sean also mentioned in this backport[4] that stestr is not constraint in Ocata[5] as it was not in g-r that time so cannot be added in u-c.

Tempest old tag cannot be fixed to have that with cap on the tempest side.

The only option left to unblock the gate is to remove the integration testing from Ocata gate. But it will untested after that as we cannot say unit testing is enough to mark it tested. At least from QA perspective, I would not say it is tested if there is no integration job running there.

Keeping it untested (no integration testing) and saying it is Maintained in the community can be very confusing things. I think its time to move it to the 'Unmaintained' phase and then EOL. Thought?

+1000

+1 from the Neutron too. I checked recently that we don't have any new patches merged to Ocata since around October 2019.

It would be really helpful if we could make this transition as soon as possible. I don't see any reason to delay, particularly since the policy [a] allows a transition back to EM should maintainers step up in the next six months.

How about putting Ocata into the 'Unmaintained' phase on 1 June 2020?

[a] https://docs.openstack.org/project-team-guide/stable-branches.html#unmaintai...

...

[1] https://releases.openstack.org/ [2] https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:...) [3] https://zuul.opendev.org/t/openstack/build/e477d0919ff14eea9cd8a6235ea00c3d/... [4] https://review.opendev.org/#/c/726931/ [5] https://opendev.org/openstack/requirements/src/branch/stable/ocata/upper-con...

-gmann

On 2020-05-28 13:30:29 -0500 (-0500), Sean McGinnis wrote: [...]

This is the first branch that has made it this long before going EOL, so it has been useful to see what other issues came up that were not discussed.

[...]

I'm actually shocked it has survived for so long. Tagged on 2017-02-22, the Ocata coordinated release has been in some active state of maintenance for well over 3 years already. That's triple the duration we used to keep stable branches open before the EM process was implemented. Three cheers for everyone who's kept stable/ocata branches alive and viable all that time!

Considering recent OSSA issues did not release fixes for Ocata[1][2], I think we should really consider making the maintenance status more clear and/or per project.

I know that Adam proposed this a while ago for Octavia, but it seems to be stuck pending reviews.[3]

Michael

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014717.html [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013251.htm... [3] https://review.opendev.org/#/c/719097/

On Thu, May 28, 2020 at 2:26 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:

---- On Thu, 28 May 2020 13:30:29 -0500 Sean McGinnis sean.mcginnis@gmx.com wrote ----

...

On 5/28/20 12:03 PM, Ghanshyam Mann wrote:

...

Hello Everyone,

Ocata is in the 'Extended Maintenance' phase[1] but its gate is broken and difficult to fix now.

The main issue started for gate is about Tempest and its deps constraint is correctly used on Ocata or not. We fixed most of those issues when things started failing during py2 drop[2].

But now things are broken due to newer stestr is installed in Ocata jobs[2]. As Sean also mentioned in this backport[4] that stestr is not constraint in Ocata[5] as it was not in g-r that time so cannot be added in u-c.

We actually did just recently merge a patch to add it:

https://review.opendev.org/#/c/725213/

There were some other breakages that had to be fixed before we could get it to land, so that took a little longer than expected. That said...

...

Tempest old tag cannot be fixed to have that with cap on the tempest side.

The only option left to unblock the gate is to remove the integration testing from Ocata gate. But it will untested after that as we cannot say unit testing is enough to mark it tested. At least from QA perspective, I would not say it is tested if there is no integration job running there.

Keeping it untested (no integration testing) and saying it is Maintained in the community can be very confusing things. I think its time to move it to the 'Unmaintained' phase and then EOL. Thought?

I do think it is getting to the point where it is becoming increasingly difficult to keep stable/ocata happy, and time is better spent on more recent stable branches.

When we discussed changing the stable policy to allow for Extended Maintenance, part of that was that we would support EM as long as someone was able to keep things working and tested. I think we've gotten to the point where that is no longer the case with ocata.

This is the first branch that has made it this long before going EOL, so it has been useful to see what other issues came up that were not discussed. One thing that it's made me realize is there is a lot more codependency than we maybe realized in those early discussions. The policy and discussions that I remember were really about whether each project would continue to put in the effort to keep things running. But it really requires a lot more than just an individual project team to do that. If any one of the "core" projects starts to have an issue, that really means all of the projects have an issue. So it would not be possible for, just as an example, Cinder to decide to transition its stable/ocata branch to EOL, but Nova to keep maintaining their stable/ocata branch.

There is also the cross-cutting concerns of requirements management and tempest. I think these two areas have been where the most issues have cropped up, and therefore requiring the most work to keep things running. Kudos to gmann and the tempest team for getting things to work after we've dropped py27. But requirements is a very small team (as is tempest/QA), and things are breaking that require some thought and time to work through the right way to handle issues in a stable branch, so a lot of the effort is falling to these smaller groups to try to prop things up.

+1, Even QA has the policy and that is what we all agreed on about not supporting the stable branches once it is Extended Maintainance but you know when we see things are broken we fix those. I think that is the usual nature I think :) But I agree that its time to move forward for Ocata.

Also once Ocata is gone, it will easy to freeze the devstack-gate. Ocata is last stable where we do not have zuulv3 native base jobs and Pike onwards we can backport everything converted in zuulv3.

...

I know I personally don't have extra time to be working on these, and no real motivation other than I like to see Zuul comments be green. Now that we've gotten the stestr issue hopefully resolved, I don't think I can spend any more time on stable/ocata issues. Others are free to pitch in, but I really think it is in the community's best interest if we just decide we've done enough (Ocata would have gone EOL over 2 years ago under the old policy) and it's time to call it and mark those branches EOL.

Yeah, we have extended it too much I think maybe due to the first one to go to 'Unmaintained' after Extended maintenance things started.

-gmann

...

Sean

Hi,

First of all, let's just start that Matt's resolution about Extended Maintenance [1] is a good and detailed description about what and how the community decided to handle the extended support of old stable branches. Even the testing difficulties part [2] is well described and actually explains the very same situation what Gmann's mail is about. According to that section it is possible that branches uses reduced sets of testing, even without tempest jobs. I think we understood that it is more risky to accept fixes that might not be fully tested, but that is a risk that we accepted when Extended Maintenance process was proposed. Furthermore, I could imagine that some patches are not accepted to get merged in such branches because stable cores consider it too risky without testing it properly, while other patches can be easily merged as those are not considered dangerous.

On the other hand of course if a project's stable maintainers do not have the capacity to deal with an old branch, then it is a possibility to propose End of Life for that branch on this mailing list (according to process [3]). If no one volunteers to help them with the reviews and backports, then it's time to EOL. And again, according to the resolution, it is imagined a per-project EOL, not an overall community-wide EOL as was before.

What I see though is that there are not that high number of incoming patches against Ocata in the last months so it shouldn't take that much extra work compared to newer stable branches, still it needs work of course and dedicated reviewers, dedicated time. So who are interested in keeping old branches alive should step up and spend time on reviewing. We shouldn't forget that this process is to help developers, vendors and operators. But we need them to show interest.

What Sean wrote about requirement team's burden is much more real as that is the biggest pain point of the old branches. (Especially now, when not even py27 is dropped, but random modules decide to drop also py35, py36, py37, too - not to mention that sometimes they do it improperly, causing extra chaos). This could really push us toward EOL'ing a complete branch. So maybe this team needs more help from interested parties.

Finally, my last comment: it's a great success that two years after the originally planned EOL date, Ocata could still accept fixes. That was/is the goal of Extended Maintenance.

TL;DR: If it's not feasible to fix a general issue of a job, then drop that job. And I think we should not EOL Ocata in general, rather let projects EOL their ocata branch if they cannot invest more time on fixing them.

[1] https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.h... [2] https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.h... [3] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-li...

Előd

On 2020. 05. 28. 20:30, Sean McGinnis wrote:

On 5/28/20 12:03 PM, Ghanshyam Mann wrote:

...

Hello Everyone,

Ocata is in the 'Extended Maintenance' phase[1] but its gate is broken and difficult to fix now.

The main issue started for gate is about Tempest and its deps constraint is correctly used on Ocata or not. We fixed most of those issues when things started failing during py2 drop[2].

But now things are broken due to newer stestr is installed in Ocata jobs[2]. As Sean also mentioned in this backport[4] that stestr is not constraint in Ocata[5] as it was not in g-r that time so cannot be added in u-c.

We actually did just recently merge a patch to add it:

https://review.opendev.org/#/c/725213/

There were some other breakages that had to be fixed before we could get it to land, so that took a little longer than expected. That said...

...

Tempest old tag cannot be fixed to have that with cap on the tempest side.

The only option left to unblock the gate is to remove the integration testing from Ocata gate. But it will untested after that as we cannot say unit testing is enough to mark it tested.  At least from QA perspective, I would not say it is tested if there is no integration job running there.

Keeping it untested (no integration testing) and saying it is Maintained in the community can be very confusing things. I think its time to move it to the 'Unmaintained' phase and then EOL. Thought?

I do think it is getting to the point where it is becoming increasingly difficult to keep stable/ocata happy, and time is better spent on more recent stable branches.

When we discussed changing the stable policy to allow for Extended Maintenance, part of that was that we would support EM as long as someone was able to keep things working and tested. I think we've gotten to the point where that is no longer the case with ocata.

This is the first branch that has made it this long before going EOL, so it has been useful to see what other issues came up that were not discussed. One thing that it's made me realize is there is a lot more codependency than we maybe realized in those early discussions. The policy and discussions that I remember were really about whether each project would continue to put in the effort to keep things running. But it really requires a lot more than just an individual project team to do that. If any one of the "core" projects starts to have an issue, that really means all of the projects have an issue. So it would not be possible for, just as an example, Cinder to decide to transition its stable/ocata branch to EOL, but Nova to keep maintaining their stable/ocata branch.

There is also the cross-cutting concerns of requirements management and tempest. I think these two areas have been where the most issues have cropped up, and therefore requiring the most work to keep things running. Kudos to gmann and the tempest team for getting things to work after we've dropped py27. But requirements is a very small team (as is tempest/QA), and things are breaking that require some thought and time to work through the right way to handle issues in a stable branch, so a lot of the effort is falling to these smaller groups to try to prop things up.

I know I personally don't have extra time to be working on these, and no real motivation other than I like to see Zuul comments be green. Now that we've gotten the stestr issue hopefully resolved, I don't think I can spend any more time on stable/ocata issues. Others are free to pitch in, but I really think it is in the community's best interest if we just decide we've done enough (Ocata would have gone EOL over 2 years ago under the old policy) and it's time to call it and mark those branches EOL.

Sean

---- On Fri, 29 May 2020 07:54:05 -0500 Sean McGinnis sean.mcginnis@gmx.com wrote ----

On 5/29/20 6:34 AM, Előd Illés wrote:

...

[snip]

TL;DR: If it's not feasible to fix a general issue of a job, then drop that job. And I think we should not EOL Ocata in general, rather let projects EOL their ocata branch if they cannot invest more time on fixing them.

The interdependency is the trick here. Some projects can easily EOL on their own and it's isolated enough that it doesn't cause issues. But for other projects, like Cinder and Nova that I mentioned, it's kind of an all-or-nothing situation.

I suppose it is feasible that we drop testing to only running unit tests. If we don't run any kind of integration testing, then it does make these projects a little more independent.

We still have the requirements issues though. Unless someone addresses any rot in the stable requirements, even unit tests become hard to run.

Just thinking out loud on some of the issues I see. We can try to follow the original EM plan and leave it up to each project to declare their intent to go EOL, then tag ocata-eol to close it out. Or we can collectively decide Ocata is done and pull the big switch.

From the stable policy if CI has broken nd no maintainer then we can move that

to unmaintained. And there is always time to revert back to EM if the maintainer shows up.

IMO, maintaining only with unit tests is not a good idea.

I have not heard from projects that they are interested to maintain it, if any then we can see how to proceed otherwise collectively marking Ocata as Unmaintained is the right thing.

[1] https://docs.openstack.org/project-team-guide/stable-branches.html#unmaintai...

-gmann

Sean

On 2020. 05. 29. 14:54, Sean McGinnis wrote:

On 5/29/20 6:34 AM, Előd Illés wrote:

...

[snip]

TL;DR: If it's not feasible to fix a general issue of a job, then drop that job. And I think we should not EOL Ocata in general, rather let projects EOL their ocata branch if they cannot invest more time on fixing them.

The interdependency is the trick here. Some projects can easily EOL on their own and it's isolated enough that it doesn't cause issues. But for other projects, like Cinder and Nova that I mentioned, it's kind of an all-or-nothing situation.

From the resolution: "Note that it is possible for our CI infrastructure to function based on EOL tags - https://review.opendev.org/#/c/520095/ [1] Does this cover the problem you mention? Or at least should we try that this is enough?

I suppose it is feasible that we drop testing to only running unit tests. If we don't run any kind of integration testing, then it does make these projects a little more independent.

We still have the requirements issues though. Unless someone addresses any rot in the stable requirements, even unit tests become hard to run.

Yes, I experienced the same: from time to time, new constraints need to be added to projects (especially those that are not using global upper-constraints.txt), and not always clear from the errors what is the exact root cause. Usually if someone takes the time to backport a patch also tries to fix the problem (not always, though, and not always with a success).

Just thinking out loud on some of the issues I see. We can try to follow the original EM plan and leave it up to each project to declare their intent to go EOL, then tag ocata-eol to close it out. Or we can collectively decide Ocata is done and pull the big switch.

Sean

Indeed it depends on what the community wants. Whether there are others who are interested in keeping a branch open, keeping Ocata open for bugfixes.

Thanks Sean and everyone, for thinking on this, and moreover for fixing the ocata branch issues so far. :)

Előd

[1] https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.h...