Open Stack

On Tue, Feb 10, 2015, at 05:19 AM, Thierry Carrez wrote:
> Joe, Matt & Matthew:
> 
> I hear your frustration with broken stable branches. With my
> vulnerability management team member hat, responsible for landing
> patches there with a strict deadline, I can certainly relate with the
> frustration of having to dive in to unbork the branch in the first
> place, rather than concentrate on the work you initially planned on
> doing.
> 
> That said, wearing my stable team member hat, I think it's a bit unfair
> to say that things are worse than they were and call for dramatic
> action. The stable branch team put a structure in place to try to
> continuously fix the stable branches rather than reactively fix it when
> we need it to work. Those champions have been quite active[1] unbreaking
> it in the past months. I'd argue that the branch is broken much less
> often than it used to. That doesn't mean it's never broken, though, or
> that those people are magicians.
> 
> One issue in the current situation is that the two groups (you and the
> stable maintainers) seem to work in parallel rather than collaborate.
> It's quite telling that the two groups maintained separate etherpads to
> keep track of the fixes that needed landing.

I agree we should figure out a way to communicate better between the
various teams involved in fixing the issues. Consolidating some of the
communication channels we use should help. For example, if we settle on
a single channel to use on IRC then we can log that channel so anyone
can review the history of a given issue, either to catch up on what
happened overnight or to understand how an issue was resolved long after
the fact. At one point we created #openstack-gate for this, but I don't
see many people there now and it's not being logged. Maybe we should
just use #openstack-dev, since that isn't as active for other purposes
now?

> 
> [1] https://etherpad.openstack.org/p/stable-tracker
> 
> Matthew Treinish wrote:
> > So I think it's time we called the icehouse branch and marked it EOL. We
> > originally conditioned the longer support window on extra people stepping
> > forward to keep things working. I believe this latest issue is just the latest
> > indication that this hasn't happened. Issue 1 listed above is being caused by
> > the icehouse branch during upgrades. The fact that a stable release was pushed
> > at the same time things were wedged on the juno branch is just the latest
> > evidence to me that things aren't being maintained as they should be. Looking at
> > the #openstack-qa irc log from today or the etherpad about trying to sort this
> > issue should be an indication that no one has stepped up to help with the
> > maintenance and it shows given the poor state of the branch.
> 
> I disagree with the assessment. People have stepped up. I think the
> stable branches are less often broken than they were, and stable branch
> champions (as their tracking etherpad shows) have made a difference.

They seem to be pretty consistently broken, at least lately, but the
causes have also been consistent. This cycle we changed the way we
manage libraries (dropping alpha release versions) and test tools
(branchless tempest) without fully understanding the effects those
changes would have throughout the complex testing systems we have in
place. As the changes have rippled through the system, we've found more
and more unintended consequences, some of which have required making
other big changes to the way test jobs are defined and to how we manage
requirements. I'm not convinced we would have identified all of those
issues even if we had spent more time reasoning through the changes
before making them, given the complexity. So, although it has been a
frustrating period, in retrospect I think there were probably a few
things we could have done differently early on but it was largely
necessary to do it this way to uncover the issues we couldn't predict.

I think we're approaching a good state, too. The series of patches Joe,
Matt, and Sean came up with yesterday should unwedge icehouse. Then we
can resume the work to cap the requirements lists, and that will avoid
breaking backwards compatibility test jobs with new releases of
libraries. We've also stopped testing master branches of libraries
against the stable branches where we won't be running the code, so
development on the master branches of those libraries should already be
unblocked (although we're not releasing anything until this is sorted
out, which is another sort of block).

> There just has been more issues as usual recently and they probably
> couldn't keep track. It's not a fun job to babysit stable branches,
> belittling the stable branch champions results is not the best way to
> encourage them to continue in this position. I agree that they could
> work more with the QA team when they get overwhelmed, and raise more red
> flags when they just can't keep up.
> 
> I also disagree with the proposed solution. We announced a support
> timeframe for Icehouse, our downstream users made plans around it, so we
> should stick to it as much as we can. If we dropped stable branch
> support every time a patch can't be landed there, there would just not
> be any stable branch.
> 
> Joe Gordon wrote:
> > Where is it documented that Adam is the Juno branch champion and Ihar is
> > Icehouse's? I didn't see it anywhere in the wiki.
> 
> It was announced here:
> http://lists.openstack.org/pipermail/openstack-dev/2014-November/050390.html
> 
> I agree it should be documented on the wiki.
> 
> -- 
> Thierry Carrez (ttx)
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev