Open Stack

On Tue, 2014-07-29 at 14:04 +0200, Thierry Carrez wrote:
> Ihar Hrachyshka a écrit :
> > On 29/07/14 12:15, Daniel P. Berrange wrote:
> >> Looking at the current review backlog I think that we have to
> >> seriously question whether our stable branch review process in
> >> Nova is working to an acceptable level
> > 
> >> On Havana
> > 
> >>   - 43 patches pending
> >>   - 19 patches with a single +2
> >>   - 1 patch with a -1
> >>   - 0 patches wit a -2
> >>   - Stalest waiting 111 days since most recent patch upload
> >>   - Oldest waiting 250 days since first patch upload
> >>   - 26 patches waiting more than 1 month since most recent upload
> >>   - 40 patches waiting more than 1 month since first upload
> > 
> >> On Icehouse:
> > 
> >>   - 45 patches pending
> >>   - 17 patches with a single +2
> >>   - 4 patches with a -1
> >>   - 1 patch with a -2
> >>   - Stalest waiting 84 days since most recent patch upload
> >>   - Oldest waiting 88 days since first patch upload
> >>   - 10 patches waiting more than 1 month since most recent upload
> >>   - 29 patches waiting more than 1 month since first upload
> > 
> >> I think those stats paint a pretty poor picture of our stable branch
> >> review process, particularly Havana.
> > 
> >> It should not take us 250 days for our review team to figure out whether
> >> a patch is suitable material for a stable branch, nor should we have
> >> nearly all the patches waiting more than 1 month in Havana.
> > 
> >> These branches are not getting sufficient reviewer attention and we need
> >> to take steps to fix that.
> > 
> >> If I had to set a benchmark, assuming CI passes, I'd expect us to either
> >> approve or reject submissions for stable within a 2 week window in the
> >> common case, 1 month at the worst case.
> > 
> > Totally agreed.
> 
> A bit of history.
> 
> At the dawn of time there were no OpenStack stable branches, each
> distribution was maintaining its own stable branches, duplicating the
> backporting work.

I'm not sure how much backporting was going on at the time of the Essex
summit. I'm sure Ubuntu had some backports, but that was probably about
it?

>  At some point it was suggested (mostly by RedHat and
> Canonical folks) that there should be collaboration around that task,
> and the OpenStack project decided to set up "official" stable branches
> where all distributions could share the backporting work. The stable
> team group was seeded with package maintainers from all over the distro
> world.

During that first design summit session, it was mainly you, me and
Daviey discussing. Both you and Daviey saw this primarily about distros
collaborating, but I never saw it that way.

I don't see how any self-respecting open-source project can throw a
release over the wall and have no ability to address critical bugs with
that release until the next release 6 months later which will also
include a bunch of new feature work with new bugs. That's not a distro
maintainer point of view.

At that Essex summit, we were lamenting how many critical bugs in Nova
had been discovered shortly after the Diablo release. Our inability to
do a bugfix release of Nova for Diablo seemed like a huge problem to me.

> So these branches originally only exist as a convenient place to
> collaborate on backporting work. This is completely separate from
> development work, even if those days backports are often proposed by
> developers themselves. The stable branch team is separate from the rest
> of OpenStack teams. We have always been very clear tht if the stable
> branches are no longer maintained (i.e. if the distributions don't see
> the value of those anymore), then we'll consider removing them. We, as a
> project, only signed up to support those as long as the distros wanted them.

You can certainly argue that the project never signed up for the
responsibility. I don't see it that way, but there was certainly always
a debate whether this was the project taking responsibility for bugfix
releases or whether it was just downstream distros collaborating.

The thing about branches going away if they're not maintained isn't
anything unusual. If *any* effort within the project becomes so
unmaintained due to a lack of interest such that we can't stand over it,
then we should consider retiring it.

> We have been adding new members to the stable branch teams recently, but
> those tend to come from development teams rather than downstream
> distributions, and that starts to bend the original landscape.
> Basically, the stable branch needs to be very conservative to be a
> source of safe updates -- downstream distributions understand the need
> to weigh the benefit of the patch vs. the disruption it may cause.
> Developers have another type of incentive, which is to get the fix they
> worked on into stable releases, without necessarily being very
> conservative. Adding more -core people to the stable team to compensate
> the absence of distro maintainers will ultimately kill those branches.

That's quite a leap to say that -core team members will be so incapable
of the appropriate level of conservatism that the branch will be killed.

The idea behind not automatically granting +2 on stable/* to -core
members was simply we didn't want people diving in and approving
unsuitable stuff out of ignorance.

I could totally see an argument now that everyone is a lot more familiar
with gerrit, the concept of stable releases is well established and we
should be able to trust -core team members to learn how to make the risk
vs benefit tradeoffs needed for stable reviews.

Mark.