Open Stack

Ihar Hrachyshka a écrit :
> On 29/07/14 12:15, Daniel P. Berrange wrote:
>> Looking at the current review backlog I think that we have to
>> seriously question whether our stable branch review process in
>> Nova is working to an acceptable level
> 
>> On Havana
> 
>>   - 43 patches pending
>>   - 19 patches with a single +2
>>   - 1 patch with a -1
>>   - 0 patches wit a -2
>>   - Stalest waiting 111 days since most recent patch upload
>>   - Oldest waiting 250 days since first patch upload
>>   - 26 patches waiting more than 1 month since most recent upload
>>   - 40 patches waiting more than 1 month since first upload
> 
>> On Icehouse:
> 
>>   - 45 patches pending
>>   - 17 patches with a single +2
>>   - 4 patches with a -1
>>   - 1 patch with a -2
>>   - Stalest waiting 84 days since most recent patch upload
>>   - Oldest waiting 88 days since first patch upload
>>   - 10 patches waiting more than 1 month since most recent upload
>>   - 29 patches waiting more than 1 month since first upload
> 
>> I think those stats paint a pretty poor picture of our stable branch
>> review process, particularly Havana.
> 
>> It should not take us 250 days for our review team to figure out whether
>> a patch is suitable material for a stable branch, nor should we have
>> nearly all the patches waiting more than 1 month in Havana.
> 
>> These branches are not getting sufficient reviewer attention and we need
>> to take steps to fix that.
> 
>> If I had to set a benchmark, assuming CI passes, I'd expect us to either
>> approve or reject submissions for stable within a 2 week window in the
>> common case, 1 month at the worst case.
> 
> Totally agreed.

A bit of history.

At the dawn of time there were no OpenStack stable branches, each
distribution was maintaining its own stable branches, duplicating the
backporting work. At some point it was suggested (mostly by RedHat and
Canonical folks) that there should be collaboration around that task,
and the OpenStack project decided to set up "official" stable branches
where all distributions could share the backporting work. The stable
team group was seeded with package maintainers from all over the distro
world.

So these branches originally only exist as a convenient place to
collaborate on backporting work. This is completely separate from
development work, even if those days backports are often proposed by
developers themselves. The stable branch team is separate from the rest
of OpenStack teams. We have always been very clear tht if the stable
branches are no longer maintained (i.e. if the distributions don't see
the value of those anymore), then we'll consider removing them. We, as a
project, only signed up to support those as long as the distros wanted them.

We have been adding new members to the stable branch teams recently, but
those tend to come from development teams rather than downstream
distributions, and that starts to bend the original landscape.
Basically, the stable branch needs to be very conservative to be a
source of safe updates -- downstream distributions understand the need
to weigh the benefit of the patch vs. the disruption it may cause.
Developers have another type of incentive, which is to get the fix they
worked on into stable releases, without necessarily being very
conservative. Adding more -core people to the stable team to compensate
the absence of distro maintainers will ultimately kill those branches.

>> If we are trying to throttle down the rate of change in Havana, that
>> totally makes sense, but we should be more active at rejecting patches
>> if that is our current goal, not let them hang around in limbo for
>> many months.
> 
> Tip: to be notified in time about new backport requests, you may add
> those branches you're interested in to watched, in Gerrit, go to
> Settings -> Watched Projects, and add whatever you like. Then you'll
> receive emails for each backport request.
> 
> 
>> I'm actually unclear on who even has permission to approve patches
>> on stable branches ? Despite being in Nova core I don't have any perm
>> to approve patches on stable. I think it is pretty odd that we've got a
>> system where the supposed experts of the Nova team can't approve patches
>> for stable. I get that we've probably got people on stable team who are
>> not in core, but IMHO we should have the stable team comprising a superset
>> of core, not a subset.
> 
> AFAIK stable team consists of project PTLs + people interested in stable
> branches specifically (that got added to the team after their request).
> Anyone can start reviewing the patches and ask to be added to the team.
> 
> I also think it's weird that project cores don't have +2 for stable
> branches of their projects. They do not require global +2 for all stable
> branches though.

The key reason why $PROJECT-core don't automatically get stable branch
+2 is that the rules for accepting a patch there are VERY different from
the rules for accepting a patch for master, and most -core people don't
know those.

We need to ensure those -core people know the stable branch acceptance
rules before we grant them +2 there.

-- 
Thierry Carrez (ttx)