[openstack-dev] [nova] stable branches & failure to handle review backlog

Sean Dague sean at dague.net
Wed Jul 30 13:48:37 UTC 2014


On 07/30/2014 02:21 AM, Daniel P. Berrange wrote:
> On Wed, Jul 30, 2014 at 11:16:00AM +0200, Ihar Hrachyshka wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> On 29/07/14 18:12, Daniel P. Berrange wrote:
>>> On Tue, Jul 29, 2014 at 08:30:09AM -0700, Jay Pipes wrote:
>>>> On 07/29/2014 06:13 AM, Daniel P. Berrange wrote:
>>>>> On Tue, Jul 29, 2014 at 02:04:42PM +0200, Thierry Carrez
>>>>> wrote:
>>>>>> Ihar Hrachyshka a écrit : At the dawn of time there were no
>>>>>> OpenStack stable branches, each distribution was maintaining
>>>>>> its own stable branches, duplicating the backporting work. At
>>>>>> some point it was suggested (mostly by RedHat and Canonical
>>>>>> folks) that there should be collaboration around that task, 
>>>>>> and the OpenStack project decided to set up "official" stable
>>>>>> branches where all distributions could share the backporting
>>>>>> work. The stable team group was seeded with package
>>>>>> maintainers from all over the distro world.
>>>>>>
>>>>>> So these branches originally only exist as a convenient place
>>>>>> to collaborate on backporting work. This is completely
>>>>>> separate from development work, even if those days backports
>>>>>> are often proposed by developers themselves. The stable
>>>>>> branch team is separate from the rest of OpenStack teams. We
>>>>>> have always been very clear tht if the stable branches are no
>>>>>> longer maintained (i.e. if the distributions don't see the
>>>>>> value of those anymore), then we'll consider removing them.
>>>>>> We, as a project, only signed up to support those as long as
>>>>>> the distros wanted them.
>>>>>>
>>>>>> We have been adding new members to the stable branch teams
>>>>>> recently, but those tend to come from development teams
>>>>>> rather than downstream distributions, and that starts to bend
>>>>>> the original landscape. Basically, the stable branch needs to
>>>>>> be very conservative to be a source of safe updates --
>>>>>> downstream distributions understand the need to weigh the
>>>>>> benefit of the patch vs. the disruption it may cause. 
>>>>>> Developers have another type of incentive, which is to get
>>>>>> the fix they worked on into stable releases, without
>>>>>> necessarily being very conservative. Adding more -core people
>>>>>> to the stable team to compensate the absence of distro
>>>>>> maintainers will ultimately kill those branches.
>>>>>
>>>>> The situation I'm seeing is that the broader community believe
>>>>> that the Nova core team is responsible for the nova stable
>>>>> branches. When stuff sits in review for ages it is the core
>>>>> team that is getting pinged about it and on the receiving end
>>>>> of the complaints the inaction of review.
>>>>>
>>>>> Adding more people to the stable team won't kill those
>>>>> branches. I'm not suggesting we change the criteria for
>>>>> accepting patches, or that we dramatically increase the number
>>>>> of patches we accept. There is clearly alot of stuff proposed
>>>>> to stable that the existing stable team think is a good idea -
>>>>> as illustrated by the number of patches with at least one +2
>>>>> present. On the contrary, having a bigger stable team comprises
>>>>> all of core + interested distro maintainers will ensure that
>>>>> the stable branches are actually gettting the patches people
>>>>> in the field need to provide a stable cloud.
>>>>
>>>> -1
>>>>
>>>> In my experience, the distro maintainers who pioneered the stable
>>>> branch teams had opposite viewpoints to core teams in regards to
>>>> what was appropriate to put into a stable release. I think it's
>>>> dangerous to populate the stable team with the core team members
>>>> just because of long review and merge times.
>>>
>>> Sure there was some debate about what criteria were desired
>>> acceptance when stable trees were started. Once the criteria are
>>> defined I don't think it is credible to say that people are
>>> incapable of following the rules. In the unlikely event that people
>>> were to willfully ignore the agreed upon rules for stable tree,
>>> then I'd not trust them to be part of a core team working on any
>>> branch at all. With responsibility comes trust and an acceptance to
>>> follow the agreed upon processes.
>>
>> Still, it's quite common to see patches with no cherry-pick info, or
>> Conflicts section removed, or incorrect Change-Id being approved and
>> pushed. It's also common for people to review backports as if they are
>> sent against master (with all nit picking, unit test changes
>> requested, no consideration for stable branch applicability...)
> 
> As said before, if people reviewing the code consistently fail to
> follow the agreed review guidelines, then point out what they're
> doing wrong and if they fail to improve, remove their +2 privileges.
> There's nothing unique about stable branches in this regard. If
> people reviewing master consistently did the wrong thing they'd
> have their +2 removed too.

Honestly, I think if anyone wants to be added to stable-maint, they
should raise their hand and join the team. I'm sure they would be welcomed.

I don't feel that stable maintenance is what we actually asked the core
teams to do as a community.

The risks are much higher for people approving stable backports, so it
should be something people explicitly get in the mindset to do.

I think it's also worth figuring out if the stable backports are
growing, and why. Because if they are only bug fixing, they should be
the exception. If there are things we're doing a bad job catching on
code correctness in master before it gets to the user, we should
probably spend more time on how we address that, vs. just fixing after
it's broken real people. Addressing these things close to the origin
creates less pain for everyone.

	-Sean

-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140730/a1cb0720/attachment.pgp>


More information about the OpenStack-dev mailing list