Open Stack

David Kranz wrote:
> [...]
> There is a different way to do this. We could adopt the same methodology
> we have now around gating, but applied to each project on its own
> branch. These project branches would be integrated into master at some
> frequency or when some new feature in project X is needed by project Y. 
> Projects would want to pull from the master branch often, but the push
> process would be less frequent and run a much larger battery of tests
> than we do now.

So we would basically discover the cross-project bugs when we push to
the "master master" branch. I think you're just delaying discovery of
the most complex issues, and push the responsibility to resolve them
onto a inexistent set of people. Adding integration branches only makes
sense if you have an integration team. We don't have one, so we'd call
back on the development teams to solve the same issues... with a delay.

In our specific open development setting, delaying is bad because you
don't have a static set of developers that you can assume will be on
call ready to help with what they have written a few months later:
shorter feedback loops are key to us.

> Doing this would have the following advantages:
> 
> 1. It would be much harder for a race bug to get in. Each commit would
> be tested many more times on its branch before being merged to master
> than at present, including tests specialized for that project. The
> qa/infra teams and others would continue to define acceptance at the
> master level.
> 2. If a race bug does get in, projects have at least some chance to
> avoid merging the bad code.
> 3. Each project can develop its own gating policy for its own branch
> tailored to the issues and tradeoffs it has. This includes focus on
> spending time running their own tests. We would no longer run a complete
> battery of nova tests on every commit to swift.
> 4. If a project branch gets into the situation we are now in:
>      a) it does not impact the ability of other projects to merge code
>      b) it is highly likely the bad code is actually in the project so
> it is known who should help fix it
>      c) those trying to fix it will be domain experts in the area that
> is failing
> 5. Distributing the gating load and policy to projects makes the whole
> system much more scalable as we add new projects.
> 
> Of course there are some drawbacks:
> 
> 1. It will take longer, sometimes much longer, for any individual commit
> to make it to master. Of course if a super-serious issue made it to
> master and had to be fixed immediately it could be committed to master
> directly.
> 2. Branch management at the project level would be required. Projects
> would have to decide gating criteria, timing of pulls, and coordinate
> around integration to master with other projects.
> 3. There may be some technical limitations with git/gerrit/whatever that
> I don't understand but which would make this difficult.
> 4. It makes the whole thing more complicated from a process standpoint.

An extra drawback is that you can't really do CD anymore, because your
"master master" branch gets big chunks of new code in one go at push time.

> I have used this model in previous large software projects and it worked
> quite well. This may also be somewhat similar to what the linux kernel
> does in some ways.

Please keep in mind that some techniques which are perfectly valid (and
even recommended) when you have a captive set of developers just can't
work in our open development setting. Some techniques which work
perfectly for a release-oriented product just don't cut it when you also
want the software to be consumable in a continuous delivery fashion. We
certainly can and should learn from other experiences, but we also need
to recognize our challenges are unique and might call for some unique
solution, with its own drawbacks and benefits.

-- 
Thierry Carrez (ttx)