Open Stack

Just when you thought this had been forgotten....

On 1 May 2013 21:46, Thierry Carrez <thierry at openstack.org> wrote:
> Clint Byrum wrote:
>> On 2013-04-30 14:52, Russell Bryant wrote:
>>> My biggest concern with this is applying it to an open source project
>>> with an unmanaged workforce.  Thierry captured it well when he said:
>>>
>>>   Personally my experience of OpenStack project management makes me fear
>>>   that your proposal ignores the reality of our unmanaged development
>>>   workforce: there is no way we can be sure that a feature will actually
>>>   be completed. So you might end up having disabled code in releases...
>>>   or force the removal of the already-landed parts at the very last
>>>   moment, which is disruptive to release quality and also sounds more
>>>   disruptive to CD than what you are trying to protect against.
>>>
>>> I'm really worried about the transition of ownership going from the
>>> submitter to the project as a whole much earlier in the process.  I'd
>>> like to have a high confidence level that it's worth having this code
>>> for the long run before it starts coming in.  That to me requires it
>>> baking out of tree and getting completed before we accept it.
>>
>> What is the negative impact of having disabled code in a release? I
>> think we would need a good reason before we would back accepted code
>> out. If the code is just sitting there, undocumented and turned off, not
>> interfering with anything, I'm having a hard time thinking of a logical
>> reason to back it out.
>
> I think the root of the problem with shipping incomplete features
> disabled by default is the stable maintenance expectations. Is that
> incomplete feature considered a bug in the stable branch ? The code
> might be disabled, it can still be enabled... Could we mark that code in
> a way that make it sufficiently clear it's not supported by the stable
> branch team ? I don't really want to make it easier for CD at the
> expense of the stable branch team. Or turn it into a public cloud
> deployer vs. private cloud distros argument.

https://blog.twitter.com/2013/new-tweets-per-second-record-and-how
Is a good read for anyone interested in large scale systems.

'Decider', referenced in that post is a classic service in an agile
SOA environment : it's a centralised 'is X enabled' store, which can
turn X on or off on a per-request basis.

But to answer your points:
 - an unreleased feature is an unreleased feature irrespective of what
branch it is in. It's not something you'd pull bug fixes into a frozen
branch for.
 - If the codebase knows it's a release branch (can we tell
programmatically?) we could just refuse to enable unreleased features.

I understand your concerns about making it easier for one group vs
another - I don't think this is a zero-sum game. CD disciplines make
code evolution faster and safer for /everyone/.

Since it's been a few months since I pushed on this, let me
re-summarise. The key goals are:

 - Flatten out risk spikes around code landings
 - Make it more reliable for trunk deployers to deploy trunk
 - Make it easier for folk working on features to get those features in

The key bad outcomes folk want to see addressed are:
 - The project owning lots of incomplete useless code
 - Bugs being introduced
 - Documentation and user support burdens being increased
 - Conf file setting proliferation [with backwards compat constraints
on removals]
 - Stable branch maintenance becoming harder.

The set of proposals being made to tackle this are:
 - Set a much harder upper bound on commit size - we were saying 500
lines, but the recent research paper suggests that saying 200 lines as
target, with rubber band permitting up to 400 lines before we push
back really hard.
   - This reduces the risk on any one commit.
   - It increases defect detection during review.
   - It helps reviewers review the change.
   - It helps trunk deployers assess risk more easily too.

 - Land patches when the patch is ready, not when the feature is ready.
   - All the normal quality standards : correctness, performance,
style, relevance and desirability still apply!
   - This prevents the patch being stuck in rebase-forever mode,
continually chasing backwards incompatible changes in internal APIs
(which may be rare - I haven't look for stats on it, but it's super
super painful when it happens, and where there is pain folk get weary
and stop programming well... which increases defect rate).
   - This reduces the 'and we delayed refactoring X until release, now
CHANGE THE WORLD' anti-patterns, which break stable maintenance. [I
don't know if we experience that].

 - Where specific code is considered experimental and not ready for
backwards compat support, annotate it as such... the exact mechanism
hasn't been designed.
   - Addresses documentation, user support and stable branch maintenance.

 - Where specific code should be opt in, make it so.
   - Either by a new entry point - like the v3 nova API. [BTW
brilliant stuff there - that is pure CD :)]
   - Or by a condition in the code that deployers/testers/eaaaaarly
adopters can enable via config settings.
     This will need a small patch to the sample config generator to
let it skip experimental things so they don't end up in documentation
etc.
   - Or by introspection [e.g. new optional kwarg parameter].

So - are there remaining objections to any of these things? What have I missed?

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud