Open Stack

On 17/08/15 22:29, James Slagle wrote:
> On Mon, Aug 17, 2015 at 9:28 AM, Steven Hardy <shardy at redhat.com> wrote:
>> Hi all,
>>
>> Recently I had some discussion with folks around the future strategy for
>> TripleO wrt upgrades, releases and branches, specifically:
>>
>> - How do we support a "stable" TripleO release/branch that enables folks to
>>   easily deploy the current stable release of OpenStack
>> - Related to the above, how do we allow development of TripleO components
>>   (and in particular t-h-t) to proceed without imposing undue constraints
>>   on what new features may be used (e.g new-for-liberty Heat features which
>>   aren't present in the current released OpenStack version)
>> - We're aiming to provide upgrade support, thus from and to which versions?
>>
>> I know historically TripleO has taken something of a developer and/or
>> continuous deployment model for granted, but I'd like to propose that we
>> revisit that discusion, such that we move towards something that's more
>> consumable by users/operators that are consuming the OpenStack coordinated
>> releases.
>>
>> The obvious answer is a stable branch for certain TripleO components, and
>> in particular for t-h-t, but this has disadvantages if we take the
>> OpenStack wide "no feature backports" approach - for example "upgrade
>> support to liberty" could be considered a feature, and many other TripleO
>> "features" are really more about making features of the deployed OpenStack
>> services consumable.
>>
>> I'd like propose we take a somewhat modified "release branch" approach,
>> which combines many of the advantages of the stable-branch model, but
>> allows for a somewhat more liberal approach to backports, where most things
>> are considered valid backports provided they work with the currently
>> released OpenStack services (e.g right now, a t-h-t release/kilo branch
>> would have to maintain compatibility with a kilo Heat in the undercloud)
> 
> I like the idea, it seems reasonable to me.
> 
> I do think we should clarify if the rule is:
> 
> We *can* backport anything to release/kilo that doesn't break
> compatibility with kilo Heat.
> 
> Or:
> 
> We *must* backport anything to release/kilo that doesn't break
> compatibility with kilo Heat.
> 
> If it's the former, I think we'd get into a lot of subjective
> discussions around if we want certain things backported or not.
> Essentially it's the same discussion that happens for stable/*, except
> we consider features as well. This could become quite difficult to
> manage, and lead to a lot of reviewer opinionated inconsistency into
> what actually ends up getting backported.
> 
> For instance, there could be a very large and disruptive feature that
> doesn't break compatibility at all, but some users may not want to see
> it in release/kilo. Or, something like the recent proposed patch to
> rename a bunch of templates by dropping the "-puppet". That doesn't
> break compatibility with a kilo Heat at all, however it could break
> compatibility with someone's scripts or external tooling, and might be
> a considered an "API" incompatible change. The consuming downstreams
> (RDO) may not want to consume such a change. I know we don't have any
> official published "API" for tripleo-heat-templates, I'm just trying
> to think about how people consume the templates, and what they might
> find surprising if they were to be using release/kilo.
> 
> The question kind of comes down to if release/kilo is meant to imply
> any "stability". Or, if it's just meant to imply that you will always
> be able to deploy OpenStack Kilo with release/kilo of
> tripleo-heat-templates.

yes, my understanding too is that the 'compatibility' guarantee would be
a big part of this idea - one problem we consistently hit in the last
period was not being able to use (latest master of) component x with
(latest master of) tripleo-heat-templates (for example). Using
release/latest across all the projects would mean you'd use the latest
versions of things that have actually been ci'd and seen to work with
each other to give you an OpenStack latest deployment.

> 
> I think it's important to decide this up front so we can set the
> expectation. I'm leaning towards the latter ("must backport") myself,
> but then I wonder if the release branch would really solve the
> downstream use.

+1 on the must (sorry gfidente, wanted to reply on yours but wanted to
quote slagle in response)

Your point about does it really solve the downstream use still stands in
the can case though (even more so, e.g. one downstream influencing which
patches "can" go to release branch).

> 
>>
>> I know in the past stable branches have been discounted due to capacity
>> concerns, but the reality atm is such branches are likely to be maintained
>> downstream due to release-orientated efforts based on TripleO (e.g
>> rdo-manager) anyway, so it could be better for the TripleO community if we
>> maintained such branches upstream, where they can be consumed more directly
>> by such downstream projects.
> 
> I think getting tripleo-ci set up using the release branch would be a
> requirement. When I didn't do this previously with the stable branches
> attempt, that's when they ended up getting stale and cumbersome to
> maintain.
> 
>>
>> This could have several benefits:
>>
>> - TripleO can much more easily offer an easy end-to-end deployment solution
>>   for released OpenStack versions, e.g you always use release/kilo to
>>   deploy kilo OpenStack, and in future steps may be defined for upgrading
>>   between branches when upgrades are implemented and we support upgrading
>>   e.g from kilo to liberty or whatever.
>> - RDO (and RDO Manager in particular) can consume the "release" branches to
>>   provide package-based TripleO, and effort won't be potentially duplicated
>>   over downstream efforts, since we maintain the release-orientated TripleO
>>   components directly upstream.  Obviously this benefits any projects
>>   downstream in a similar way.
> 
> If we are more selective about what we backport and don't require
> every change to be backported, then I think the release branch likely
> turns into a branch that is only supporting a single downstream well.
> That's probably fine in the short term, but not what we want in the
> long term.
> 

Yes agree, ultimately what this is trying to solve is preventing/easing
the pain of the case where my_deployment_tool won't work with
release/latest of OpenStack, but only if the people writing
my_deployment_tool are influencing the OpenStack release/latest branch.
If we are using the must land then they have no say anyway so the
pressure is on them to fixup their tooling.

> If we were to require a backport of every compatible change, would the
> release branch still be something that downstreams such as RDO would
> want to consume? There could be a lot of churn and refactorings going
> into the release branch.

I think so, in the sense one thing that happened a lot recently was
incompatibility between even OpenStack projects themselves (and not
necessarily between downstream_deployment_tool and OpenStack projects) -
i.e. I still see the value in being able to use release/kilo and know
there won't be incompatibility issues across the release/kilo branches
of projects.

I think we really need to get someone like mburns to read this

> 
> Again, I just go back to the point of the branch. Does it exist to
> provide some semblance of stability so that we make distros happy? Or
> does it exist solely for the purpose so that we can iterate faster on
> using new Heat features in master?
> 
>>
>> One way to minimise the overhead of maintaing such a branch could be to
>> implement a bot which automatically proposes commits which land in master
>> to the branch (using Depends-On to maintain the history order).
> 
> I'm not sure I'm following how this would work. Which patch depends on
> which other one? If the master patch is A'd, is the release branch
> automatically +A'd as well (as long as it's not -2'd)? It seems like
> that might be necessary to maintain consistent looking history between
> master and the release branch.

I think auto propose != auto land, in other words, the bot or commit
message tag means patches that *land* in master are then only *proposed*
as reviews for release branch. They still have to be manually voted in
right? If this is the case then it will be dificult to maintain an exact
consistency in history (e.g. ordering of commits landing in master vs
being voted into release).

> 
> Likewise, if the release branch were to fail to merge, it would need
> to block the master patch from merging so that there wasn't potential
> for different patches to merge out of order in the 2 branches.

If my understanding is correct this can't happen, in the sense that
proposed reviews to release branch are made for patches that already
merge into master. As above, yes, this causes problems if we really need
exact consistency in commit history (though can that really be the case.
Isn't the whole point that release branch can diverge from master to
prevent incompatible breakage against other component?).


> 
>>
>> Reviewers would then monitor the release/kilo queue, and -2 any changes
>> which aren't appropriate to backport (e.g use new Heat features or some
>> other backwards incompatiblity), which would cause the bot to drop the
>> patch and rebase.  An alternative to this would be a commit message tag
>> which caused the commit to be picked up (or not).
>>
>> Obviously there will be merge conflicts which require manual fixup
>> sometimes (in which case the bot would commit with the conflicts block
>> intact and propose the patch for manual fixup).
>>
>> Interested to hear feedback on the idea, as well as if anyone has direct
>> experience of implementing the bot/automation pieces.

count me in if we don't have a resident expert//has_time and it would be
useful for someone to look into this.

thanks!

>>
>> Steve
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
>