[openstack-dev] Continuous deployment - significant process change

Russell Bryant rbryant at redhat.com
Mon Apr 29 21:50:37 UTC 2013


On 04/29/2013 05:04 PM, Robert Collins wrote:
> We had a process track session about bringing in upstream continuous
> deployment for openstack.
> https://etherpad.openstack.org/HavanaContinuousDeployment
> 
> I suspect that while the session was good with both deployer and
> distributor attendees, we need to do more to make it happen, as it
> impinges on review / testing / backwards compat requirements for every
> project.
> 
> Note that CD doesn't require no-downtime deployments, CD is about
> being able to adopt *any arbitrary revision of trunk* at *any point in
> time*. The engineering required to do deployments without disruption
> is beneficial to both CD and per-release deployments.
> 
> Here are the key takeaways we came up with:
>  * No more big landings [except the purely mechanical]. Set a hard
> limit - maybe 500 lines of diff. Big landings are more risky per line
> of diff than small ones due to reviewer cognitive overhead - reviewers
> get non-linearly less effective the larger the review.
> 

I don't think we can set a # of lines that always makes sense.  However,
I feel like in Nova we already do a nice job of pushing back hard on
large patches in favor of breaking them up into a reasonable patch series.

https://wiki.openstack.org/wiki/GitCommitMessages

So, at least for Nova, this is business as usual.

>  * CD can be done many ways; we need to gate the *specific* ways that
> upstream adopts, as soon as possible. Thats a -infra thing, and there
> are already discussions on it. We don't need to support *every
> possible config for CD*. Organisations interested in a particular
> configuration(s) will need to contribute resources to permit
> gate-quality checks of those configurations.
> 
>  * No more cramming: when a freeze is happening, anything that is
> 'land this for the release' has to be pushed back on -really hard-. If
> its not ready, it's not ready.
> 
>  * -never- choose to break something that is neither experimental nor
> deprecated since the last release. If an accident happens, correct it
> as quickly as possible.
> 
>  * Land features disabled by default. Such disabled features are
> experimental and we don't need to be so careful - we can in fact
> remove them entirely if they turn out to be bad idea - when they
> become supported (individual teams can define this we think) they
> can't be broken again though: they are now part of the product.
> 
>  * 'To break' means just that - it could be an exception, it could be
> a massive jump in DB utilisation, or latency. Whatever our criteria
> are for 'fit for use', breaking something stops it being fit for use
> in *existing deployed environments*.

Most of this seems good.  It seems as if there's tension between:

1) Ensuring utmost quality so you can deploy any revision

and

2) Being less careful with some new features and justifying it by having
them disabled by default.

I don't see the value in this disabled-by-default thing.  Another one of
your points stresses the fact that if something's not ready, it's not
ready, which I definitely agree with.

-- 
Russell Bryant



More information about the OpenStack-dev mailing list