[openstack-dev] Continuous deployment - significant process change

Russell Bryant rbryant at redhat.com
Mon Apr 29 21:50:37 UTC 2013

On 04/29/2013 05:04 PM, Robert Collins wrote:
> We had a process track session about bringing in upstream continuous
> deployment for openstack.
> https://etherpad.openstack.org/HavanaContinuousDeployment
> I suspect that while the session was good with both deployer and
> distributor attendees, we need to do more to make it happen, as it
> impinges on review / testing / backwards compat requirements for every
> project.
> Note that CD doesn't require no-downtime deployments, CD is about
> being able to adopt *any arbitrary revision of trunk* at *any point in
> time*. The engineering required to do deployments without disruption
> is beneficial to both CD and per-release deployments.
> Here are the key takeaways we came up with:
>  * No more big landings [except the purely mechanical]. Set a hard
> limit - maybe 500 lines of diff. Big landings are more risky per line
> of diff than small ones due to reviewer cognitive overhead - reviewers
> get non-linearly less effective the larger the review.

I don't think we can set a # of lines that always makes sense.  However,
I feel like in Nova we already do a nice job of pushing back hard on
large patches in favor of breaking them up into a reasonable patch series.


So, at least for Nova, this is business as usual.

>  * CD can be done many ways; we need to gate the *specific* ways that
> upstream adopts, as soon as possible. Thats a -infra thing, and there
> are already discussions on it. We don't need to support *every
> possible config for CD*. Organisations interested in a particular
> configuration(s) will need to contribute resources to permit
> gate-quality checks of those configurations.
>  * No more cramming: when a freeze is happening, anything that is
> 'land this for the release' has to be pushed back on -really hard-. If
> its not ready, it's not ready.
>  * -never- choose to break something that is neither experimental nor
> deprecated since the last release. If an accident happens, correct it
> as quickly as possible.
>  * Land features disabled by default. Such disabled features are
> experimental and we don't need to be so careful - we can in fact
> remove them entirely if they turn out to be bad idea - when they
> become supported (individual teams can define this we think) they
> can't be broken again though: they are now part of the product.
>  * 'To break' means just that - it could be an exception, it could be
> a massive jump in DB utilisation, or latency. Whatever our criteria
> are for 'fit for use', breaking something stops it being fit for use
> in *existing deployed environments*.

Most of this seems good.  It seems as if there's tension between:

1) Ensuring utmost quality so you can deploy any revision


2) Being less careful with some new features and justifying it by having
them disabled by default.

I don't see the value in this disabled-by-default thing.  Another one of
your points stresses the fact that if something's not ready, it's not
ready, which I definitely agree with.

Russell Bryant

More information about the OpenStack-dev mailing list