[openstack-dev] Continuous deployment - significant process change

Joe Gordon jogo at cloudscaling.com
Mon Apr 29 21:58:21 UTC 2013

On Mon, Apr 29, 2013 at 2:04 PM, Robert Collins
<robertc at robertcollins.net>wrote:

> We had a process track session about bringing in upstream continuous
> deployment for openstack.
> https://etherpad.openstack.org/HavanaContinuousDeployment
> I suspect that while the session was good with both deployer and
> distributor attendees, we need to do more to make it happen, as it
> impinges on review / testing / backwards compat requirements for every
> project.
> Note that CD doesn't require no-downtime deployments, CD is about
> being able to adopt *any arbitrary revision of trunk* at *any point in
> time*. The engineering required to do deployments without disruption
> is beneficial to both CD and per-release deployments.
> Here are the key takeaways we came up with:
>  * No more big landings [except the purely mechanical]. Set a hard
> limit - maybe 500 lines of diff. Big landings are more risky per line
> of diff than small ones due to reviewer cognitive overhead - reviewers
> get non-linearly less effective the larger the review.

While I like this one, it sounds very hard to enforce in reality.

>  * CD can be done many ways; we need to gate the *specific* ways that
> upstream adopts, as soon as possible. Thats a -infra thing, and there
> are already discussions on it. We don't need to support *every
> possible config for CD*. Organisations interested in a particular
> configuration(s) will need to contribute resources to permit
> gate-quality checks of those configurations.
>  * No more cramming: when a freeze is happening, anything that is
> 'land this for the release' has to be pushed back on -really hard-. If
> its not ready, it's not ready.

Has this been an issue recently? I thought we were getting better with this.

>  * -never- choose to break something that is neither experimental nor
> deprecated since the last release. If an accident happens, correct it
> as quickly as possible.
>  * Land features disabled by default. Such disabled features are
> experimental and we don't need to be so careful - we can in fact
> remove them entirely if they turn out to be bad idea - when they
> become supported (individual teams can define this we think) they
> can't be broken again though: they are now part of the product.

This touches on a bigger issue, what is experimental and what is not.
 Currently we don't do a good job of differentiating the two.  I think we
first need to flesh out what it means to be experimental for users and for

>  * 'To break' means just that - it could be an exception, it could be
> a massive jump in DB utilisation, or latency. Whatever our criteria
> are for 'fit for use', breaking something stops it being fit for use
> in *existing deployed environments*.
> How do we get all this into place : I can update wiki pages etc, but
> is any more agreement needed? Should the TC eyeball it?
> -Rob
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Cloud Services
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130429/4a0e47c0/attachment.html>

More information about the OpenStack-dev mailing list