Open Stack

From: Morgan Fainberg <morgan.fainberg at gmail.com<mailto:morgan.fainberg at gmail.com>>
Date: Thursday, January 29, 2015 at 12:26 PM
To: openstack-operators <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: [Openstack-operators] [all] SQL Schema Downgrades: A Good Idea?

>From an operator perspective I wanted to get input on the SQL Schema Downgrades.
>
>Today most projects (all?) provide a way to downgrade the SQL Schemas after you’ve upgraded. Example would be moving from Juno to Kilo and then back to Juno. There are some odd concepts >when handling a SQL migration downgrade specifically around the state of the data. A downgrade, in many cases, causes permanent and irrevocable data loss. When phrased like that (and >dusting off my deployer/operator hat) I would be hesitant to run a downgrade in any production, stagings, or even QA environment.
>
>In light of what a downgrade actually means I would like to get the views of the operators on SQL Migration Downgrades:
>
>1) Would you actually perform a programatic downgrade via the cli tools or would you just do a restore-to-last-known-good-before-upgrade (e.g. from a DB dump)?
>2) Would you trust the data after a programatic downgrade or would the data only really be trustworthy if from a restore? Specifically the new code *could* be relying on new data structures and >a downgrade could result in weird states of services.
>
>I’m looking at the expectation that a downgrade is possible. Each time I look at the downgrades I feel that it doesn’t make sense to ever really perform a downgrade outside of a development >environment. The potential for permanent loss of data / inconsistent data leads me to believe the downgrade is a flawed design. Input from the operators on real-world cases would be great to >have.
>
>This is an operator specific set of questions related to a post I made to the OpenStack development mailing list: http://lists.openstack.org/pipermail/openstack-dev/2015-January/055586.html
>
>Cheers,
>Morgan

When moving major releases, we  backup our databases and shutdown most of the cluster so that portion of the cluster is still “good”. We then upgrade one node completely, validate it, then join the rest of the nodes back in. If it goes horribly wrong at that point we’d restore from backup. The main reasons for me are two fold. First, rightly or wrongly, I assume that downgrades are not well tested and rarely used by anyone. We certainly never test it during our upgrade planning. Secondly, until I’m sure that the upgrade worked well, I’d rather just go back to a clean state than rely on a downgrade just because I know that state is 100% functional without further validation. Especially if I’m in an outage window I don’t have time to mess around with a downgrade and hope it works. I’ll kill the bad node and rebuild it, either just restarting the old DB or restoring if needed. The tl;dr here is why take the chance that the downgrade works when you have saner alternatives.


(please excuse the cruft below)





________________________________
This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150129/3deef1e6/attachment.html>