Open Stack

I've mentioned this elsewhere but writing here for posterity...

Making N to N+1 upgrades seamless and work well is already challenging
today which is one of the reasons why people aren't upgrading in the
first place.
Making N to N+1 upgrades work as well as possible already puts a great
strain on developers and resources, think about the testing and CI
involved in making sure things really work.

My opinion is that of upgrades were made out to be a simple, easy and
seamless operation, it wouldn't be that much of a problem to upgrade
from N to N+3 by upgrading from release to release (three times) until
you've caught up.
But then, if upgrades are awesome, maybe operators won't be lagging 3
releases behind anymore.


David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]


On Thu, May 25, 2017 at 9:55 PM, Carter, Kevin <kevin at cloudnull.com> wrote:
> Hello Stackers,
>
> As I'm sure many of you know there was a talk about doing "skip-level"[0]
> upgrades at the OpenStack Summit which quite a few folks were interested in.
> Today many of the interested parties got together and talked about doing
> more of this in a formalized capacity. Essentially we're looking for cloud
> upgrades with the possibility of skipping releases, ideally enabling an N+3
> upgrade. In our opinion it would go a very long way to solving cloud
> consumer and deployer problems it folks didn't have to deal with an upgrade
> every six months. While we talked about various issues and some of the
> current approaches being kicked around we wanted to field our general chat
> to the rest of the community and request input from folks that may have
> already fought such a beast. If you've taken on an adventure like this how
> did you approach it? Did it work? Any known issues, gotchas, or things folks
> should be generally aware of?
>
>
> During our chat today we generally landed on an in-place upgrade with known
> API service downtime and little (at least as little as possible) data plane
> downtime. The process discussed was basically:
> a1. Create utility "thing-a-me" (container, venv, etc) which contains the
> required code to run a service through all of the required upgrades.
> a2. Stop service(s).
> a3. Run migration(s)/upgrade(s) for all releases using the utility
> "thing-a-me".
> a4. Repeat for all services.
>
> b1. Once all required migrations are complete run a deployment using the
> target release.
> b2. Ensure all services are restarted.
> b3. Ensure cloud is functional.
> b4. profit!
>
> Obviously, there's a lot of hand waving here but such a process is being
> developed by the OpenStack-Ansible project[1]. Currently, the OSA tooling
> will allow deployers to upgrade from Juno/Kilo to Newton using Ubuntu 14.04.
> While this has worked in the lab, it's early in development (YMMV). Also,
> the tooling is not very general purpose or portable outside of OSA but it
> could serve as a guide or just a general talking point. Are there other
> tools out there that solve for the multi-release upgrade? Are there any
> folks that might want to share their expertise? Maybe a process outline that
> worked? Best practices? Do folks believe tools are the right way to solve
> this or would comprehensive upgrade documentation be better for the general
> community?
>
> As most of the upgrade issues center around database migrations, we
> discussed some of the potential pitfalls at length. One approach was to
> roll-up all DB migrations into a single repository and run all upgrades for
> a given project in one step. Another was to simply have mutliple python
> virtual environments and just run in-line migrations from a version specific
> venv (this is what the OSA tooling does). Does one way work better than the
> other? Any thoughts on how this could be better? Would having N+2/3
> migrations addressable within the projects, even if they're not tested any
> longer, be helpful?
>
> It was our general thought that folks would be interested in having the
> ability to skip releases so we'd like to hear from the community to validate
> our thinking. Additionally, we'd like to get more minds together and see if
> folks are wanting to work on such an initiative, even if this turns into
> nothing more than a co-op/channel where we can "phone a friend". Would it be
> good to try and secure some PTG space to work on this? Should we try and
> create working group going?
>
> If you've made it this far, please forgive my stream of consciousness. I'm
> trying to ask a lot of questions and distill long form conversation(s) into
> as little text as possible all without writing a novel. With that said, I
> hope this finds you well, I look forward to hearing from (and working with)
> you soon.
>
> [0] https://etherpad.openstack.org/p/BOS-forum-skip-level-upgrading
> [1]
> https://github.com/openstack/openstack-ansible-ops/tree/master/leap-upgrades
>
>
> --
>
> Kevin Carter
> IRC: Cloudnull
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>