Open Stack

Hello Stackers,

As I'm sure many of you know there was a talk about doing "skip-level"[0]
upgrades at the OpenStack Summit which quite a few folks were interested
in. Today many of the interested parties got together and talked about
doing more of this in a formalized capacity. Essentially we're looking for
cloud upgrades with the possibility of skipping releases, ideally enabling
an N+3 upgrade. In our opinion it would go a very long way to solving cloud
consumer and deployer problems it folks didn't have to deal with an upgrade
every six months. While we talked about various issues and some of the
current approaches being kicked around we wanted to field our general chat
to the rest of the community and request input from folks that may have
already fought such a beast. If you've taken on an adventure like this how
did you approach it? Did it work? Any known issues, gotchas, or things
folks should be generally aware of?


During our chat today we generally landed on an in-place upgrade with known
API service downtime and little (at least as little as possible) data plane
downtime. The process discussed was basically:
a1. Create utility "thing-a-me" (container, venv, etc) which contains the
required code to run a service through all of the required upgrades.
a2. Stop service(s).
a3. Run migration(s)/upgrade(s) for all releases using the utility
"thing-a-me".
a4. Repeat for all services.

b1. Once all required migrations are complete run a deployment using the
target release.
b2. Ensure all services are restarted.
b3. Ensure cloud is functional.
b4. profit!

Obviously, there's a lot of hand waving here but such a process is being
developed by the OpenStack-Ansible project[1]. Currently, the OSA tooling
will allow deployers to upgrade from Juno/Kilo to Newton using Ubuntu
14.04. While this has worked in the lab, it's early in development (YMMV).
Also, the tooling is not very general purpose or portable outside of OSA
but it could serve as a guide or just a general talking point. Are there
other tools out there that solve for the multi-release upgrade? Are there
any folks that might want to share their expertise? Maybe a process outline
that worked? Best practices? Do folks believe tools are the right way to
solve this or would comprehensive upgrade documentation be better for the
general community?

As most of the upgrade issues center around database migrations, we
discussed some of the potential pitfalls at length. One approach was to
roll-up all DB migrations into a single repository and run all upgrades for
a given project in one step. Another was to simply have mutliple python
virtual environments and just run in-line migrations from a version
specific venv (this is what the OSA tooling does). Does one way work better
than the other? Any thoughts on how this could be better? Would having
N+2/3 migrations addressable within the projects, even if they're not
tested any longer, be helpful?

It was our general thought that folks would be interested in having the
ability to skip releases so we'd like to hear from the community to
validate our thinking. Additionally, we'd like to get more minds together
and see if folks are wanting to work on such an initiative, even if this
turns into nothing more than a co-op/channel where we can "phone a friend".
Would it be good to try and secure some PTG space to work on this? Should
we try and create working group going?

If you've made it this far, please forgive my stream of consciousness. I'm
trying to ask a lot of questions and distill long form conversation(s) into
as little text as possible all without writing a novel. With that said, I
hope this finds you well, I look forward to hearing from (and working with)
you soon.

[0] https://etherpad.openstack.org/p/BOS-forum-skip-level-upgrading
[1] https://github.com/openstack/openstack-ansible-ops/tree/
master/leap-upgrades


--

Kevin Carter
IRC: Cloudnull
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20170525/931b25fd/attachment.html>