[openstack-dev] Upgrade Notes from Grenade
sean at dague.net
Fri Apr 5 13:43:34 UTC 2013
For people not familiar with it, Dean Troyer created this project at the
end of Folsom to try to do upgrade testing between releases. Basically
it does a devstack install at a previous release, injects some data,
shuts it down, then it pulls devstack from master, runs a set up upgrade
scripts, and brings up devstack at the new level. You can then run
exercises, tempest, whatever your fancy is.
Dean, Jim, and I have been working to get this in CI, updated for Folsom
-> Grizzly testing (we're still a few weeks away), but also to use it as
a tool for how small our upgrade process can be. In an ideal world it's
a "db-sync", and that's it. So here are the findings so far (note: we
might be able to minimize these even further, and I encourage your help
in doing so).
Nova's looking very good. Yesterday I was able to delete all the upgrade
code that touches the config files at all. You have to create 2
directories (cache_dir, and keys_dir), then off you go.
The Bad (or in reality, less Good):
Glance and Cinder are both close, but require a couple modifications of
their folsom environments to run.
Glance needs new paste and policy.json file (which I think it probably
fine), it also needs keystone_authtoken signing_dir set int both
configs, less fine, but minimal. And the cache_dir needs to be created.
In an ideal world the defaults would be fine, but they aren't
Cinder also needs paste files fixed, and it has an incompatible value
change in osapi_volume_extension that's needed (that's really the only
one I'd call it out on). It also needs to build the cache_dir.
Right now we are completely rerunning the configure steps for both Swift
and Keystone. If you don't they don't work. This is partially my
ignorance, I'm sure a smaller subset might work, but especially with
keystone that mixes user conf and paste stanzas in one file, the
differences between working Folsom and working Grizzly are large enough
that it's hard to figure out by trial and error what's really needed.
Quantum is an unknown. It wasn't in devstack in Folsom (it wasn't in CI
until g-3, and full Tempest still doesn't work on it) so it's untestable
Horizon looks good, we don't need an upgrade script for it. However the
existing devstack tests are just the sanity test that the login screen
shows up, hardly a comprehensive test. :)
The Call for Help:
Grenade is available at http://github.org/openstack-dev/grenade, and it
would be nice if people from the various core projects could help us
further reduce what's needed during upgrade. Keystone and Swift in
particular, but expert eyes on the others would be great as well. The
upgrade-* scripts are the things to check out.
I want to thank John Griffith for spending the time to help me really
reduce what was required for Cinder upgrade to that single variable.
It's super useful when such expert eyes get put on a problem like this.
There are some lessons to think about here (these are not comprehensive,
just some first thoughts). First off, we're getting a lot better at
rolling forward, the upgrade script size is quite small for a lot of
projects. Good on us.
There are still places we could, and should, do better. Like the fact
that at some level we treat the paste.ini files and policy.json as code
(i.e. doesn't matter if G requires incompatible lines from F), but then
we stick it in /etc/ and treat it as user configs. I think we need to
decide it's either code, and squirel it out of etc, or that it's config,
and handle deprecations well.
Keystone mixing paste and user config lines in one file by default is
currently madness, that really should change in Havana if at all possible.
As always, help is appreciated. The goal is that this will be in CI
before Havana-1 so that we'll be enforcing good upgrade hygiene early in
More information about the OpenStack-dev