Open Stack

For people not familiar with it, Dean Troyer created this project at the 
end of Folsom to try to do upgrade testing between releases. Basically 
it does a devstack install at a previous release, injects some data, 
shuts it down, then it pulls devstack from master, runs a set up upgrade 
scripts, and brings up devstack at the new level. You can then run 
exercises, tempest, whatever your fancy is.

Dean, Jim, and I have been working to get this in CI, updated for Folsom 
-> Grizzly testing (we're still a few weeks away), but also to use it as 
a tool for how small our upgrade process can be. In an ideal world it's 
a "db-sync", and that's it. So here are the findings so far (note: we 
might be able to minimize these even further, and I encourage your help 
in doing so).

The Good:

Nova's looking very good. Yesterday I was able to delete all the upgrade 
code that touches the config files at all. You have to create 2 
directories (cache_dir, and keys_dir), then off you go.

The Bad (or in reality, less Good):

Glance and Cinder are both close, but require a couple modifications of 
their folsom environments to run.

Glance needs new paste and policy.json file (which I think it probably 
fine), it also needs keystone_authtoken signing_dir set int both 
configs, less fine, but minimal. And the cache_dir needs to be created. 
In an ideal world the defaults would be fine, but they aren't

Cinder also needs paste files fixed, and it has an incompatible value 
change in osapi_volume_extension that's needed (that's really the only 
one I'd call it out on). It also needs to build the cache_dir.

The Ugly:

Right now we are completely rerunning the configure steps for both Swift 
and Keystone. If you don't they don't work. This is partially my 
ignorance, I'm sure a smaller subset might work, but especially with 
keystone that mixes user conf and paste stanzas in one file, the 
differences between working Folsom and working Grizzly are large enough 
that it's hard to figure out by trial and error what's really needed.

The Unknown:

Quantum is an unknown. It wasn't in devstack in Folsom (it wasn't in CI 
until g-3, and full Tempest still doesn't work on it) so it's untestable 
right now.

Horizon looks good, we don't need an upgrade script for it. However the 
existing devstack tests are just the sanity test that the login screen 
shows up, hardly a comprehensive test. :)

The Call for Help:

Grenade is available at http://github.org/openstack-dev/grenade, and it 
would be nice if people from the various core projects could help us 
further reduce what's needed during upgrade. Keystone and Swift in 
particular, but expert eyes on the others would be great as well. The 
upgrade-* scripts are the things to check out.

I want to thank John Griffith for spending the time to help me really 
reduce what was required for Cinder upgrade to that single variable. 
It's super useful when such expert eyes get put on a problem like this.

Lessons:

There are some lessons to think about here (these are not comprehensive, 
just some first thoughts). First off, we're getting a lot better at 
rolling forward, the upgrade script size is quite small for a lot of 
projects. Good on us.

There are still places we could, and should, do better. Like the fact 
that at some level we treat the paste.ini files and policy.json as code 
(i.e. doesn't matter if G requires incompatible lines from F), but then 
we stick it in /etc/ and treat it as user configs. I think we need to 
decide it's either code, and squirel it out of etc, or that it's config, 
and handle deprecations well.

Keystone mixing paste and user config lines in one file by default is 
currently madness, that really should change in Havana if at all possible.


As always, help is appreciated. The goal is that this will be in CI 
before Havana-1 so that we'll be enforcing good upgrade hygiene early in 
the release.

	-Sean

-- 
Sean Dague
http://dague.net