Open Stack

Hi All,

I recently did a Folsom to Grizzly upgrade on a production OpenStack
environment and wanted to post the process and some notes. I understand
that Havana is coming out in a few weeks, so this might be old news... I
hope to do Grizzly to Havana within 3 months of Havana's release.

tl;dr: Short of a few minor manual changes, doing a straight "apt-get
install" provided a clean and safe upgrade. The instances themselves saw no
downtime.

I definitely didn't see the issues that Jon ran into (whose posts I had
starred in my inbox as warnings :)

Environment:

Ubuntu 12.04 and OpenStack Folsom with two distinct regions and a cloud
controller in each region. The controller runs all services but
nova-compute. There's a NetApp providing NFS for shared instance storage as
well as iSCSI for block storage.

nova-network is being used (and is still being used).

Swift is used, but it was upgraded to Grizzly months ago to utilize the new
quota feature.

Everything is strictly controlled by Puppet.

Prep:

I ran through this process several times in a sandbox environment. I highly
recommend doing this. Even if you don't have the budget for several
servers, virtualize your cloud controller and a few compute nodes with the
same configuration.

With Puppet, I use hiera to split my environment data into "sandbox" and
"production" data sets. This way, the same Puppet config is used in both
environments, but the proper environment data (subnets, passwords etc) is
swapped out.

Cloud Controller Upgrade:

1. Turn off Puppet

2. Backup the databases

3. Remove the Folsom cloud repo, install the Grizzly repo, apt-get update

4. apt-get dist-upgrade and say No to apt wanting to install new config
files. The new files will be saved with a .dpkg-dist extension.

5. cp /etc/keystone/keystone.conf{,.orig}
6. cp /etc/keystone/keystone.conf.dpkg-dist /etc/keystone/keystone.conf
7. Manually edit /etc/keystone/keystone.conf with the relevant options.
There's a significant different between the Folsom and Grizzly conf files
that will prevent Keystone from running correctly before Puppet is turned
back on.

8. Repeat for /etc/cinder/api-paste.ini.

All other Folsom-based configs were able to be used to get the services
running with Grizzly before Puppet jumped back in to make any
Grizzly-specific changes.

9. Run db migrations on all services:

keystone-manage db_sync
glance-manage db_sync
nova-manage db sync
cinder-manage db sync

10. Install nova-conductor manually

11. Manually patch keystone. See this thread for details:

http://www.gossamer-threads.com/lists/openstack/dev/30683

12. Verify all services are back up. nova-network needed to be restarted
once all other nova services were running... happened in both regions. No
big deal.

13. Install the latest Puppet Keystone module because older versions are
not compatible with Grizzly and newer versions are not compatible with
Folsom.

14. Test Puppet: puppet agent --verbose --onetime --no-daemonize --noop

15. Turn on Puppet

Compute Node Upgrade:

1. Turn off Puppet

2. Remove Folsom repo, add Grizzly repo, apt-get update

3. apt-get install <long list of packages>

apt-get dist-upgrade was not possible because there was an updated version
of open-iscsi available. Upgrading the package would cause all current
iSCSI connections to disconnect effectively disconnecting all volumes. Not
cool. I think the only way around this would be to evacuate compute nodes
and install the update.

4. Test and turn on Puppet

Final Notes:

Why not have Puppet running during the upgrade or let Puppet do the
upgrade? I probably could have done this but I preferred to "personally"
oversee the upgrade myself.

I saw a recent thread on the -dev list about providing reverse database
migrations for the possibility of having to roll back an upgrade. A comment
was made (I forget by whom) asking if this was really needed and if an
admin would simply perform a database restore instead. I fully agree with
that statement. If I had to roll back, I would uninstall the Grizzly
packages, install the Folsom packages, and then import my latest database
backup.

I hope someone finds this useful. Please let me know if you'd like further
details.

Thanks,
Joe



-- 
Joe Topjian
Systems Architect
Cybera Inc.

www.cybera.ca

Cybera is a not-for-profit organization that works to spur and support
innovation, for the economic benefit of Alberta, through the use
of cyberinfrastructure.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130919/b6282a8b/attachment.html>