Open Stack

Final posting on the upgrade at CERN and summary at http://openstack-in-production.blogspot.ch/2014/02/our-cloud-in-havana.html with a few hints and some steps to run for a cells environment. A couple of problems seen


-        euca2ools did not work on our standard SL 6 configuration (version 2.1.4). An error instance-type should be of type string was raised on VM creation. The root cause was a downlevel boto version.


-        When creating multiple machines using --num-instances on the nova command line with the cells configuration, the unique name created was invalid.

Tim

From: Tim Bell
Sent: 23 January 2014 22:10
To: openstack-operators at lists.openstack.org
Subject: RE: Keystone Grizzly to Havana upgrade


Following on the sequence of quick notes for those doing Grizzly to Havana upgrades.

At CERN, we've just migrated Keystone from Grizzly to Havana today. It actually happened much faster than we had originally intended!

The overall approach taken was to try an online upgrade. We have a keystone in each of our cells (so we can talk to each of them independently of the top level cell API nodes) so these were good candidates to upgrade first. We use Active Directory for the user credentials so the Keystone service itself has a limited amount of state only (related to EC2 credentials and token management).

For the main keystone instance, we were aiming to benefit from the load balancing layer and that there were minimal database changes in Keystone. The largest problem foreseen was the EC2 credentials since these, in Grizzly, are stored in a different column to those in Havana (just Credentials). These columns had to be kept in sync with a script during the phase where both versions were running.

In practice, we performed the upgrade more rapidly than originally planned as some of the clients gave errors when they had received a token from a Grizzly keystone and were authenticating against a Havana one. Thus, we upgraded all the keystone servers as soon as the final functional tests in the production environment were completed on the first keystone.

The detailed error message of the error on a Grizzly client was

2014-01-23 09:10:02    ERROR [root] 'token'
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 265, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 541, in validate_token
    self._assert_default_domain(context, token_ref)
  File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 482, in _assert_default_domain
    if (token_ref['token_data']['token']['user']['domain']['id'] !=
KeyError: 'token'

The solution was to complete the upgrade to Keystone on all the servers.

Other than that, things are running fine and now we can start the work to take advantage of the new features.

Tim

From: Tim Bell
Sent: 15 January 2014 20:23
To: openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>
Subject: Glance Grizzly to Havana upgrade


Quick warning to those doing the Grizzly to Havana migration,

Our upgrade went smoothly (around 45 minutes to complete).

Following the migration, we found that nova image-list and glance image-list were showing different results.

The root cause was related to the bug https://bugs.launchpad.net/glance/+bug/1152716. The fix for the problem was to add the policy statement for the parameter context_is_admin which is needed to limit access to private images for projects.

Tim

--
[cid:image001.png at 01CF3221.77735370]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140225/ac1e7b9b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12650 bytes
Desc: image001.png
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140225/ac1e7b9b/attachment.png>