Open Stack

I am working on our Kilo->Liberty upgrade path at present, and when upgrading existing test Kilo controller nodes to Liberty, Keystone gets extremely weird. When evaluating token timestamps for revocation, it looks like sometimes the timestamps are processed as datetime.datetime objects and other times as strings. When Keystone gets a string, it explodes (lines shortened for readability):

[req-12f88024-cc71-429e-aea1-19dec45ac804 - - - - -] 'str' object has no attribute 'strftime'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 248, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 163, in inner
    return f(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 560, in revocation_list
    t['expires'] = utils.isotime(expires)
  File "/usr/lib/python2.7/dist-packages/keystone/common/utils.py", line 501, in isotime
    st = at.strftime(_ISO8601_TIME_FORMAT
AttributeError: 'str' object has no attribute 'strftime'

I have no idea what is causing it to sometimes be strings and sometimes not. When it does happen, anything trying to authenticate to Keystone gets back a 401, so it renders the entire cluster useless. Flushing memcached seemed to fix it but now does not.

Has anyone else seen anything like this? https://review.openstack.org/#/c/302904/ seems to be a related subject, but this is Neutron-specific.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160527/aa1c7d75/attachment.html>