On 8/11/20 14:53, melanie witt wrote: > Howdy all, > > FYI the *-grenade-multinode gate jobs are currently failing with the following error in keystone: > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte > > This appears to be an issue with a new default data format in msgpack v1.0 [1] which was brought in by a recent bump of upper constraints [2]. > > *-grenade-multinode jobs are affected because they test a rolling upgrade where the controller is upgraded to the N release version but one compute node is on the N-1 release version. It looks like cached keystone tokens being used by the N-1 node are erroring out during msgpack unpacking because they are in the old data format and msgpack v1.0 has a new default data format. > > I've opened a bug [3] about and I'm trying out the following keystone patch to fix it: > > https://review.opendev.org/745752 > > Reviews appreciated. > > If this is not the best approach or if this affects other projects as well, alternatively we could revert the upper constraint bump to msgpack v1.0 while we figure out the best fix. Here's a patch for reverting the upper constraint for msgpack in case that approach is preferred: https://review.opendev.org/745769 > [1] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10 > [2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 > [3] https://launchpad.net/bugs/1891244 >