[gate][keystone] *-grenade-multinode jobs failing with UnicodeDecodeError in keystone
Howdy all, FYI the *-grenade-multinode gate jobs are currently failing with the following error in keystone: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte This appears to be an issue with a new default data format in msgpack v1.0 [1] which was brought in by a recent bump of upper constraints [2]. *-grenade-multinode jobs are affected because they test a rolling upgrade where the controller is upgraded to the N release version but one compute node is on the N-1 release version. It looks like cached keystone tokens being used by the N-1 node are erroring out during msgpack unpacking because they are in the old data format and msgpack v1.0 has a new default data format. I've opened a bug [3] about and I'm trying out the following keystone patch to fix it: https://review.opendev.org/745752 Reviews appreciated. If this is not the best approach or if this affects other projects as well, alternatively we could revert the upper constraint bump to msgpack v1.0 while we figure out the best fix. Cheers, -melanie [1] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaki... [2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 [3] https://launchpad.net/bugs/1891244
On 8/11/20 14:53, melanie witt wrote:
Howdy all,
FYI the *-grenade-multinode gate jobs are currently failing with the following error in keystone:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte
This appears to be an issue with a new default data format in msgpack v1.0 [1] which was brought in by a recent bump of upper constraints [2].
*-grenade-multinode jobs are affected because they test a rolling upgrade where the controller is upgraded to the N release version but one compute node is on the N-1 release version. It looks like cached keystone tokens being used by the N-1 node are erroring out during msgpack unpacking because they are in the old data format and msgpack v1.0 has a new default data format.
I've opened a bug [3] about and I'm trying out the following keystone patch to fix it:
https://review.opendev.org/745752
Reviews appreciated.
If this is not the best approach or if this affects other projects as well, alternatively we could revert the upper constraint bump to msgpack v1.0 while we figure out the best fix.
Here's a patch for reverting the upper constraint for msgpack in case that approach is preferred: https://review.opendev.org/745769
[1] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaki... [2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 [3] https://launchpad.net/bugs/1891244
Adding [nova][neutron] since their gates will continue to be blocked until one of the following proposed fixes merges. They are linked inline. On 8/11/20 19:57, melanie witt wrote:
On 8/11/20 14:53, melanie witt wrote:
Howdy all,
FYI the *-grenade-multinode gate jobs are currently failing with the following error in keystone:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte
This appears to be an issue with a new default data format in msgpack v1.0 [1] which was brought in by a recent bump of upper constraints [2].
*-grenade-multinode jobs are affected because they test a rolling upgrade where the controller is upgraded to the N release version but one compute node is on the N-1 release version. It looks like cached keystone tokens being used by the N-1 node are erroring out during msgpack unpacking because they are in the old data format and msgpack v1.0 has a new default data format.
I've opened a bug [3] about and I'm trying out the following keystone patch to fix it:
https://review.opendev.org/745752
Reviews appreciated.
I tested ^ with a DNM patch to nova and nova-grenade-multinode passes with it.
If this is not the best approach or if this affects other projects as well, alternatively we could revert the upper constraint bump to msgpack v1.0 while we figure out the best fix.
Here's a patch for reverting the upper constraint for msgpack in case that approach is preferred:
And this reqs pin ^ is also available if the reviewers find the keystone patch unsuitable.
[1] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaki...
[2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 [3] https://launchpad.net/bugs/1891244
Hi, Thx Melanie for the proposed fix for this issue. On Wed, Aug 12, 2020 at 12:49:10AM -0700, melanie witt wrote:
Adding [nova][neutron] since their gates will continue to be blocked until one of the following proposed fixes merges. They are linked inline.
On 8/11/20 19:57, melanie witt wrote:
On 8/11/20 14:53, melanie witt wrote:
Howdy all,
FYI the *-grenade-multinode gate jobs are currently failing with the following error in keystone:
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte
This appears to be an issue with a new default data format in msgpack v1.0 [1] which was brought in by a recent bump of upper constraints [2].
*-grenade-multinode jobs are affected because they test a rolling upgrade where the controller is upgraded to the N release version but one compute node is on the N-1 release version. It looks like cached keystone tokens being used by the N-1 node are erroring out during msgpack unpacking because they are in the old data format and msgpack v1.0 has a new default data format.
I've opened a bug [3] about and I'm trying out the following keystone patch to fix it:
https://review.opendev.org/745752
Reviews appreciated.
I tested ^ with a DNM patch to nova and nova-grenade-multinode passes with it.
If this is not the best approach or if this affects other projects as well, alternatively we could revert the upper constraint bump to msgpack v1.0 while we figure out the best fix.
Here's a patch for reverting the upper constraint for msgpack in case that approach is preferred:
And this reqs pin ^ is also available if the reviewers find the keystone patch unsuitable.
[1] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaki...
[2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 [3] https://launchpad.net/bugs/1891244
-- Slawek Kaplonski Senior software engineer Red Hat
On 8/12/20 00:49, melanie witt wrote:
I've opened a bug [3] about and I'm trying out the following keystone patch to fix it:
https://review.opendev.org/745752
Reviews appreciated.
I tested ^ with a DNM patch to nova and nova-grenade-multinode passes with it.
The fix has merged and it is now safe to recheck your patches. Thank you all for the code reviews. Cheers, -melanie
[1] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaki...
[2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 [3] https://launchpad.net/bugs/1891244
participants (2)
-
melanie witt
-
Slawek Kaplonski