Open Stack

Adding [nova][neutron] since their gates will continue to be blocked 
until one of the following proposed fixes merges. They are linked inline.

On 8/11/20 19:57, melanie witt wrote:
> On 8/11/20 14:53, melanie witt wrote:
>> Howdy all,
>>
>> FYI the *-grenade-multinode gate jobs are currently failing with the 
>> following error in keystone:
>>
>>    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in 
>> position 3: invalid start byte
>>
>> This appears to be an issue with a new default data format in msgpack 
>> v1.0 [1] which was brought in by a recent bump of upper constraints [2].
>>
>> *-grenade-multinode jobs are affected because they test a rolling 
>> upgrade where the controller is upgraded to the N release version but 
>> one compute node is on the N-1 release version. It looks like cached 
>> keystone tokens being used by the N-1 node are erroring out during 
>> msgpack unpacking because they are in the old data format and msgpack 
>> v1.0 has a new default data format.
>>
>> I've opened a bug [3] about and I'm trying out the following keystone 
>> patch to fix it:
>>
>> https://review.opendev.org/745752
>>
>> Reviews appreciated.

I tested ^ with a DNM patch to nova and nova-grenade-multinode passes 
with it.

>> If this is not the best approach or if this affects other projects as 
>> well, alternatively we could revert the upper constraint bump to 
>> msgpack v1.0 while we figure out the best fix.
> 
> Here's a patch for reverting the upper constraint for msgpack in case 
> that approach is preferred:
> 
> https://review.opendev.org/745769

And this reqs pin ^ is also available if the reviewers find the keystone 
patch unsuitable.

>> [1] 
>> https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10 
>>
>> [2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373
>> [3] https://launchpad.net/bugs/1891244
>>
>