Open Stack

Hi,

Thx Melanie for the proposed fix for this issue.

On Wed, Aug 12, 2020 at 12:49:10AM -0700, melanie witt wrote:
> Adding [nova][neutron] since their gates will continue to be blocked until
> one of the following proposed fixes merges. They are linked inline.
> 
> On 8/11/20 19:57, melanie witt wrote:
> > On 8/11/20 14:53, melanie witt wrote:
> > > Howdy all,
> > > 
> > > FYI the *-grenade-multinode gate jobs are currently failing with the
> > > following error in keystone:
> > > 
> > > Â Â  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in
> > > position 3: invalid start byte
> > > 
> > > This appears to be an issue with a new default data format in
> > > msgpack v1.0 [1] which was brought in by a recent bump of upper
> > > constraints [2].
> > > 
> > > *-grenade-multinode jobs are affected because they test a rolling
> > > upgrade where the controller is upgraded to the N release version
> > > but one compute node is on the N-1 release version. It looks like
> > > cached keystone tokens being used by the N-1 node are erroring out
> > > during msgpack unpacking because they are in the old data format and
> > > msgpack v1.0 has a new default data format.
> > > 
> > > I've opened a bug [3] about and I'm trying out the following
> > > keystone patch to fix it:
> > > 
> > > https://review.opendev.org/745752
> > > 
> > > Reviews appreciated.
> 
> I tested ^ with a DNM patch to nova and nova-grenade-multinode passes with
> it.
> 
> > > If this is not the best approach or if this affects other projects
> > > as well, alternatively we could revert the upper constraint bump to
> > > msgpack v1.0 while we figure out the best fix.
> > 
> > Here's a patch for reverting the upper constraint for msgpack in case
> > that approach is preferred:
> > 
> > https://review.opendev.org/745769
> 
> And this reqs pin ^ is also available if the reviewers find the keystone
> patch unsuitable.
> 
> > > [1] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10
> > > 
> > > [2] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373
> > > [3] https://launchpad.net/bugs/1891244
> > > 
> > 
> 
> 

-- 
Slawek Kaplonski
Senior software engineer
Red Hat