Open Stack

Oslo.cache version 2.1.0 is now blacklisted [1] to avoid similar issues for
now.

A new `memcache_pool` backend job have been introduced (not yet merged)
through a patch [2] to oslo.cache to help us to catch similar errors during
CI.

Now we have 2 ways to definitely fix the situation on oslo.cache:
- fix the broken code [3] and release a new patched version (2.1.1), still
WIP;
- revert the initial changes [4] and release a new version free from this
bug (2.2.0).

After some discussions with other oslo cores we want to give priority to
the fix [3] first.

Do not hesitate to correct me if something is wrong here.

Cheers,

PS: improvements and fix described in my previous email have been merged
together [3].

[1] https://review.opendev.org/#/c/711427/
[2] https://review.opendev.org/#/c/711422/
[3] https://review.opendev.org/#/c/711220/
[4] https://review.opendev.org/#/c/711439/


Le mer. 4 mars 2020 à 19:23, Herve Beraud <hberaud at redhat.com> a écrit :

> I proposed the following two patches to address the issue and improve this
> module beyond the current issue:
> - https://review.opendev.org/711220 (the fix)
> - https://review.opendev.org/711247 (the improvements)
>
> After these patches will be merged and the issue fixed we will blacklist
> the version 2.1.0 of oslo.cache and propose a new release with the previous
> fixes embedded.
>
> Do not hesitate to review them and leave comments.
>
> Thanks for your reading.
>
> Le mer. 4 mars 2020 à 14:16, Herve Beraud <hberaud at redhat.com> a écrit :
>
>> Fix proposed https://review.opendev.org/#/c/711220/
>>
>> Le mer. 4 mars 2020 à 13:42, Moises Guimaraes de Medeiros <
>> moguimar at redhat.com> a écrit :
>>
>>> `dead_timeout`++
>>>
>>> On Wed, Mar 4, 2020 at 1:36 PM Herve Beraud <hberaud at redhat.com> wrote:
>>>
>>>> `dead_timeout` [1] looks more appropriate in this case.
>>>>
>>>> [1]
>>>> https://github.com/pinterest/pymemcache/blob/master/pymemcache/client/hash.py#L58
>>>>
>>>> Le mer. 4 mars 2020 à 13:28, Herve Beraud <hberaud at redhat.com> a
>>>> écrit :
>>>>
>>>>> What do you think about adding a mapping between `retry_timeout` [1]
>>>>> and `dead_retry` [2]?
>>>>>
>>>>> [1]
>>>>> https://github.com/pinterest/pymemcache/blob/master/pymemcache/client/hash.py#L56
>>>>> [2]
>>>>> https://github.com/linsomniac/python-memcached/blob/bad41222379102e3f18f6f2f7be3ee608de6fbff/memcache.py#L183
>>>>>
>>>>> Le mer. 4 mars 2020 à 13:20, Herve Beraud <hberaud at redhat.com> a
>>>>> écrit :
>>>>>
>>>>>> I think our issue is due to the fact that python-memcached accept a
>>>>>> param named `dead_retry` [1] which is not defined in pymemcache.
>>>>>>
>>>>>> We just need to define it in our oslo.cache mapping. During testing
>>>>>> we faced the same kind of issue with connection timeout.
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/linsomniac/python-memcached/blob/bad41222379102e3f18f6f2f7be3ee608de6fbff/memcache.py#L183
>>>>>> [2]
>>>>>> https://github.com/openstack/oslo.cache/blob/8a8248d764bbb1db6c0089a58745803c03e38fdb/oslo_cache/_memcache_pool.py#L193,L201
>>>>>>
>>>>>> Le mer. 4 mars 2020 à 12:21, Radosław Piliszek <
>>>>>> radoslaw.piliszek at gmail.com> a écrit :
>>>>>>
>>>>>>> Please be informed that oslo.cache 2.1.0 breaks
>>>>>>> oslo_cache.memcache_pool
>>>>>>>
>>>>>>> Kolla-Ansible gate is already RED and a quick codesearch revealed
>>>>>>> other deployment methods might be in trouble soon as well.
>>>>>>>
>>>>>>> This does not affect devstack/tempest as they use
>>>>>>> dogpile.cache.memcached instead.
>>>>>>>
>>>>>>> The error is TypeError: __init__() got an unexpected keyword argument
>>>>>>> 'dead_retry'
>>>>>>>
>>>>>>> For details see: https://bugs.launchpad.net/oslo.cache/+bug/1866008
>>>>>>>
>>>>>>> -yoctozepto
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Hervé Beraud
>>>>>> Senior Software Engineer
>>>>>> Red Hat - Openstack Oslo
>>>>>> irc: hberaud
>>>>>> -----BEGIN PGP SIGNATURE-----
>>>>>>
>>>>>> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
>>>>>> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
>>>>>> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
>>>>>> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
>>>>>> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
>>>>>> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
>>>>>> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
>>>>>> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
>>>>>> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
>>>>>> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
>>>>>> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
>>>>>> v6rDpkeNksZ9fFSyoY2o
>>>>>> =ECSj
>>>>>> -----END PGP SIGNATURE-----
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Hervé Beraud
>>>>> Senior Software Engineer
>>>>> Red Hat - Openstack Oslo
>>>>> irc: hberaud
>>>>> -----BEGIN PGP SIGNATURE-----
>>>>>
>>>>> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
>>>>> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
>>>>> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
>>>>> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
>>>>> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
>>>>> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
>>>>> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
>>>>> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
>>>>> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
>>>>> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
>>>>> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
>>>>> v6rDpkeNksZ9fFSyoY2o
>>>>> =ECSj
>>>>> -----END PGP SIGNATURE-----
>>>>>
>>>>>
>>>>
>>>> --
>>>> Hervé Beraud
>>>> Senior Software Engineer
>>>> Red Hat - Openstack Oslo
>>>> irc: hberaud
>>>> -----BEGIN PGP SIGNATURE-----
>>>>
>>>> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
>>>> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
>>>> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
>>>> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
>>>> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
>>>> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
>>>> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
>>>> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
>>>> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
>>>> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
>>>> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
>>>> v6rDpkeNksZ9fFSyoY2o
>>>> =ECSj
>>>> -----END PGP SIGNATURE-----
>>>>
>>>>
>>>
>>> --
>>>
>>> Moisés Guimarães
>>>
>>> Software Engineer
>>>
>>> Red Hat <https://www.redhat.com>
>>>
>>> <https://red.ht/sig>
>>>
>>
>>
>> --
>> Hervé Beraud
>> Senior Software Engineer
>> Red Hat - Openstack Oslo
>> irc: hberaud
>> -----BEGIN PGP SIGNATURE-----
>>
>> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
>> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
>> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
>> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
>> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
>> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
>> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
>> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
>> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
>> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
>> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
>> v6rDpkeNksZ9fFSyoY2o
>> =ECSj
>> -----END PGP SIGNATURE-----
>>
>>
>
> --
> Hervé Beraud
> Senior Software Engineer
> Red Hat - Openstack Oslo
> irc: hberaud
> -----BEGIN PGP SIGNATURE-----
>
> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
> v6rDpkeNksZ9fFSyoY2o
> =ECSj
> -----END PGP SIGNATURE-----
>
>

-- 
Hervé Beraud
Senior Software Engineer
Red Hat - Openstack Oslo
irc: hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200306/78d82712/attachment-0001.html>