Re: [dev][oslo] oslo.cache and dogpile 0.7.0+ cache errors

27 Feb 2019

      To close the loop on this, we just merged a unit test fix that unblocks 
oslo.cache ci. We'll continue to work on sorting out where these tests 
should live as a followup.
On 2/26/19 1:40 PM, Herve Beraud wrote:
...
Submit a patch to dogpile.cache to add some related tests cases:
https://github.com/sqlalchemy/dogpile.cache/pull/145/
Le mar. 26 févr. 2019 à 19:35, Herve Beraud <hberaud@redhat.com 
mailto:hberaud@redhat.com> a écrit :
FYI dogpile.cache issue was opened:
https://github.com/sqlalchemy/dogpile.cache/issues/144

Come with a possible oslo.cache solution that I've introduce there
=> https://review.openstack.org/#/c/638788/8

Le mar. 26 févr. 2019 à 16:49, Ben Nemec <openstack@nemebean.com
<mailto:openstack@nemebean.com>> a écrit :

    Copying Mike. More thoughts inline.

    On 2/26/19 9:24 AM, Herve Beraud wrote:
     > Hi,
     >
     > Just a heads up that the latest version of dogpile (0.7.0
    onwards)
     > have become incompatible with oslo.cache.  This is causing a few
     > issues for jobs.  It's a little complex due to functional
    code and many
     > decorated functions.
     >
     > The error you will see is:
     > /
     > /
     >
     >
    /oslo_cache.//tests.test_//cache.CacheRegi//onTest.//test_function_//key_generator_//with_kwargs
     >
    -------//-------//-------//-------//-------//-------//-------//-------//-------//-------//-------//------/
     >
     > //
     >
     > /Captured traceback:
     > ~~~~~~~~~~~~~~~~~~~
     >      b'Traceback (most recent call last):'
     >      b' File
    "/tmp/oslo.//cache/oslo_//cache/tests///test_cache.//py",
     > line 324, in test_function_//key_generator_//with_kwargs'
     >      b' value=self.//test_value)//'
     >      b' File
     >
    "/tmp/oslo.//cache/.//tox/py37///lib/python3.//7/site-//packages///testtools///testcase.//py",

     > line 485, in assertRaises'
     >      b' self.assertThat//(our_callable, matcher)'
     >      b' File
     >
    "/tmp/oslo.//cache/.//tox/py37///lib/python3.//7/site-//packages///testtools///testcase.//py",

     > line 498, in assertThat'
     >      b' raise mismatch_error'
     >      b'testtools//.matchers.//_impl.MismatchE//rror: <function
     >
    CacheRegionTest//._get_cacheable//_function.//<locals>//.cacheable_//function

     > at 0x7fec3f795400> returned
     > <oslo_cache.//tests.test_//cache.TestProxy//Value object at
     > 0x7fec3f792550>'/
     >
     >
     > The problem appear since we uncap dogpile.cache on oslo.cache:
     >
    https://github.com/openstack/oslo.cache/commit/62b53099861134859482656dc92db81243b88bd9
     >
     > The following unit test fail since we uncap dogpile =>
     >
    https://github.com/openstack/oslo.cache/blob/master/oslo_cache/tests/test_cache.py#L318
     >
     > The problem was introduced by:
     > https://gerrit.sqlalchemy.org/#/c/sqlalchemy/dogpile.cache/+/996/
     >
     > Your main issue on oslo.cache side is that keyword arguments are
     > tranformed in positionnal arguments when we use
     > dogpile.cache.region.cache_on_arguments.
     >
     > I've try to revert the changes introduced by the previous
    dogpile.cache
     > change and everything works fine on the oslo.cache side when
    changes was
     > reverted (reverted to revision
     >
    https://github.com/sqlalchemy/dogpile.cache/blob/2762ada1f5e43075494d91c512f7c1ec68907258/dogpile/cache/region.py).
     >
     > The expected behavior is that
    dogpile.cache.util.function_key_generator
     > raise a ValueError if **kwargs founds, but our kwargs is
    empty and our
     > `value=self.test_value was` is recognized as a positionnal
    argument.
     > Our unit test looking for an assertRaise(ValueError) on cachable
     > decorated function when we pass kwargs but it doesn't happen
    due to
     > empty kwargs.
     >
     > For these reasons we guess that is an dogpile.cache issue and
    not an
     > oslo.cache issue due to the changes introduced by `decorator`
    module.
     >
     > The following are related:
     >
     > -
     >
    https://github.com/openstack/oslo.cache/blob/master/oslo_cache/tests/test_cache.py#L318

     > : unit test where the problem occure
     > - https://review.openstack.org/#/c/638788/ : possible fix but
    we don't
     > think that is the right way
     > - https://review.openstack.org/#/c/638732/ : possible remove
    of the unit
     > test who fail

    As I noted in the reviews, I don't think this is something we
    should
    have been testing in oslo.cache in the first place. The failing
    test is
    testing the dogpile interface, not the oslo.cache one. I've seen no
    evidence that oslo.cache is doing anything wrong here, so our
    unit tests
    are clearly testing something that should be out of scope.

    And to be clear, I'm not even sure this is a bug in dogpile. It
    may be a
    happy side-effect of the decorator change that the regular
    decorator now
    works for kwargs too. I don't know dogpile well enough to make a
    definitive statement on that though. Hence cc'ing Mike. :-)

     >
     > The issue is being tracked in:
     >
     > https://bugs.launchpad.net/oslo.cache/+bug/1817032
     >
     > If some dogpile expert can take a look and send feedback on
    this thread
     > you are welcome.
     >
     > Thanks,
     >
     > --
     > Hervé Beraud
     > Senior Software Engineer
     > Red Hat - Openstack Oslo
     > irc: hberaud
     > -----BEGIN PGP SIGNATURE-----
     >
     > wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
     > Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
     > RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
     > F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
     > 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
     > glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
     > m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
     > hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
     > qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
     > F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
     > B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
     > v6rDpkeNksZ9fFSyoY2o
     > =ECSj
     > -----END PGP SIGNATURE-----
     >

-- 
Hervé Beraud
Senior Software Engineer
Red Hat - Openstack Oslo
irc: hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----

-- 
Hervé Beraud
Senior Software Engineer
Red Hat - Openstack Oslo
irc: hberaud
-----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----