[dev][openstacksdk] openstacksdk and dogpile 0.7.0+ cache errors
Hi, Just a heads up that the latest version of dogpile (0.7.0 onwards) have become incompatible with openstacksdk. This is causing a few issues for jobs. As usual, it's a little complex :) The error you will see is something like user_func.set = set_ AttributeError: 'method' object has no attribute 'set' The following are related: https://review.openstack.org/624697 : requirements change to pin https://review.openstack.org/624485 : openstacksdk pin https://review.openstack.org/624855 : fixes a nodepool job that openstacksdk depends on (note 624485 turns the job non-voting to break the circular dependency). The issue is being tracked in: https://storyboard.openstack.org/#!/story/2004605 : related story https://github.com/sqlalchemy/dogpile.cache/issues/140 : upstream github issue Thanks, -i
On Thu, Dec 13, 2018 at 10:46 PM Ian Wienand <iwienand@redhat.com> wrote:
Hi,
Just a heads up that the latest version of dogpile (0.7.0 onwards) have become incompatible with openstacksdk. This is causing a few issues for jobs. As usual, it's a little complex :)
Can you let me know which openstack projects had a failure? For this specific change, I enabled my own openstack CI for dogpile.cache, which runs a selected set of openstack tests for the Neutron, Nova, Keystone and oslo.db projects, and I didn't observe any failures: https://gerrit.sqlalchemy.org/#/x/verify-status/jobs/996/4
The error you will see is something like
user_func.set = set_ AttributeError: 'method' object has no attribute 'set'
The following are related:
https://review.openstack.org/624697 : requirements change to pin https://review.openstack.org/624485 : openstacksdk pin https://review.openstack.org/624855 : fixes a nodepool job that openstacksdk depends on (note 624485 turns the job non-voting to break the circular dependency).
The issue is being tracked in:
https://storyboard.openstack.org/#!/story/2004605 : related story https://github.com/sqlalchemy/dogpile.cache/issues/140 : upstream github issue
I've commented on what is likely a simple fix, but it depends on what the Python decorator module, which here is our new third party dependency, is capable of doing. This review sat in the queue for a few weeks as I vaguely worried what problems this dependency might cause, so I guess I failed to get enough people to review this change. Are there folks here who would like to watch the dogpile.cache github project (now at https://github.com/sqlalchemy/dogpile.cache) as well as our gerrit so that future changes can get some more review?
Thanks,
-i
On 2018-12-14 09:11:39 -0500 (-0500), Mike Bayer wrote: [...]
Are there folks here who would like to watch the dogpile.cache github project (now at https://github.com/sqlalchemy/dogpile.cache) as well as our gerrit so that future changes can get some more review?
Alternatively, zuul.openstack.org has the ability to test pull requests for GitHub repositories automatically and report back configured job results (basically acting as a third-party CI system). We already do this for Ansible, for example. If we can identify a good set of candidate jobs to run, is this something you'd be interested in? -- Jeremy Stanley
The issue was with OpenStack SDK. SDK is doing a strange thing and wraps with dogpile the already bound-methods (from instantiated objects). It turns out this isn't really needed and the solution will be to not add the extra layer of wrapping into the SDK bits. I expect I can roll a patch for that later today. The extra layer of wrapping was to bubble up an .invalidate method, which strictly should have already existed when using dogpile. I'll make sure to unroll all the potential issues and get SDK into a sane place. In my opinion dogpile.cache is doing the right thing and assuming it is in charge of wrapping the methods. Wrapping (with a decorator) bound methods is a weird bit of meta-programming I don't see as necessary to support. With that said, this was a change in behavior. I've been unable (need to find my hardware token) to login to github to comment on the issue (will do this today) but I am going to comment much the same as I have here on the mailing list. Finally, I do think we should add the feedback loop to dogpile from openstack as we lean on it pretty heavily. I believe the big projects to catch are SDK/shade, Keystone, Keystone Middleware, and Oslo.Cache (oslo.cache should cover everything but openstackSDK/shade). I might be missing bits of infra as well in this list. On Fri, Dec 14, 2018 at 6:55 AM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2018-12-14 09:11:39 -0500 (-0500), Mike Bayer wrote: [...]
Are there folks here who would like to watch the dogpile.cache github project (now at https://github.com/sqlalchemy/dogpile.cache) as well as our gerrit so that future changes can get some more review?
Alternatively, zuul.openstack.org has the ability to test pull requests for GitHub repositories automatically and report back configured job results (basically acting as a third-party CI system). We already do this for Ansible, for example. If we can identify a good set of candidate jobs to run, is this something you'd be interested in? -- Jeremy Stanley
On Fri, Dec 14, 2018 at 10:48 AM Morgan Fainberg <morgan.fainberg@gmail.com> wrote:
The issue was with OpenStack SDK. SDK is doing a strange thing and wraps with dogpile the already bound-methods (from instantiated objects). It turns out this isn't really needed and the solution will be to not add the extra layer of wrapping into the SDK bits. I expect I can roll a patch for that later today. The extra layer of wrapping was to bubble up an .invalidate method, which strictly should have already existed when using dogpile. I'll make sure to unroll all the potential issues and get SDK into a sane place.
In my opinion dogpile.cache is doing the right thing and assuming it is in charge of wrapping the methods. Wrapping (with a decorator) bound methods is a weird bit of meta-programming I don't see as necessary to support. With that said, this was a change in behavior.
I wasn't sure that the use of decorator instead of a homegrown was really going to be noticeable but I guess it is. Nevertheless I did move from 0.6.x to 0.7.x which in my own projects means behaviorally-modifying change (my versioning scheme is <the laws of physics have changed>.<major features with potential behavioral changes>.<non breaking bug fixes and small features>).
I've been unable (need to find my hardware token) to login to github to comment on the issue (will do this today) but I am going to comment much the same as I have here on the mailing list.
Finally, I do think we should add the feedback loop to dogpile from openstack as we lean on it pretty heavily. I believe the big projects to catch are SDK/shade, Keystone, Keystone Middleware, and Oslo.Cache (oslo.cache should cover everything but openstackSDK/shade). I might be missing bits of infra as well in this list.
On Fri, Dec 14, 2018 at 6:55 AM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2018-12-14 09:11:39 -0500 (-0500), Mike Bayer wrote: [...]
Are there folks here who would like to watch the dogpile.cache github project (now at https://github.com/sqlalchemy/dogpile.cache) as well as our gerrit so that future changes can get some more review?
Alternatively, zuul.openstack.org has the ability to test pull requests for GitHub repositories automatically and report back configured job results (basically acting as a third-party CI system). We already do this for Ansible, for example. If we can identify a good set of candidate jobs to run, is this something you'd be interested in? -- Jeremy Stanley
Yep. I didn't mean to imply the change in behavior was wrong on a pre-1.0 in this case. We will fix SDK to do a more correct thing. On Fri, Dec 14, 2018, 10:06 Mike Bayer <mike_mp@zzzcomputing.com wrote:
On Fri, Dec 14, 2018 at 10:48 AM Morgan Fainberg <morgan.fainberg@gmail.com> wrote:
The issue was with OpenStack SDK. SDK is doing a strange thing and wraps
with dogpile the already bound-methods (from instantiated objects). It turns out this isn't really needed and the solution will be to not add the extra layer of wrapping into the SDK bits. I expect I can roll a patch for that later today. The extra layer of wrapping was to bubble up an .invalidate method, which strictly should have already existed when using dogpile. I'll make sure to unroll all the potential issues and get SDK into a sane place.
In my opinion dogpile.cache is doing the right thing and assuming it is
in charge of wrapping the methods. Wrapping (with a decorator) bound methods is a weird bit of meta-programming I don't see as necessary to support. With that said, this was a change in behavior.
I wasn't sure that the use of decorator instead of a homegrown was really going to be noticeable but I guess it is. Nevertheless I did move from 0.6.x to 0.7.x which in my own projects means behaviorally-modifying change (my versioning scheme is <the laws of physics have changed>.<major features with potential behavioral changes>.<non breaking bug fixes and small features>).
I've been unable (need to find my hardware token) to login to github to
comment on the issue (will do this today) but I am going to comment much the same as I have here on the mailing list.
Finally, I do think we should add the feedback loop to dogpile from
openstack as we lean on it pretty heavily. I believe the big projects to catch are SDK/shade, Keystone, Keystone Middleware, and Oslo.Cache (oslo.cache should cover everything but openstackSDK/shade). I might be missing bits of infra as well in this list.
On Fri, Dec 14, 2018 at 6:55 AM Jeremy Stanley <fungi@yuggoth.org>
wrote:
On 2018-12-14 09:11:39 -0500 (-0500), Mike Bayer wrote: [...]
Are there folks here who would like to watch the dogpile.cache github project (now at https://github.com/sqlalchemy/dogpile.cache) as well as our gerrit so that future changes can get some more review?
Alternatively, zuul.openstack.org has the ability to test pull requests for GitHub repositories automatically and report back configured job results (basically acting as a third-party CI system). We already do this for Ansible, for example. If we can identify a good set of candidate jobs to run, is this something you'd be interested in? -- Jeremy Stanley
This is fixed with a quick added function wrapper before passing to dogpile's decorator/wrapper method(s) [0]. I have confirmed it has remedied the issue directly and locally, but have been unable to produce a full unit test yet. [0] https://review.openstack.org/#/c/625370/ --Morgan On Fri, Dec 14, 2018 at 10:06 AM Mike Bayer <mike_mp@zzzcomputing.com> wrote:
On Fri, Dec 14, 2018 at 10:48 AM Morgan Fainberg <morgan.fainberg@gmail.com> wrote:
The issue was with OpenStack SDK. SDK is doing a strange thing and wraps
with dogpile the already bound-methods (from instantiated objects). It turns out this isn't really needed and the solution will be to not add the extra layer of wrapping into the SDK bits. I expect I can roll a patch for that later today. The extra layer of wrapping was to bubble up an .invalidate method, which strictly should have already existed when using dogpile. I'll make sure to unroll all the potential issues and get SDK into a sane place.
In my opinion dogpile.cache is doing the right thing and assuming it is
in charge of wrapping the methods. Wrapping (with a decorator) bound methods is a weird bit of meta-programming I don't see as necessary to support. With that said, this was a change in behavior.
I wasn't sure that the use of decorator instead of a homegrown was really going to be noticeable but I guess it is. Nevertheless I did move from 0.6.x to 0.7.x which in my own projects means behaviorally-modifying change (my versioning scheme is <the laws of physics have changed>.<major features with potential behavioral changes>.<non breaking bug fixes and small features>).
I've been unable (need to find my hardware token) to login to github to
comment on the issue (will do this today) but I am going to comment much the same as I have here on the mailing list.
Finally, I do think we should add the feedback loop to dogpile from
openstack as we lean on it pretty heavily. I believe the big projects to catch are SDK/shade, Keystone, Keystone Middleware, and Oslo.Cache (oslo.cache should cover everything but openstackSDK/shade). I might be missing bits of infra as well in this list.
On Fri, Dec 14, 2018 at 6:55 AM Jeremy Stanley <fungi@yuggoth.org>
wrote:
On 2018-12-14 09:11:39 -0500 (-0500), Mike Bayer wrote: [...]
Are there folks here who would like to watch the dogpile.cache github project (now at https://github.com/sqlalchemy/dogpile.cache) as well as our gerrit so that future changes can get some more review?
Alternatively, zuul.openstack.org has the ability to test pull requests for GitHub repositories automatically and report back configured job results (basically acting as a third-party CI system). We already do this for Ansible, for example. If we can identify a good set of candidate jobs to run, is this something you'd be interested in? -- Jeremy Stanley
On Fri, Dec 14, 2018 at 9:53 AM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2018-12-14 09:11:39 -0500 (-0500), Mike Bayer wrote: [...]
Are there folks here who would like to watch the dogpile.cache github project (now at https://github.com/sqlalchemy/dogpile.cache) as well as our gerrit so that future changes can get some more review?
Alternatively, zuul.openstack.org has the ability to test pull requests for GitHub repositories automatically and report back configured job results (basically acting as a third-party CI system). We already do this for Ansible, for example. If we can identify a good set of candidate jobs to run, is this something you'd be interested in?
Hi Jeremy - not sure if this is an option but I'd be more interested if this would be possible from my Gerrit server at https://gerrit.sqlalchemy.org. I move PRs immediately into gerrit for testing and that's also where I do all my work on SQLAlchemy / Alembic / dogpile /etc. My own Jenkins instance is using the gerrit plugin to add verified status to gerrits. I'm right now working on an improved PR flow where github PRs would go straight into Gerrit based on a reviewer add right up front.
-- Jeremy Stanley
On 2018-12-14 13:03:07 -0500 (-0500), Mike Bayer wrote: [...]
not sure if this is an option but I'd be more interested if this would be possible from my Gerrit server at https://gerrit.sqlalchemy.org. [...]
Oh, yep, zuul.openstack.org can also connect to multiple Gerrit deployments, so if that's where you're doing the bulk of your code review it's possible for us to provide third-party CI results there instead. -- Jeremy Stanley
On Fri, Dec 14, 2018 at 1:56 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2018-12-14 13:03:07 -0500 (-0500), Mike Bayer wrote: [...]
not sure if this is an option but I'd be more interested if this would be possible from my Gerrit server at https://gerrit.sqlalchemy.org. [...]
Oh, yep, zuul.openstack.org can also connect to multiple Gerrit deployments, so if that's where you're doing the bulk of your code review it's possible for us to provide third-party CI results there instead.
OK, so first step is...identify some zuul jobs? I have CI for sqlalchemy, alembic, dogpile-cache that each would benefit from db /cache-centric builds.
-- Jeremy Stanley
On 2018-12-14 17:20:05 -0500 (-0500), Mike Bayer wrote: [...]
OK, so first step is...identify some zuul jobs? I have CI for sqlalchemy, alembic, dogpile-cache that each would benefit from db /cache-centric builds.
I mean, if there's a job which was failing immediately after the release in our CI system and we think that's a reasonable integration test to perform, we could work out a way to have it install dogpile from the master branch source instead of a released package. I've not been following that part of the discussion closely (maybe some OpenStack-SDK job?) but presumably others on this thread can make recommendations. -- Jeremy Stanley
On Fri, Dec 14, 2018 at 11:32:00PM +0000, Jeremy Stanley wrote:
On 2018-12-14 17:20:05 -0500 (-0500), Mike Bayer wrote: [...]
OK, so first step is...identify some zuul jobs? I have CI for sqlalchemy, alembic, dogpile-cache that each would benefit from db /cache-centric builds.
I mean, if there's a job which was failing immediately after the release in our CI system and we think that's a reasonable integration test to perform, we could work out a way to have it install dogpile from the master branch source instead of a released package.
I would suggest we use the nodepool-functional-py35-src jobs. These jobs are failing [1] These jobs install a devstack environment, then nodepool, openstacksdk, diskimage-builder and glean from git; then they build an image with dib, upload it to the devstack cloud and ensure it boots and communicates. This means openstacksdk (and hence dogpile) is being used by nodepool against a real-life cloud doing very real-life things like creating servers, listing them and connecting all the various bits up. Connecting to dogpile's gerrit as 3rd party CI is possible. But I'd suggest we KISS to start with. As mentioned above, cloning from github master during the job is a good place to start, as I'm not sure at this point if devstack correctly handles installing non-openstack dependencies from Zuul checkouts during test runs. Reviews: https://review.openstack.org/625467 Add github dogpile.cache to project list https://review.openstack.org/625457 [wip] Add dogpile.cache master to the -src tests -i [1] http://logs.openstack.org/55/624855/3/check/nodepool-functional-py35-src/c63...
participants (4)
-
Ian Wienand
-
Jeremy Stanley
-
Mike Bayer
-
Morgan Fainberg