[dev][cinder][keystone] Properly consuming system-scope in cinder
Hey folks, As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0]. The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1] Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests. In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6]. This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7]. Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more. First, we remove project IDs from cinder's API path. This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot. Second, we update cinder's clients, including tempest, to put the project ID on the URL. After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone. Thoughts? [0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e... [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
On 28/01, Lance Bragstad wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance, Sorry to hear that the Cinder is giving you such trouble.
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests? I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID? If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now. So I think your second proposal is the way to go. Thanks for all the work you are putting into this. Cheers, Gorka.
[0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e... [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
---- On Fri, 29 Jan 2021 11:23:47 -0600 Gorka Eguileor <geguileo@redhat.com> wrote ----
On 28/01, Lance Bragstad wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance,
Sorry to hear that the Cinder is giving you such trouble.
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
We can do it by supporting both URL with and without project_id. Nova did the same way in Mitaka cycle and also bumped the microversion but just for notification. It was done in 2.18 microversion[1]. That way you can request compute API with or without project_id and later is recommended. I think the same approach Cinder can consider. [1] https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht... -gmann
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests?
I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID?
If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now.
So I think your second proposal is the way to go.
Thanks for all the work you are putting into this.
Cheers, Gorka.
[0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e... [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
On 29/01, Ghanshyam Mann wrote:
---- On Fri, 29 Jan 2021 11:23:47 -0600 Gorka Eguileor <geguileo@redhat.com> wrote ----
On 28/01, Lance Bragstad wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance,
Sorry to hear that the Cinder is giving you such trouble.
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
We can do it by supporting both URL with and without project_id. Nova did the same way
Hi, I was not doubting that this was technically possible, I was arguing that a change that affects every single API endpoint in Cinder would not be described as "micro" and doing so could be considered a bit of abuse to the microversion infrastructure. This is just my opinion, not the Cinder official position.
in Mitaka cycle and also bumped the microversion but just for notification. It was done in 2.18 microversion[1].
That way you can request compute API with or without project_id and later is recommended. I think the same approach Cinder can consider.
Thanks for this information. It will definitely come in handy knowing where we have code references if we decide to go with the microversion route. Cheers, Gorka.
[1] https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht...
-gmann
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests?
I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID?
If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now.
So I think your second proposal is the way to go.
Thanks for all the work you are putting into this.
Cheers, Gorka.
[0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e... [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
On Mon, Feb 1, 2021 at 5:07 AM Gorka Eguileor <geguileo@redhat.com> wrote:
On 29/01, Ghanshyam Mann wrote:
---- On Fri, 29 Jan 2021 11:23:47 -0600 Gorka Eguileor <geguileo@redhat.com> wrote ----
On 28/01, Lance Bragstad wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance,
Sorry to hear that the Cinder is giving you such trouble.
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
We can do it by supporting both URL with and without project_id. Nova did the same way
Hi,
I was not doubting that this was technically possible, I was arguing that a change that affects every single API endpoint in Cinder would not be described as "micro" and doing so could be considered a bit of abuse to the microversion infrastructure.
This is just my opinion, not the Cinder official position.
in Mitaka cycle and also bumped the microversion but just for notification. It was done in 2.18 microversion[1].
That way you can request compute API with or without project_id and later is recommended. I think the same approach Cinder can consider.
Thanks for this information. It will definitely come in handy knowing where we have code references if we decide to go with the microversion route.
Cheers, Gorka.
[1] https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht...
-gmann
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests?
I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID?
If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now.
Hi, As I've discussed in private with you folks, manila has the exact same issue as cinder in this domain. As has been mentioned in this thread, having the project_id in the API routes was something that most projects forked out of nova had, since the beginning. The manila team weighed the options we had and we concluded that the nova approach [1] of advertising routes without project_id and raising the API microversion is a good step for the API consumers. This is a grey area for micro versioning, I agree - however, we're maintaining backwards compatibility by not breaking any consumer that would like to use project_ids in the v2 URLs. It would be a more painful pill to swallow if we wanted to make this change at another major API microversion given all the client automation, sdks and CLIs written against the micro-versioned API. You could also, in theory, make an argument that the "project_id" resolution/substitution is owned by Keystone and the service catalog mechanism - it is controlled by the cloud administrator who creates these endpoints/services, and the catalog endpoints can be updated at any time. These sort of changes are outside of the service API's control. For example, the catalog entry for manila today can be: https://192.168.10.1/share/v2/{project_id} and the administrator may decide to rename it to: https://shared-file-systems.fancycloud.org/v2/{project_id} We shouldn't expect this to break anyone if the application always used the service catalog as the source of truth, and hence, after upgrading to Wallaby, if administrators can change the catalog endpoint to remove the project_id, it shouldn't affect applications negatively - and no matter our choice of major or minor versioning of the API here, endpoint URL changes are really outside of our control. manila's changes to remove the "project_id" requirement from URLs are here: https://review.opendev.org/q/topic:%22bp/remove-project-id-from-urls%22 [1] https://review.opendev.org/233076
So I think your second proposal is the way to go.
Thanks for all the work you are putting into this.
Cheers, Gorka.
[0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e... [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
On Fri, Jan 29, 2021 at 11:24 AM Gorka Eguileor <geguileo@redhat.com> wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed
service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of
On 28/01, Lance Bragstad wrote: that the
assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance,
Sorry to hear that the Cinder is giving you such trouble.
No worries - I think these types of issues just come with the territory when we're trying to make such large changes across different projects and APIs. :)
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests?
I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
I think that's an option. For example, if I add a fourth cinder service to my catalog without project ID templating in the endpoint URL [0], my system-scoped token contains endpoints for that cinder service [1]. Project-scoped tokens contain all four cinder services, and each service has endpoints populated since URL templating works with project IDs [2]. So, maybe the question becomes, what should the service type and name be for this new entry? I think the cinder clients would need to know this new service is a actually a cinder endpoint without the project ID in the URL. It'll also need to know it's on the hook for appending a project ID to the URL. [0] http://paste.openstack.org/show/802148/ [1] http://paste.openstack.org/show/802145/ (lines 66 - 78) [2] http://paste.openstack.org/show/802149/
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID?
If cinder always assumes and validates a project_id exists in the URL path, then clients making requests, regardless of the scope (system, domain, or project), will need to find a project to use, or use a fake project. And I think the answer to that depends on how cinder wants to handle that validation in the API. I believe this is where that's handled now [3]. If the client sends a request with a system-scoped token in the X-Auth-Token header, cinder could detect that and ignore whatever the project ID is from the path. So maybe it can be fake? At least until we figure out if we can version the API to introduce a project-less URL path? [3] https://opendev.org/openstack/cinder/src/commit/500f5100c8531b4995537c495238...
If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now.
So I think your second proposal is the way to go.
Thanks for all the work you are putting into this.
Cheers, Gorka.
[0]
https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac
[1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6]
https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e...
[7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
Circling back on this topic. I marked all the patches that incorporate system-scope support as WIP [0]. I think we can come back to these after we have a chance to decouple project IDs from cinder's API in Xena. I imagine that's going to be a pretty big change so we can push those reviews to the back burner for now. In the meantime, I reproposed all patches that touch the ADMIN_OR_OWNER rule and updated them to use the member and reader roles [1]. I also removed any system-scope policies from those patches. The surface area of these changes is a lot less than what we were originally expecting to get done for Wallaby. These changes should at least allow operators to use the member and reader roles on projects consistently with cinder when Wallaby goes out the door. To recap, this would mean anyone with the admin role on a project is still considered a system administrator in cinder (we can try and fix this in Xena). Operators can now use the member role to denote owners and give users the reader role on a project and those users shouldn't be able to make writable changes within cinder. [0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac+labe... [1] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac+labe... On Fri, Jan 29, 2021 at 11:24 AM Gorka Eguileor <geguileo@redhat.com> wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed
service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of
On 28/01, Lance Bragstad wrote: that the
assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance,
Sorry to hear that the Cinder is giving you such trouble.
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests?
I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID?
If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now.
So I think your second proposal is the way to go.
Thanks for all the work you are putting into this.
Cheers, Gorka.
[0]
https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac
[1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6]
https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e...
[7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
Brian and I had a discussion about all of this yesterday and we revisited the idea of a project-less URL template. This would allow us to revisit system-scope support for Wallaby under the assumption the client handles project IDs properly for system-scoped requests and cinder relaxes its project ID validation for system-scoped contexts. It's possible to get a cinder endpoint in the service catalog if you create a separate endpoint without project ID templating in the URL. I hacked this together in devstack [0] using a couple of changes to python-cinderclient [1] and cinder's API [2]. After that, I was able to list all volumes in a deployment as a system-administrator (using a system-scoped admin token) [3]. The only hiccup I hit was that I was supplying two endpoints for the volumev3 service. If the endpoint without project ID templating appears first in the catalog for project-scoped tokens, then requests to cinder will fail because the project ID isn't in the URL. Remember, the only cinder endpoint in the catalog for system-scoped tokens was the one without templating, so this issue doesn't appear there. Also, we would need a separate patch to the tempest volume client before we could add any system-scope testing there. Thoughts? [0] https://review.opendev.org/c/openstack/devstack/+/776520 [1] https://review.opendev.org/c/openstack/python-cinderclient/+/776469 [2] https://review.opendev.org/c/openstack/cinder/+/776468 [3] http://paste.openstack.org/show/802786/ On Wed, Feb 17, 2021 at 12:11 PM Lance Bragstad <lbragstad@gmail.com> wrote:
Circling back on this topic.
I marked all the patches that incorporate system-scope support as WIP [0]. I think we can come back to these after we have a chance to decouple project IDs from cinder's API in Xena. I imagine that's going to be a pretty big change so we can push those reviews to the back burner for now.
In the meantime, I reproposed all patches that touch the ADMIN_OR_OWNER rule and updated them to use the member and reader roles [1]. I also removed any system-scope policies from those patches. The surface area of these changes is a lot less than what we were originally expecting to get done for Wallaby. These changes should at least allow operators to use the member and reader roles on projects consistently with cinder when Wallaby goes out the door.
To recap, this would mean anyone with the admin role on a project is still considered a system administrator in cinder (we can try and fix this in Xena). Operators can now use the member role to denote owners and give users the reader role on a project and those users shouldn't be able to make writable changes within cinder.
[0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac+labe... [1] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac+labe...
On Fri, Jan 29, 2021 at 11:24 AM Gorka Eguileor <geguileo@redhat.com> wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed
service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of
On 28/01, Lance Bragstad wrote: that the
assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance,
Sorry to hear that the Cinder is giving you such trouble.
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests?
I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID?
If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now.
So I think your second proposal is the way to go.
Thanks for all the work you are putting into this.
Cheers, Gorka.
[0]
https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac
[1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6]
https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e...
[7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
---- On Thu, 18 Feb 2021 17:53:06 -0600 Lance Bragstad <lbragstad@gmail.com> wrote ----
Brian and I had a discussion about all of this yesterday and we revisited the idea of a project-less URL template. This would allow us to revisit system-scope support for Wallaby under the assumption the client handles project IDs properly for system-scoped requests and cinder relaxes its project ID validation for system-scoped contexts.
It's possible to get a cinder endpoint in the service catalog if you create a separate endpoint without project ID templating in the URL. I hacked this together in devstack [0] using a couple of changes to python-cinderclient [1] and cinder's API [2]. After that, I was able to list all volumes in a deployment as a system-administrator (using a system-scoped admin token) [3]. The only hiccup I hit was that I was supplying two endpoints for the volumev3 service. If the endpoint without project ID templating appears first in the catalog for project-scoped tokens, then requests to cinder will fail because the project ID isn't in the URL. Remember, the only cinder endpoint in the catalog for system-scoped tokens was the one without templating, so this issue doesn't appear there. Also, we would need a separate patch to the tempest volume client before we could add any system-scope testing there.
Thoughts?
To solve the issue of which service catalog Tempest service clients should query, We can register the new endpoint with new name. In nova case when we moved the URL without project-id, we did move the old endpoint (with project_id) with name 'compute_legacy' and added new endpoint (without project-id)to 'compute' which is the default service catalog in Tempest to query[2]. Same way we can do for cinder too, the new endpoint with the name 'volumev3' (default catalog for cinder in Tempest) and old one can be moved to 'volumev3_legacy'. And to keep testing the old endpoint, we can add a separate job to query on old endpoint and rest of everything default to new endpoint. [1] https://github.com/openstack/devstack/blob/e53142ed0d314f07d974a104005be2120... [2] https://github.com/openstack/tempest/blob/fa0a40b8bbc4f7e93a976f5575f8ad7c18... -gmann
[0] https://review.opendev.org/c/openstack/devstack/+/776520[1] https://review.opendev.org/c/openstack/python-cinderclient/+/776469[2] https://review.opendev.org/c/openstack/cinder/+/776468[3] http://paste.openstack.org/show/802786/
On Wed, Feb 17, 2021 at 12:11 PM Lance Bragstad <lbragstad@gmail.com> wrote: Circling back on this topic. I marked all the patches that incorporate system-scope support as WIP [0]. I think we can come back to these after we have a chance to decouple project IDs from cinder's API in Xena. I imagine that's going to be a pretty big change so we can push those reviews to the back burner for now.
In the meantime, I reproposed all patches that touch the ADMIN_OR_OWNER rule and updated them to use the member and reader roles [1]. I also removed any system-scope policies from those patches. The surface area of these changes is a lot less than what we were originally expecting to get done for Wallaby. These changes should at least allow operators to use the member and reader roles on projects consistently with cinder when Wallaby goes out the door.
To recap, this would mean anyone with the admin role on a project is still considered a system administrator in cinder (we can try and fix this in Xena). Operators can now use the member role to denote owners and give users the reader role on a project and those users shouldn't be able to make writable changes within cinder.
[0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac+label:Workflow%253C0[1] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac+labe... On Fri, Jan 29, 2021 at 11:24 AM Gorka Eguileor <geguileo@redhat.com> wrote: On 28/01, Lance Bragstad wrote:
Hey folks,
As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0].
The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1]
Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests.
In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6].
This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7].
Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more.
Hi Lance,
Sorry to hear that the Cinder is giving you such trouble.
First, we remove project IDs from cinder's API path.
This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot.
In my opinion such a change should not be treated as a microversion and would require us to go into v4, which is not something that is feasible in the short term.
Second, we update cinder's clients, including tempest, to put the project ID on the URL.
After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the
I'm not familiar with keystone catalog entries, so maybe I'm saying something stupid, but couldn't we have multiple entries? A project-specific URL and another one for the project and system scoped requests?
I know it sounds kind of hackish, but if we add them in the right order, first the project one and then the new one, it would probably be backward compatible, as older clients would get the first endpoint and new clients would be able to select the right one.
admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone.
What would happen in this additional roundtrip? Would we be converting provided project's name into its UUID?
If that's the case then it wouldn't happen when UUIDs are being provided, so for cases where this extra request means a performance problem they could just provide the UUID.
Thoughts?
Truth is that I would love to see the Cinder API move into URLs without the project id as well as move out everything from contrib, but that doesn't seem like a realistic piece of work we can bite right now.
So I think your second proposal is the way to go.
Thanks for all the work you are putting into this.
Cheers, Gorka.
[0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3e... [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/
participants (4)
-
Ghanshyam Mann
-
Gorka Eguileor
-
Goutham Pacha Ravi
-
Lance Bragstad