[oslo] admin-ness not properly scoped and oslo.policy current status about this issue
Hello ## Overview I want to bring up this topic (admin-ness not properly scoped)[1] to get a big picture of the state of this issue and that was needed on the oslo.policy side. Few weeks ago some RHOSP customers request for an enhancement of oslo.policy since their admin domain can manage other domains. They use OSP13. The goal of this ML thread is to help us to track informations about this topic and I also planned to discuss about this topic during the next oslo meeting (Monday 18 of March). ## Details After some investigations I've found a lot of related issues on launchpad[1][2][3], and a lot of disucssions inside the openstack community about this topic. First I guess it's not an RFE but it's a known issue. This bug has side-effects across several services, not just oslo or keystone, making the fix complex to orchestrate across services. In a first time, I want to know more about the latest events on this topic on the oslo side: - the states of the related specs ( https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-... ). - if we need to add more changes to completely fix this issue and/or if everything is complete on the oslo side and know since which version. I guess this one[4] is related to. Also due to the complexity of this issue I guess is not totally fixed on the whole openstack components on stein and it can't be fully (whole) backported to stable branches, but your point of view is really appreciate. In other words I guess some parts are already fixed on some components but some services still need to be fixed and the issue partially occur on stein, so fix that on stable branches is not really possible, can you confirm? Also I've found few related specs that I guess can be useful to track the evolution: - https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/ca... - https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/de... - https://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/s... - https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-... If I missed something useful do not hesitate to reply on and to share it with us. [1] https://bugs.launchpad.net/keystone/+bug/968696 [2] https://bugs.launchpad.net/keystone/+bug/1783659 [3] https://bugs.launchpad.net/nova/+bug/1649532 [4] https://bugs.launchpad.net/oslo.policy/+bug/1577996 -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Tagging Keystone as I think they are better suited to answering this. A bit more from my limited knowledge inline. On 3/13/19 12:07 PM, Herve Beraud wrote:
Hello
## Overview I want to bring up this topic (admin-ness not properly scoped)[1] to get a big picture of the state of this issue and that was needed on the oslo.policy side.
Few weeks ago some RHOSP customers request for an enhancement of oslo.policy since their admin domain can manage other domains. They use OSP13.
For those not rocking fedoras, OSP 13 corresponds to Queens. :-)
The goal of this ML thread is to help us to track informations about this topic and I also planned to discuss about this topic during the next oslo meeting (Monday 18 of March).
## Details
After some investigations I've found a lot of related issues on launchpad[1][2][3], and a lot of disucssions inside the openstack community about this topic.
First I guess it's not an RFE but it's a known issue.
This bug has side-effects across several services, not just oslo or keystone, making the fix complex to orchestrate across services.
In a first time, I want to know more about the latest events on this topic on the oslo side: - the states of the related specs (https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-...). - if we need to add more changes to completely fix this issue and/or if everything is complete on the oslo side and know since which version. I guess this one[4] is related to.
To my knowledge the Oslo side is done. I think we actually added the necessary fields to oslo.policy (and oslo.context?) at the end of last cycle. I'm not sure where the Keystone side stands, but I'm sure someone from that team can provide an update. Unfortunately, even if Keystone is completely finished, to consume this I _think_ it's going to require policy changes in all of the consuming services, and I don't know that any of those have happened yet. I believe it's a PTG topic for Keystone.
Also due to the complexity of this issue I guess is not totally fixed on the whole openstack components on stein and it can't be fully (whole) backported to stable branches, but your point of view is really appreciate. In other words I guess some parts are already fixed on some components but some services still need to be fixed and the issue partially occur on stein, so fix that on stable branches is not really possible, can you confirm?
Yeah, I don't expect most of this would be backportable, especially all the way to Queens.
Also I've found few related specs that I guess can be useful to track the evolution: - https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/ca... - https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/de... - https://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/s... - https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-...
If I missed something useful do not hesitate to reply on and to share it with us.
[1] https://bugs.launchpad.net/keystone/+bug/968696 [2] https://bugs.launchpad.net/keystone/+bug/1783659 [3] https://bugs.launchpad.net/nova/+bug/1649532 [4] https://bugs.launchpad.net/oslo.policy/+bug/1577996
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Le mer. 13 mars 2019 à 18:33, Ben Nemec <openstack@nemebean.com> a écrit :
Tagging Keystone as I think they are better suited to answering this.
Yep make sense :)
A bit more from my limited knowledge inline.
On 3/13/19 12:07 PM, Herve Beraud wrote:
Hello
## Overview I want to bring up this topic (admin-ness not properly scoped)[1] to get a big picture of the state of this issue and that was needed on the oslo.policy side.
Few weeks ago some RHOSP customers request for an enhancement of oslo.policy since their admin domain can manage other domains. They use OSP13.
For those not rocking fedoras, OSP 13 corresponds to Queens. :-)
The goal of this ML thread is to help us to track informations about this topic and I also planned to discuss about this topic during the next oslo meeting (Monday 18 of March).
## Details
After some investigations I've found a lot of related issues on launchpad[1][2][3], and a lot of disucssions inside the openstack community about this topic.
First I guess it's not an RFE but it's a known issue.
This bug has side-effects across several services, not just oslo or keystone, making the fix complex to orchestrate across services.
In a first time, I want to know more about the latest events on this topic on the oslo side: - the states of the related specs (
https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-... ).
- if we need to add more changes to completely fix this issue and/or if everything is complete on the oslo side and know since which version. I guess this one[4] is related to.
To my knowledge the Oslo side is done. I think we actually added the necessary fields to oslo.policy (and oslo.context?) at the end of last cycle. I'm not sure where the Keystone side stands, but I'm sure someone from that team can provide an update.
Yeah I guess we can bring oslo.context too since these changes like looks to this topic too: https://github.com/openstack/oslo.context/commit/f65408df5cd5924f2879c3ee94d...
Unfortunately, even if Keystone is completely finished, to consume this I _think_ it's going to require policy changes in all of the consuming services, and I don't know that any of those have happened yet. I believe it's a PTG topic for Keystone.
Also due to the complexity of this issue I guess is not totally fixed on the whole openstack components on stein and it can't be fully (whole) backported to stable branches, but your point of view is really appreciate. In other words I guess some parts are already fixed on some components but some services still need to be fixed and the issue partially occur on stein, so fix that on stable branches is not really possible, can you confirm?
Yeah, I don't expect most of this would be backportable, especially all the way to Queens.
Thanks.
Also I've found few related specs that I guess can be useful to track the evolution: -
https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/ca...
-
https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/de...
-
https://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/s...
-
https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-...
If I missed something useful do not hesitate to reply on and to share it with us.
[1] https://bugs.launchpad.net/keystone/+bug/968696 [2] https://bugs.launchpad.net/keystone/+bug/1783659 [3] https://bugs.launchpad.net/nova/+bug/1649532 [4] https://bugs.launchpad.net/oslo.policy/+bug/1577996
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Thanks to Ben for looping keystone into this discussion and providing some initial details. More comments inline below. On 3/13/19 1:00 PM, Herve Beraud wrote:
Le mer. 13 mars 2019 à 18:33, Ben Nemec <openstack@nemebean.com <mailto:openstack@nemebean.com>> a écrit :
Tagging Keystone as I think they are better suited to answering this.
Yep make sense :)
A bit more from my limited knowledge inline.
On 3/13/19 12:07 PM, Herve Beraud wrote: > Hello > > ## Overview > I want to bring up this topic (admin-ness not properly scoped)[1] to get > a big picture of the state of this issue and that was needed on the > oslo.policy side. > > Few weeks ago some RHOSP customers request for an enhancement of > oslo.policy since their admin domain can manage other domains. They use > OSP13.
For those not rocking fedoras, OSP 13 corresponds to Queens. :-)
> > The goal of this ML thread is to help us to track informations about > this topic and I also planned to discuss about this topic during the > next oslo meeting (Monday 18 of March). > > ## Details > > After some investigations I've found a lot of related issues on > launchpad[1][2][3], and a lot of disucssions inside the openstack > community about this topic. > > First I guess it's not an RFE but it's a known issue. > > This bug has side-effects across several services, not just oslo or > keystone, making the fix complex to orchestrate across services.
Yup... Regardless of how we write or develop things in keystone, we have to be good ambassadors to the services looking to consume this work. There have been past proposals to pull *all* policy enforcement and decisions into keystone, but that's prone to performance issues and might hinder development pace for other teams (depending on the implementation).
> > In a first time, I want to know more about the latest events on this > topic on the oslo side: > - the states of the related specs > (https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-...). > - if we need to add more changes to completely fix this issue and/or if > everything is complete on the oslo side and know since which version. I > guess this one[4] is related to.
To my knowledge the Oslo side is done. I think we actually added the necessary fields to oslo.policy (and oslo.context?) at the end of last cycle. I'm not sure where the Keystone side stands, but I'm sure someone from that team can provide an update.
Yeah I guess we can bring oslo.context too since these changes like looks to this topic too: https://github.com/openstack/oslo.context/commit/f65408df5cd5924f2879c3ee94d...
As Ben noted, everything client/library-wise is done. To be verbose: - keystone implemented support for users and groups to have system role assignments in Queens [0][1], along with the ability for users to generate system-scoped tokens - keystoneauth implemented support for building system-scoped authentication requests [2] - python-keystoneclient has support for managing system role assignments just like you'd manage project or domain role assignments [3] - so does python-openstackclient [4] - keystonemiddleware knows what system-scoped tokens are and populates request headers properly [5] (this is important for services to consume system-scoped tokens) - services using oslo.context can generate RequestContext objects from request environments and oslo.context will handle the appropriate attributes if the request was made with a system-scoped token [6] (making consumption even easier) - oslo.policy actually knows how to understand RequestContext objects from oslo.context, making the whole authorization code across services simpler (since services shouldn't have to build credential dictionaries with system scope set in a particular way) [7] - finally, in Rocky keystone formally added support for default roles, which are available for services to write default policies for [8] Long story short, all the plumbing should be in place for services to start consuming system-scoped tokens to protect API accordingly. [0] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/sy... [1] https://review.openstack.org/#/q/topic:bp/system-scope+project:openstack/key... [2] https://review.openstack.org/#/c/529665/ [3] https://review.openstack.org/#/c/524415/ [4] https://review.openstack.org/#/c/524416/ [5] https://review.openstack.org/#/c/564072/ [6] https://review.openstack.org/#/c/530509/ [7] https://review.openstack.org/#/c/578995/ [8] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/def...
Unfortunately, even if Keystone is completely finished, to consume this I _think_ it's going to require policy changes in all of the consuming services, and I don't know that any of those have happened yet. I believe it's a PTG topic for Keystone.
Yes to both. (first yes) We have time set aside during the forum and the PTG to talk about this [9]. The forum session should be more operator specific, so we can answer questions and share knowledge the best we can. We're hoping the PTG session will be developer focused, where we can walk through how services can get from point A to point B (point A being whatever they're doing with policy today that doesn't fall inline with admin-ness and point B being everything enforced at the proper scope allowing more functionality to be exposed to more users). (second yes) Despite keystone being the authorization and authentication service, it's not exempt from these issues or having to consume system scope itself. The team dedicated a bunch of time in Stein to work on this. We certainly encourage other projects to do the same, but we're also hoping to share what experiences we built up in keystone so that it will be easier for other teams. I provided a summary on the status of the keystone-specific work last week [10], including the process we used for breaking everything down. It's apparent that despite the work we've done here, there are still a lot of knowledge gaps in how to actually consume all this stuff, not to mention the overall concept of scopes. We have a document up for review to fix that, too [11][12]. [9] https://etherpad.openstack.org/p/DEN-keystone-forum-sessions [10] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003552.htm... warning: this is a glorious wall of text [11] https://review.openstack.org/#/c/638563/ [12] http://logs.openstack.org/63/638563/9/check/openstack-tox-docs/44b339a/html/... (rendered version)
> > Also due to the complexity of this issue I guess is not totally fixed on > the whole openstack components on stein and it can't be fully (whole) > backported to stable branches, but your point of view is really > appreciate. In other words I guess some parts are already fixed on some > components but some services still need to be fixed and the issue > partially occur on stein, so fix that on stable branches is not really > possible, can you confirm?
Yeah, I don't expect most of this would be backportable, especially all the way to Queens.
Thanks.
Correct. The plumbing alone was spread across 8 different repositories, some of which were libraries. This doesn't include the 100+ patches to keystone that include extensive protection tests or updated policies [13] * to fix our own APIs *. [13] https://review.openstack.org/#/q/(status:open+OR+status:merged)+project:open... * My intention isn't to scare anyone off from tackling this work, but I do want to be upfront in how we approached the problem and how we're addressing it. I think teams will need that information in order to size tasks appropriately and set realistic expectations. For what it's worth, I think the way we broke things down made everything a lot easier for people to chip in and contribute. At the very least, it gave us the opportunity to air out the issue instead of tracking *all* of it against 968696.
> > Also I've found few related specs that I guess can be useful to track > the evolution: > - > https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/ca... > - > https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/de... > - > https://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/s... > - > https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-... > > If I missed something useful do not hesitate to reply on and to share it > with us. > > [1] https://bugs.launchpad.net/keystone/+bug/968696 > [2] https://bugs.launchpad.net/keystone/+bug/1783659 > [3] https://bugs.launchpad.net/nova/+bug/1649532 > [4] https://bugs.launchpad.net/oslo.policy/+bug/1577996 > > -- > Hervé Beraud > Senior Software Engineer > Red Hat - Openstack Oslo > irc: hberaud > -----BEGIN PGP SIGNATURE----- > > wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ > Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ > RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP > F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G > 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g > glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw > m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ > hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 > qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y > F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 > B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O > v6rDpkeNksZ9fFSyoY2o > =ECSj > -----END PGP SIGNATURE----- >
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Thank you Lance for this really useful big picture. Le mer. 13 mars 2019 à 19:50, Lance Bragstad <lbragstad@gmail.com> a écrit :
Thanks to Ben for looping keystone into this discussion and providing some initial details. More comments inline below.
On 3/13/19 1:00 PM, Herve Beraud wrote:
Le mer. 13 mars 2019 à 18:33, Ben Nemec <openstack@nemebean.com> a écrit :
Tagging Keystone as I think they are better suited to answering this.
Yep make sense :)
A bit more from my limited knowledge inline.
On 3/13/19 12:07 PM, Herve Beraud wrote:
Hello
## Overview I want to bring up this topic (admin-ness not properly scoped)[1] to get a big picture of the state of this issue and that was needed on the oslo.policy side.
Few weeks ago some RHOSP customers request for an enhancement of oslo.policy since their admin domain can manage other domains. They use OSP13.
For those not rocking fedoras, OSP 13 corresponds to Queens. :-)
The goal of this ML thread is to help us to track informations about this topic and I also planned to discuss about this topic during the next oslo meeting (Monday 18 of March).
## Details
After some investigations I've found a lot of related issues on launchpad[1][2][3], and a lot of disucssions inside the openstack community about this topic.
First I guess it's not an RFE but it's a known issue.
This bug has side-effects across several services, not just oslo or keystone, making the fix complex to orchestrate across services.
Yup... Regardless of how we write or develop things in keystone, we have to be good ambassadors to the services looking to consume this work. There have been past proposals to pull *all* policy enforcement and decisions into keystone, but that's prone to performance issues and might hinder development pace for other teams (depending on the implementation).
In a first time, I want to know more about the latest events on this topic on the oslo side: - the states of the related specs ( https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-... ). - if we need to add more changes to completely fix this issue and/or if everything is complete on the oslo side and know since which version. I guess this one[4] is related to.
To my knowledge the Oslo side is done. I think we actually added the necessary fields to oslo.policy (and oslo.context?) at the end of last cycle. I'm not sure where the Keystone side stands, but I'm sure someone from that team can provide an update.
Yeah I guess we can bring oslo.context too since these changes like looks to this topic too:
https://github.com/openstack/oslo.context/commit/f65408df5cd5924f2879c3ee94d...
As Ben noted, everything client/library-wise is done.
To be verbose:
- keystone implemented support for users and groups to have system role assignments in Queens [0][1], along with the ability for users to generate system-scoped tokens - keystoneauth implemented support for building system-scoped authentication requests [2] - python-keystoneclient has support for managing system role assignments just like you'd manage project or domain role assignments [3] - so does python-openstackclient [4] - keystonemiddleware knows what system-scoped tokens are and populates request headers properly [5] (this is important for services to consume system-scoped tokens) - services using oslo.context can generate RequestContext objects from request environments and oslo.context will handle the appropriate attributes if the request was made with a system-scoped token [6] (making consumption even easier) - oslo.policy actually knows how to understand RequestContext objects from oslo.context, making the whole authorization code across services simpler (since services shouldn't have to build credential dictionaries with system scope set in a particular way) [7] - finally, in Rocky keystone formally added support for default roles, which are available for services to write default policies for [8]
Long story short, all the plumbing should be in place for services to start consuming system-scoped tokens to protect API accordingly.
[0] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/sy... [1] https://review.openstack.org/#/q/topic:bp/system-scope+project:openstack/key... [2] https://review.openstack.org/#/c/529665/ [3] https://review.openstack.org/#/c/524415/ [4] https://review.openstack.org/#/c/524416/ [5] https://review.openstack.org/#/c/564072/ [6] https://review.openstack.org/#/c/530509/ [7] https://review.openstack.org/#/c/578995/ [8] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/def...
Unfortunately, even if Keystone is completely finished, to consume this I _think_ it's going to require policy changes in all of the consuming services, and I don't know that any of those have happened yet. I believe it's a PTG topic for Keystone.
Yes to both.
(first yes)
We have time set aside during the forum and the PTG to talk about this [9]. The forum session should be more operator specific, so we can answer questions and share knowledge the best we can. We're hoping the PTG session will be developer focused, where we can walk through how services can get from point A to point B (point A being whatever they're doing with policy today that doesn't fall inline with admin-ness and point B being everything enforced at the proper scope allowing more functionality to be exposed to more users).
(second yes)
Despite keystone being the authorization and authentication service, it's not exempt from these issues or having to consume system scope itself. The team dedicated a bunch of time in Stein to work on this. We certainly encourage other projects to do the same, but we're also hoping to share what experiences we built up in keystone so that it will be easier for other teams. I provided a summary on the status of the keystone-specific work last week [10], including the process we used for breaking everything down. It's apparent that despite the work we've done here, there are still a lot of knowledge gaps in how to actually consume all this stuff, not to mention the overall concept of scopes. We have a document up for review to fix that, too [11][12].
[9] https://etherpad.openstack.org/p/DEN-keystone-forum-sessions [10] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003552.htm... warning: this is a glorious wall of text [11] https://review.openstack.org/#/c/638563/ [12] http://logs.openstack.org/63/638563/9/check/openstack-tox-docs/44b339a/html/... (rendered version)
Also due to the complexity of this issue I guess is not totally fixed
on
the whole openstack components on stein and it can't be fully (whole) backported to stable branches, but your point of view is really appreciate. In other words I guess some parts are already fixed on some components but some services still need to be fixed and the issue partially occur on stein, so fix that on stable branches is not really possible, can you confirm?
Yeah, I don't expect most of this would be backportable, especially all the way to Queens.
Thanks.
Correct. The plumbing alone was spread across 8 different repositories, some of which were libraries. This doesn't include the 100+ patches to keystone that include extensive protection tests or updated policies [13] * to fix our own APIs *.
[13] https://review.openstack.org/#/q/(status:open+OR+status:merged)+project:open... * My intention isn't to scare anyone off from tackling this work, but I do want to be upfront in how we approached the problem and how we're addressing it. I think teams will need that information in order to size tasks appropriately and set realistic expectations. For what it's worth, I think the way we broke things down made everything a lot easier for people to chip in and contribute. At the very least, it gave us the opportunity to air out the issue instead of tracking *all* of it against 968696.
Also I've found few related specs that I guess can be useful to track the evolution: -
https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/ca...
-
https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/de...
-
https://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/s...
-
https://specs.openstack.org/openstack/oslo-specs/specs/queens/include-scope-...
If I missed something useful do not hesitate to reply on and to share
it
with us.
[1] https://bugs.launchpad.net/keystone/+bug/968696 [2] https://bugs.launchpad.net/keystone/+bug/1783659 [3] https://bugs.launchpad.net/nova/+bug/1649532 [4] https://bugs.launchpad.net/oslo.policy/+bug/1577996
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
participants (3)
-
Ben Nemec
-
Herve Beraud
-
Lance Bragstad