<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 17, 2014 at 7:17 AM, Dolph Mathews <span dir="ltr"><<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Jul 17, 2014 at 7:56 AM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">
<div><div>On Wed, Jul 16, 2014 at 5:07 PM, Morgan Fainberg <span dir="ltr"><<a href="mailto:morgan.fainberg@gmail.com" target="_blank">morgan.fainberg@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Reposted now will a lot less bad quote issues. Thanks for being patient with the re-send!<br>
<br>
------------------------------------------------------<br>
From: Joe Gordon <a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a><br>
Reply: OpenStack Development Mailing List (not for usage questions) <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
Date: July 16, 2014 at 02:27:42<br>
To: OpenStack Development Mailing List (not for usage questions) <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
Subject: Re: [openstack-dev] [devstack][keystone] Devstack, auth_token and keystone v3<br>
<div><br>
> On Tue, Jul 15, 2014 at 7:20 AM, Morgan Fainberg<br>
</div>> wrote:<br>
<div><div>><br>
> ><br>
> ><br>
> > On Tuesday, July 15, 2014, Steven Hardy wrote:<br>
> ><br>
> >> On Mon, Jul 14, 2014 at 02:43:19PM -0400, Adam Young wrote:<br>
> >> > On 07/14/2014 11:47 AM, Steven Hardy wrote:<br>
> >> > >Hi all,<br>
> >> > ><br>
> >> > >I'm probably missing something, but can anyone please tell me when<br>
> >> devstack<br>
> >> > >will be moving to keystone v3, and in particular when API auth_token<br>
> >> will<br>
> >> > >be configured such that auth_version is v3.0 by default?<br>
> >> > ><br>
> >> > >Some months ago, I posted this patch, which switched auth_version to<br>
> >> v3.0<br>
> >> > >for Heat:<br>
> >> > ><br>
> >> > ><a href="https://review.openstack.org/#/c/80341/" target="_blank">https://review.openstack.org/#/c/80341/</a><br>
> >> > ><br>
> >> > >That patch was nack'd because there was apparently some version<br>
> >> discovery<br>
> >> > >code coming which would handle it, but AFAICS I still have to manually<br>
> >> > >configure auth_version to v3.0 in the heat.conf for our API to work<br>
> >> > >properly with requests from domains other than the default.<br>
> >> > ><br>
> >> > >The same issue is observed if you try to use non-default-domains via<br>
> >> > >python-heatclient using this soon-to-be-merged patch:<br>
> >> > ><br>
> >> > ><a href="https://review.openstack.org/#/c/92728/" target="_blank">https://review.openstack.org/#/c/92728/</a><br>
> >> > ><br>
> >> > >Can anyone enlighten me here, are we making a global devstack move to<br>
> >> the<br>
> >> > >non-deprecated v3 keystone API, or do I need to revive this devstack<br>
> >> patch?<br>
> >> > ><br>
> >> > >The issue for Heat is we support notifications from "stack domain<br>
> >> users",<br>
> >> > >who are created in a heat-specific domain, thus won't work if the<br>
> >> > >auth_token middleware is configured to use the v2 keystone API.<br>
> >> > ><br>
> >> > >Thanks for any information :)<br>
> >> > ><br>
> >> > >Steve<br>
> >> > There are reviews out there in client land now that should work. I was<br>
> >> > testing discover just now and it seems to be doing the right thing. If<br>
> >> the<br>
> >> > AUTH_URL is chopped of the V2.0 or V3 the client should be able to<br>
> >> handle<br>
> >> > everything from there on forward.<br>
> >><br>
> >> Perhaps I should restate my problem, as I think perhaps we still have<br>
> >> crossed wires:<br>
> >><br>
> >> - Certain configurations of Heat *only* work with v3 tokens, because we<br>
> >> create users in a non-default domain<br>
> >> - Current devstack still configures versioned endpoints, with v2.0<br>
> >> keystone<br>
> >> - Heat breaks in some circumstances on current devstack because of this.<br>
> >> - Adding auth_version='v3.0' to the auth_token section of heat.conf fixes<br>
> >> the problem.<br>
> >><br>
> >> So, back in March, client changes were promised to fix this problem, and<br>
> >> now, in July, they still have not - do I revive my patch, or are fixes for<br>
> >> this really imminent this time?<br>
> >><br>
> >> Basically I need the auth_token middleware to accept a v3 token for a user<br>
> >> in a non-default domain, e.g validate it *always* with the v3 API not<br>
> >> v2.0,<br>
> >> even if the endpoint is still configured versioned to v2.0.<br>
> >><br>
> >> Sorry to labour the point, but it's frustrating to see this still broken<br>
> >> so long after I proposed a fix and it was rejected.<br>
> >><br>
> >><br>
> > We just did a test converting over the default to v3 (and falling back to<br>
> > v2 as needed, yes fallback will still be needed) yesterday (Dolph posted a<br>
> > couple of test patches and they seemed to succeed - yay!!) It looks like it<br>
> > will just work. Now there is a big caveate, this default will only change<br>
> > in the keystone middleware project, and it needs to have a patch or three<br>
> > get through gate converting projects over to use it before we accept the<br>
> > code.<br>
> ><br>
> > Nova has approved the patch to switch over, it is just fighting with Gate.<br>
> > Other patches are proposed for other projects and are in various states of<br>
> > approval.<br>
> ><br>
><br>
> I assume you mean switch over to keystone middleware project [0], not<br>
<br>
</div></div>Correct, switch to middleware (a requirement before we landed this patch in middleware). I was unclear in that statement. Sorry didn’t mean to make anyone jumpy that something was approved in Nova that shouldn’t have been or that did massive re-workings internal to Nova.<br>
<div><br>
> switch over to keystone v3. Based on [1] my understanding is no changes to<br>
> nova are needed to use the v2 compatible parts of the v3 API, But are<br>
> changes needed to support domains or is this not a problem because the auth<br>
> middleware uses uuids for user_id and project_id, so nova doesn't need to<br>
> have any concept of domains? Are any nova changes needed to support the v3<br>
> API?<br>
> <br>
<br>
</div>This change simply makes it so the middleware will prefer v3 over v2 if both are available <br>
for validating UUID tokens and fetching certs. It still falls back to v2 as needed. It <br>
is transparent to all services (it was blocking on Nova and some uniform catalog related <br>
issues a while back, but Jamie Lennox resolved those, see below for more details). <br>
<br>
It does not mean Nova (or anyone else) are magically using features they weren't already <br>
using. It just means Heat isn't needing to do a bunch of conditional stuff to get the V3 <br>
information out of the middleware. This change is only used in the case that V2 and V3 are <br>
available when auth_token middleware looks at the auth_url (limited discovery). It <br>
is still possible to force V2 by setting the ‘identity_uri' to the V2.0 specific root <br>
(no discovery performed). <br></blockquote><div><br></div><div><br></div></div></div><div>So this will be used in the gate?</div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
><br>
> Switching over the default to v3 in the middleware doesn't test nova + v3<br>
> user tokens, tempest nova tests don't generate v3 user tokens (although I<br>
> hear there is an experimental job to do this). So you are testing that<br>
> moving the middleware to v3 but accepting v2 API user tokens works. But<br>
> what happens if someone tries to use a the non-default domain? Or using<br>
> other v3 only features? Switching over to v3 for the middleware without<br>
> actually testing any v3 user facing features sounds like a big testing gap.<br>
> <br>
<br>
</div>I agree that we should increase our testing for V3 (specifically in tempest). For question as to what happens when a non-default-domain token is passed to Nova has the same answer as if a PKI token with a non-default domain is passed to Nova via the middleware. The logic in handling a V3 token vs a V2 token has not changed in that regard. <br>
</blockquote><div><br></div></div><div>I wasn't aware that PKI tokens had domains in them. What happens to nova in this case, It just works?</div></div></div></div></blockquote><div><br></div></div></div><div>Both PKI and UUID responses from v3 contain:</div>
<div><br></div><div>1. the user's domain</div><div><br></div><div>And if it's a project scoped token:</div><div><br></div><div>2. the project's domain</div><div><br></div><div>Or if it's a domain-scoped token:</div>
<div><br></div><div>3. a domain scope</div><div><br></div><div>The answer to your question is that if nova receives a project-scoped token (1 & 2), it doesn't need to be domain-aware: project ID's are globally unique and nova doesn't need to know about project-domain relationships.</div>
<div><br></div><div>If nova receives a domain-scoped token (1 & 3), the policy layer can balk with an HTTP 401 because there's no project in scope, and it's not domain-aware. From nova's perspective, this is identical to the scenario where the policy layer returns an HTTP 401 because nova was presented with an unscoped token (1 only) from keystone.</div>
</div></div></div></blockquote><div><br></div><div>You say the policy layer "can" balk with an HTTP 401. What is required to make this the default behavior in nova's policy layer?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><div dir="ltr"><div class="gmail_extra">
<div class="gmail_quote">
<div><br></div><div>Based on <a href="https://review.openstack.org/#/c/103617/2/specs/juno/support-keystone-v3-api.rst" target="_blank">https://review.openstack.org/#/c/103617/2/specs/juno/support-keystone-v3-api.rst</a> it sounds like just switching over the keystonemiddleware and the novaclient to support keystone v3 nova will support keystone v3 (domains and all). But until a nova patch updates the neutron config section in nova, nova->neutron communication will be over keystone v2. Is that correct?</div>
<div>
<div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
The biggest difference between a V2 and V3 token once it gets past the auth_token middleware is the catalog. This is addressed with <a href="https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token.py#L975-L976" target="_blank">https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token.py#L975-L976</a> . We build the header values passed from the middleware here: <a href="https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token.py#L932-L967" target="_blank">https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token.py#L932-L967</a> . This will pull the values from known locations from the token, meaning that the headers should be the same regardless of token version (with exception of the catalog and we’re doing conversions now). If the application doesn’t use the data from the header, (some v3-specific values), it shouldn’t affect anything. <br>
<div><br>
> I see the keystone middleware patch has landed [3]<br>
<br>
</div>Please note that this does not mean we have *released* the updated keystonemiddleware package. The current released version does not have this change. This change is an important one on the road to getting us to V3 across the board. <br>
</blockquote><div><br></div></div><div>Ahh, thanks for the clarification.</div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
><br>
> [0] <a href="https://review.openstack.org/#/c/102342/" target="_blank">https://review.openstack.org/#/c/102342/</a><br>
> [1] <a href="http://docs.openstack.org/developer/keystone/http-api.htm" target="_blank">http://docs.openstack.org/developer/keystone/http-api.htm</a><br>
> [2]<br>
> <a href="http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/models.py#n200" target="_blank">http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/models.py#n200</a><br>
> [3] <a href="https://review.openstack.org/#/c/106819" target="_blank">https://review.openstack.org/#/c/106819</a><br>
<br>
</div>I hope this has helped with some of the concerns on how this impacts Nova and other services. <br>
<br>
Cheers, <br>
<span><font color="#888888">Morgan <br>
<br>
<br>
</font></span></blockquote></div></div><br></div></div>
<br></div><div class="">_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></div></blockquote></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>