Open Stack

On Wednesday, July 16, 2014, Joe Gordon <joe.gordon0 at gmail.com> wrote:



On Tue, Jul 15, 2014 at 7:20 AM, Morgan Fainberg <morgan.fainberg at gmail.com> wrote:


On Tuesday, July 15, 2014, Steven Hardy <shardy at redhat.com> wrote:
On Mon, Jul 14, 2014 at 02:43:19PM -0400, Adam Young wrote:
> On 07/14/2014 11:47 AM, Steven Hardy wrote:
> >Hi all,
> >
> >I'm probably missing something, but can anyone please tell me when devstack
> >will be moving to keystone v3, and in particular when API auth_token will
> >be configured such that auth_version is v3.0 by default?
> >
> >Some months ago, I posted this patch, which switched auth_version to v3.0
> >for Heat:
> >
> >https://review.openstack.org/#/c/80341/
> >
> >That patch was nack'd because there was apparently some version discovery
> >code coming which would handle it, but AFAICS I still have to manually
> >configure auth_version to v3.0 in the heat.conf for our API to work
> >properly with requests from domains other than the default.
> >
> >The same issue is observed if you try to use non-default-domains via
> >python-heatclient using this soon-to-be-merged patch:
> >
> >https://review.openstack.org/#/c/92728/
> >
> >Can anyone enlighten me here, are we making a global devstack move to the
> >non-deprecated v3 keystone API, or do I need to revive this devstack patch?
> >
> >The issue for Heat is we support notifications from "stack domain users",
> >who are created in a heat-specific domain, thus won't work if the
> >auth_token middleware is configured to use the v2 keystone API.
> >
> >Thanks for any information :)
> >
> >Steve
> There are reviews out there in client land now that should work.  I was
> testing discover just now and it seems to be doing the right thing.  If the
> AUTH_URL is chopped of the V2.0 or V3 the client should be able to handle
> everything from there on forward.

Perhaps I should restate my problem, as I think perhaps we still have
crossed wires:

- Certain configurations of Heat *only* work with v3 tokens, because we
  create users in a non-default domain
- Current devstack still configures versioned endpoints, with v2.0 keystone
- Heat breaks in some circumstances on current devstack because of this.
- Adding auth_version='v3.0' to the auth_token section of heat.conf fixes
  the problem.

So, back in March, client changes were promised to fix this problem, and
now, in July, they still have not - do I revive my patch, or are fixes for
this really imminent this time?

Basically I need the auth_token middleware to accept a v3 token for a user
in a non-default domain, e.g validate it *always* with the v3 API not v2.0,
even if the endpoint is still configured versioned to v2.0.

Sorry to labour the point, but it's frustrating to see this still broken
so long after I proposed a fix and it was rejected.


We just did a test converting over the default to v3 (and falling back to v2 as needed, yes fallback will still be needed) yesterday (Dolph posted a couple of test patches and they seemed to succeed - yay!!) It looks like it will just work. Now there is a big caveate, this default will only change in the keystone middleware project, and it needs to have a patch or three get through gate converting projects over to use it before we accept the code.

Nova has approved the patch to switch over, it is just fighting with Gate. Other patches are proposed for other projects and are in various states of approval.

I assume you mean switch over to keystone middleware project [0], not switch over to keystone v3. Based on [1] my understanding is no changes to nova are needed to use the v2 compatible parts of the v3 API, But are changes needed to support domains or is this not a problem because the auth middleware uses uuids for user_id and project_id, so nova doesn't need to have any concept of domains? Are any nova changes needed to support the v3 API?


 
This change simply makes it so the middleware will prefer v3 over v2 if both are available for validating UUID tokens and fetching certs. It still falls back to v2 as needed. It is transparent to all services (it was blocking on Nova and some uniform catalog related issues a while back, but Jamie Lennox resolved those, see below for more details).

It does not mean Nova (or anyone else) are magically using features they weren't already using. It just means Heat isn't needing to do a bunch of conditional stuff to get the V3 information out of the middleware. This change is only used in the case that V2 and V3 are available when auth_token middleware looks at the auth_url (limited discovery). It is still possible to force V2 by setting the ‘identity_uri' to the V2.0 specific root (no discovery performed).

Switching over the default to v3 in the middleware doesn't test nova + v3 user tokens, tempest nova tests don't generate v3 user tokens (although I hear there is an experimental job to do this).  So you are testing that moving the middleware to v3 but accepting v2 API user tokens works. But what happens if someone tries to use a the non-default domain? Or using other v3 only features? Switching over to v3 for the middleware without actually testing any v3 user facing features sounds like a big testing gap.
I agree that we should increase our testing for V3 (specifically in tempest). The question as to what happens when a non-default-domain token is passed to Nova has the same answer as if a PKI token with a non-default domain is passed to Nova via the middleware. The logic in handling a V3 token vs a V2 token has not changed in that regard.
 
The biggest difference between a V2 and V3 token once it gets past the auth_token middleware is the catalog. This is addressed with https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token.py#L975-L976 . We build the header values passed from the middleware here: https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token.py#L932-L967 . This will pull the values from known locations from the token, meaning that the headers should be the same regardless of token version (with exception of the catalog and we’re doing conversions now). If the application doesn’t use the data from the header, (some v3-specific values), it shouldn’t affect anything .

I see the keystone middleware patch has landed [3]

Please note that this does not mean we have *released* the updated keystonemiddleware package. The current released version does not have this change. This change is an important one on the road to getting us to V3 across the board.
 

[0] https://review.openstack.org/#/c/102342/
[1] http://docs.openstack.org/developer/keystone/http-api.htm
[2] http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/models.py#n200
[3] https://review.openstack.org/#/c/106819


I hope this has helped with some of the concerns on how this impacts Nova and other services.

Cheers,

Morgan