Open Stack

Hello, Jamie, 

Thanks for your guide. If the item " include_service_catalog" is not configured, then the region name can be used to specify the region for the token validation.

But if I configure" include_service_catalog = False", then the token validation will be redirected to incorrect keystone server.

In multi-site cloud scenario, there are dozens of endpoints, it's reasonable to " include_service_catalog = False".

The log is attached here. 172.17.0.135:35357 and 172.17.0.36:35357 are KeyStone server, our intention is to use the local KeyStone server 172.17.0.135 for token validation, but it forward the request to 172.17.0.36, KeyStone server in another region. 

It seems that override endpoint is a better choice, just like what I did in the https://docs.google.com/document/d/1258g0VTC4wktevo2ymS7SaNhDeY8-S2QWY45them7ZM/edit, ( I just borrowed the configuration item auth_uri, so many close name of configuration item, confused ).

----------------------------------------------------------- 

2015-09-07 09:02:16.257 242 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://172.17.0.135:35357 -H "Accept: application/json" -H "User-Agent
: python-keystoneclient" _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:195
2015-09-07 09:02:16.280 242 DEBUG keystoneclient.session [-] RESP: [300] content-length: 595 vary: X-Auth-Token connection: keep-alive date: Mon, 07 Sep 2
015 09:02:16 GMT content-type: application/json x-distribution: Ubuntu
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "applicat
ion/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://172.17.0.135:35357/v3/", "rel": "self"}]}, {"status": "stable", "updated":
 "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"
href": "http://172.17.0.135:35357/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:223
2015-09-07 09:02:16.280 242 DEBUG keystoneclient.auth.identity.v3 [-] Making authentication request to http://172.17.0.135:35357/v3/auth/tokens get_auth_r
ef /usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3.py:125
2015-09-07 09:02:19.382 242 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://172.17.0.36:35357 -H "Accept: application/json" -H "User-Agent:
 python-keystoneclient" _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:195
2015-09-07 09:02:19.386 242 DEBUG keystoneclient.session [-] RESP: [300] content-length: 593 vary: X-Auth-Token connection: keep-alive date: Mon, 07 Sep 2
015 09:02:19 GMT content-type: application/json x-distribution: Ubuntu
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "applicat
ion/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://172.17.0.36:35357/v3/", "rel": "self"}]}, {"status": "stable", "updated":
"2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"h
ref": "http://172.17.0.36:35357/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:223
2015-09-07 09:02:19.387 242 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://172.17.0.36:35357/v3/auth/tokens?nocatalog -H "X-Subject-Token:
 {SHA1}6e306214e70d1c9547b2d22d6962cefb6354164f" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}f9014aa76c16
2b0db646b325daf813e258c8e2a5" _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:195
2015-09-07 09:02:19.492 242 DEBUG keystoneclient.session [-] RESP: [200] content-length: 491 x-subject-token: {SHA1}6e306214e70d1c9547b2d22d6962cefb635416
4f vary: X-Auth-Token x-distribution: Ubuntu connection: keep-alive date: Mon, 07 Sep 2015 09:02:19 GMT content-type: application/json x-openstack-request
-id: req-c752beb4-c87b-4812-93dc-b2ea00fbf7b1
RESP BODY: {"token": {"methods": ["password"], "roles": [{"id": "a4935779c40f45d3ba7a8eeada0f7714", "name": "admin"}], "expires_at": "2015-09-07T09:02:20.
000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "79185427679a44019d919ce34112c971", "name": "admin"}, "extras": {}, "user": {"
domain": {"id": "default", "name": "Default"}, "id": "3c94437bbd0a455d938ed1d95c7049d1", "name": "osadmin"}, "audit_ids": ["t9QR8DxdSXiQjuJW-nYxfw"], "iss
ued_at": "2015-09-07T08:02:20.000000Z"}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:223
2015-09-07 09:02:19.499 242 DEBUG oslo_policy.openstack.common.fileutils [req-e905e497-2f45-4bb2-a384-ee4f97a23cec 3c94437bbd0a455d938ed1d95c7049d1 791854
27679a44019d919ce34112c971 - - -] Reloading cached file /etc/glance/policy.json read_cached_file /usr/lib/python2.7/dist-packages/oslo_policy/openstack/co
mmon/fileutils.py:64
2015-09-07 09:02:19.499 242 DEBUG oslo_policy.policy [req-e905e497-2f45-4bb2-a384-ee4f97a23cec 3c94437bbd0a455d938ed1d95c7049d1 79185427679a44019d919ce341
12c971 - - -] Reloaded policy file: /etc/glance/policy.json _load_policy_file /usr/lib/python2.7/dist-packages/oslo_policy/policy.py:403
2015-09-07 09:02:19.501 242 DEBUG glance.common.client [req-e905e497-2f45-4bb2-a384-ee4f97a23cec 3c94437bbd0a455d938ed1d95c7049d1 79185427679a44019d919ce3
4112c971 - - -] Constructed URL: http://172.17.0.144:9191/images/detail?sort_key=name&sort_dir=asc&limit=20 _construct_url /usr/lib/python2.7/dist-package
s/glance/common/client.py:401

Best Regards
Chaoyi Huang ( Joe Huang )


-----Original Message-----
From: Jamie Lennox [mailto:jamielennox at redhat.com] 
Sent: Monday, September 07, 2015 1:24 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Keystone][Glance] keystonemiddleware & multiple keystone endpoints



----- Original Message -----
> From: "joehuang" <joehuang at huawei.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev at lists.openstack.org>
> Sent: Sunday, 6 September, 2015 7:28:04 PM
> Subject: Re: [openstack-dev] [Keystone][Glance] keystonemiddleware & 
> multiple keystone endpoints
> 
> Hello, Jamie and Hans,
> 
> The patch " Allow specifying a region name to auth_token "
> https://review.openstack.org/#/c/216579 has just been merged.
> 
> But unfortunately, when I modify the source code as this patch did in 
> the multisite cloud with Fernet token, the issue is still there, and 
> routed to incorrect endpoint.
> 
> I also check the region_name configuration in the source code, it's correct.
> 
> The issue mentioned in the bug report not addressed yet:
> https://bugs.launchpad.net/keystonemiddleware/+bug/1488347
> 
> Is there anyone who tested it successfully in your environment?

Hey Joe, 

The way that this patch is implemented requires you to have configured auth_token middleware with an auth plugin. For example [1]. I should have called this out better in the help for the config option. To make it so that the old admin_user etc options were region aware is kind of a big change because in that case the URL that is configured as identity_uri is always used for all keystone options.

Can you try and configure with the auth plugin options and see if the regions work after that? 


Jamie

[1] http://www.jamielennox.net/blog/2015/02/23/v3-authentication-with-auth-token-middleware/



> The log of Glance API, the request was redirected to 
> http://172.17.0.95:35357, but this address is not a KeyStone endpoint.
> (http://172.17.0.98:35357 and http://172.17.0.41:35357 are correct 
> KeyStone endpoints ) //////////////////////////////////////////
> 2015-09-06 07:50:43.447 194 DEBUG keystoneclient.session [-] REQ: curl 
> -g -i -X GET http://172.17.0.98:35357 -H "Accept: application/json" -H
> "User-Agent: python-keystoneclient" _http_log_request
> /usr/lib/python2.7/dist-packages/keystoneclient/session.py:195
> 2015-09-06 07:50:43.468 194 DEBUG keystoneclient.session [-] RESP: 
> [300]
> content-length: 593 vary: X-Auth-Token connection: keep-alive date: 
> Sun, 06 Sep 2015 07:50:43 GMT content-type: application/json 
> x-distribution: Ubuntu RESP BODY: {"versions": {"values": [{"status": "stable", "updated":
> "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type":
> "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links":
> [{"href": "http://172.17.0.98:35357/v3/", "rel": "self"}]}, {"status":
> "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base":
> "application/json", "type":
> "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links":
> [{"href": "http://172.17.0.98:35357/v2.0/", "rel": "self"}, {"href":
> "http://docs.openstack.org/", "type": "text/html", "rel":
> "describedby"}]}]}}
>  _http_log_response
>  /usr/lib/python2.7/dist-packages/keystoneclient/session.py:223
> 2015-09-06 07:50:43.469 194 DEBUG keystoneclient.auth.identity.v3 [-] 
> Making authentication request to 
> http://172.17.0.98:35357/v3/auth/tokens
> get_auth_ref
> /usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3.py:12
> 5
> 2015-09-06 07:50:43.574 194 DEBUG keystoneclient.session [-] REQ: curl 
> -g -i -X GET http://172.17.0.95:35357 -H "Accept: application/json" -H
> "User-Agent: python-keystoneclient" _http_log_request
> /usr/lib/python2.7/dist-packages/keystoneclient/session.py:195
> 2015-09-06 07:50:46.576 194 WARNING keystoneclient.auth.identity.base 
> [-] Failed to contact the endpoint at http://172.17.0.95:35357 for discovery.
> Fallback to using that endpoint as the base url.
> 2015-09-06 07:50:46.576 194 DEBUG keystoneclient.session [-] REQ: curl 
> -g -i -X GET http://172.17.0.95:35357/auth/tokens -H "X-Subject-Token:
> {SHA1}640964e1f8716ecbb10ca3d8b5b08c8e7abfac1d" -H "User-Agent:
> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
> {SHA1}386777062718e0992cc818780e3ec7fa0671d8e9" _http_log_request
> /usr/lib/python2.7/dist-packages/keystoneclient/session.py:195
> 2015-09-06 07:50:49.576 194 INFO keystoneclient.session [-] Failure: 
> Unable to establish connection to 
> http://172.17.0.95:35357/auth/tokens. Retrying in 0.5s.
> 2015-09-06 07:50:52.576 194 INFO keystoneclient.session [-] Failure: 
> Unable to establish connection to 
> http://172.17.0.95:35357/auth/tokens. Retrying in 1.0s.
> 2015-09-06 07:50:55.576 194 INFO keystoneclient.session [-] Failure: 
> Unable to establish connection to 
> http://172.17.0.95:35357/auth/tokens. Retrying in 2.0s.
> 2015-09-06 07:50:58.576 194 WARNING keystonemiddleware.auth_token [-] 
> Authorization failed for token
> 
> 
> Best Regards
> Chaoyi Huang ( Joe Huang )
> 
> 
> -----Original Message-----
> From: Hans Feldt [mailto:hans.feldt at ericsson.com]
> Sent: Tuesday, August 25, 2015 5:06 PM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [Keystone][Glance] keystonemiddleware & 
> multiple keystone endpoints
> 
> 
> 
> On 2015-08-25 09:37, Jamie Lennox wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Hans Feldt" <hans.feldt at ericsson.com>
> >> To: openstack-dev at lists.openstack.org
> >> Sent: Thursday, August 20, 2015 10:40:28 PM
> >> Subject: [openstack-dev] [Keystone][Glance] keystonemiddleware & multiple
> >> 	keystone endpoints
> >>
> >> How do you configure/use keystonemiddleware for a specific identity 
> >> endpoint among several?
> >>
> >> In an OPNFV multi region prototype I have keystone endpoints per 
> >> region. I would like keystonemiddleware (in context of glance-api) 
> >> to use the local keystone for performing user token validation. 
> >> Instead keystonemiddleware seems to use the first listed keystone 
> >> endpoint in the service catalog (which could be wrong/non-optimal 
> >> in most regions).
> >>
> >> I found this closed, related bug:
> >> https://bugs.launchpad.net/python-keystoneclient/+bug/1147530
> >
> > Hey,
> >
> > There's two points to this.
> >
> > * If you are using an auth plugin then you're right it will just 
> > pick the first endpoint. You can look at project specific 
> > endpoints[1] so that there is only one keystone endpoint returned for the services project.
> > I've also just added a review for this feature[2].
> 
> I am not.
> 
> > * If you're not using an auth plugin (so the admin_X options) then 
> > keystone will always use the endpoint that is configured in the 
> > options (identity_uri).
> 
> Yes for getting its own admin/service token. But for later user token 
> validation it seems to pick the first identity service in the stored 
> (?) service catalog.
> 
> By patching keystonemiddleware, _create_identity_server and the call 
> to Adapter constructor with an endpoint_override parameter I can get 
> it to use the local keystone for token validation. I am looking for an 
> official way of achieving the same.
> 
> Thanks,
> Hans
> 
> >
> > Hope that helps,
> >
> > Jamie
> >
> >
> > [1]
> > https://github.com/openstack/keystone-specs/blob/master/specs/juno/e
> > nd point-group-filter.rst [2] 
> > https://review.openstack.org/#/c/216579
> >
> >> Thanks,
> >> Hans
> >>
> >> ___________________________________________________________________
> >> __ _____ OpenStack Development Mailing List (not for usage 
> >> questions)
> >> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> > ____________________________________________________________________
> > __ ____ OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> 
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev