[Openstack-operators] Authentication problems with cinder
Juan José Pavlik Salles
jjpavlik at gmail.com
Thu May 2 19:01:37 UTC 2013
Hi guys, i don't want to be annoying but i'm still having this problem. I
don't understand this (from /var/log/cinder/cinder-api.log):
2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token] Token
validation failure.
Traceback (most recent call last):
File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 688, in _validate_user_token
verified = self.verify_signed_token(user_token)
File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1043, in verify_signed_token
if self.is_signed_token_revoked(signed_text):
File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1007, in is_signed_token_revoked
revocation_list = self.token_revocation_list
File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1079, in token_revocation_list
self.token_revocation_list = self.fetch_revocation_list()
File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1109, in fetch_revocation_list
return self.cms_verify(data['signed'])
File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1038, in cms_verify
raise err
CalledProcessError: Command 'openssl' returned non-zero exit status 4
*2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
Marking token *MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzY
...
i4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
*as unauthorized in memcache*
*2013-04-30 20:00:42 WARNING [keystoneclient.middleware.auth_token]
Authorization failed for
token*MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzYjc4NmFiMTUxYzhmOGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTcifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuY
...
dmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MSIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjgwODAvdjEvQVVUSF82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJpZCI6ICI2NTkxMTExNGMzNjM0MWExOTAwNmMzMjhjNmQwYTJhZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MS9BVVRIXzZhYTNiZjFhYjY4MDQwMjE4ODczYTc4MmY5MGNmZmE3In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm9iamVjdC1zdG9yZSIsICJuYW1lIjogInN3aWZ0In0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6MzUzNTcvdjIuMCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjUwMDAvdjIuMCIsICJpZCI6ICIwZjkzODlkMDQ4NWU0ZjJmOWY3ODc0YzQxMTgxYmQyOCIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
*2013-04-30 20:00:42 INFO [keystoneclient.middleware.auth_token]
Invalid user token - rejecting request*
*
*
It seems that cinder can't recognise my auth_token so it tries to ban it.
Does anybody have any idea about this? Thanks!!!
2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com>
> I ran tcpdump on my cinder node (172.19.136.245) and this is what i saw:
>
> From 172.19.136.10 i ran "cinder --os-username=admin
> --os-tenant-name=admin --os-password=zGp05Nsa --os-auth-url=
> http://172.19.136.1:35357/v2.0 list":
>
> After getting a valid token from keystone.
>
> -----Request from cinder-client to cinder-api:
>
> GET /v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail HTTP/1.1
> Host: 172.19.136.245:8776
> X-Auth-Project-Id: admin
> Accept-Encoding: gzip, deflate, compress
> Content-Length: 0
> Accept: application/json
> User-Agent: python-cinderclient
> X-Auth-Token: MIIMbwYJKoZIhvcNAQcCoIIMY.....oiRM1nsw==
>
> -----Request from cinder-api to keystone:
>
> GET /v2.0/tokens/revoked HTTP/1.1
> Host: 172.19.136.11:35357
> Accept-Encoding: identity
> Content-type: application/json
> Accept: application/json
> X-Auth-Token:
> MIIMKAYJKoZIhvcNAQcCoIIMGTCCDBUCAQExCTAHBgUrDgMCGjCCCwEGCS...eufVytyk=
>
> -----Answer from keystone to cinder-api:
>
> HTTP/1.1 200 OK
> Vary: X-Auth-Token
> Content-Type: application/json
> Content-Length: 612
> Date: Tue, 30 Apr 2013 19:55:04 GMT
>
> {"signed": "-----BEGIN
> CMS-----\nMIIBkAYJKoZIhvcNAQcCoIIBgTCCAX0CAQExCTAHBgUrDgMCGjBrBgkqhkiG9w0B\nBwGgXgRceyJyZXZva2VkIjogW3siZXhwaXJlcyI6ICIyMDEzLTA0LTMwVDIwOjQy\nOjQ3WiIsICJpZCI6ICJhMDRhMjAwZGZlZTI2NjNkNDNjN2UyNzkzZTU3YWE1OCJ9\nXX0xgf8wgfwCAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVW5zZXQxDjAM\nBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEYMBYGA1UEAxMPd3d3LmV4YW1w\nbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIGAE4mgl+c2wGz0+71j\n5Am0KCI+lKHtYJppPtBvVDJ194J1hgMEMz7Yxlqtn1qMoJm3o5fCTl8pU3IszX/f\nb36zOZCrRXTCqgb32O7HfhPKT+N8kqZxMvtDTzv+3uQOC0xw7cAh+sNPgG1EHrL3\nIO8cMEUJqOkXjhwQPKXSqYVrwg4=\n-----END
> CMS-----\n"}
>
>
> -----Answer from cinder-api to cinder-client:
>
> HTTP/1.1 401 Unauthorized
> Www-Authenticate: Keystone uri='http://172.19.136.11:35357'
> Content-Length: 276
> Content-Type: text/plain; charset=UTF-8
> Date: Tue, 30 Apr 2013 19:55:04 GMT
>
> 401 Unauthorized
>
> This server could not verify that you are authorized to access the
> document you requested. Either you supplied the wrong credentials (e.g.,
> bad password), or your browser does not understand how to supply the
> credentials required.
>
> Authentication required
>
>
> Is there any chance that cinder-api is breaking up my token??
>
>
>
> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com>
>
>> I can get valid credentials with this line:
>>
>> root at heladera:/etc/cinder# cinder --os-username=admin
>> --os-tenant-name=admin --os-password=XXX --os-auth-url=
>> http://172.19.136.1:35357/v2.0 credentials
>>
>> +------------------+----------------------------------------------------------------------------------------+
>> | User Credentials | Value
>> |
>>
>> +------------------+----------------------------------------------------------------------------------------+
>> | id |
>> 3f82673b5fe0411ab5fd8216bdb693c6 |
>> | name | admin
>> |
>> | roles | [{u'name': u'KeystoneServiceAdmin'}, {u'name':
>> u'KeystoneAdmin'}, {u'name': u'admin'}] |
>> | roles_links | []
>> |
>> | username | admin
>> |
>>
>> +------------------+----------------------------------------------------------------------------------------+
>>
>> +-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>> | Token |
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Value
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> |
>>
>> +-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>> | expires |
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013-05-01T18:47:48Z
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> |
>> | id | MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQEx...tcWW6xvpLgWsr3A== |
>> | issued_at |
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013-04-30T18:47:48.512440
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> |
>> | tenant |
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> {u'id':
>> u'6aa3bf1ab68040218873a782f90cffa7', u'enabled': True, u'description':
>> None, u'name': u'admin'}
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> |
>>
>> +-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>>
>> So, it must be something that happens AFTER getting the credentials,
>> something involving the cinder api. I'm not sure how the authentication
>> process work but this is what i think:
>>
>> 1-cinder client request for an auth token
>> 2-keystone validates the credentials, creates the token and sends it back
>> to the client
>> 3-the cinder client uses the received token to connect against the cinder
>> api
>> 4-the cinder api validates the token against ¿keystone? Here is where the
>> problem might be.
>> 5-somehow the api can't validate the token and rejects me.
>>
>> I'm running out of ideas.
>>
>>
>>
>> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com>
>>
>>> When i try to list the volumes this is what i see in the cinder api logs
>>> file:
>>>
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Authenticating user token
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Removing headers from request environment:
>>> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
>>> 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms] Verify error:
>>> Verification failure
>>>
>>> 140606277047968:error:0407006A:rsa
>>> routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
>>> 140606277047968:error:04067072:rsa
>>> routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:
>>> 140606277047968:error:2E09A09E:CMS
>>> routines:CMS_SignerInfo_verify_content:verification failure:cms_sd.c:900:
>>> 140606277047968:error:2E09D06D:CMS routines:CMS_verify:content verify
>>> error:cms_smime.c:425:
>>>
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Token validation failure.
>>> Traceback (most recent call last):
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 688, in _validate_user_token
>>> verified = self.verify_signed_token(user_token)
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1043, in verify_signed_token
>>> if self.is_signed_token_revoked(signed_text):
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1007, in is_signed_token_revoked
>>> revocation_list = self.token_revocation_list
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1079, in token_revocation_list
>>> self.token_revocation_list = self.fetch_revocation_list()
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1109, in fetch_revocation_list
>>> return self.cms_verify(data['signed'])
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1038, in cms_verify
>>> raise err
>>> CalledProcessError: Command 'openssl' returned non-zero exit status 4
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Marking token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw== as unauthorized in memcache
>>> 2013-04-30 17:43:07 WARNING [keystoneclient.middleware.auth_token]
>>> Authorization failed for token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw==
>>> 2013-04-30 17:43:07 INFO [keystoneclient.middleware.auth_token]
>>> Invalid user token - rejecting request
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Authenticating user token
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Removing headers from request environment:
>>> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
>>> 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms] Verify error:
>>> Verification failure
>>>
>>> 140558031275680:error:0407006A:rsa
>>> routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
>>> 140558031275680:error:04067072:rsa
>>> routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:
>>> 140558031275680:error:2E09A09E:CMS
>>> routines:CMS_SignerInfo_verify_content:verification failure:cms_sd.c:900:
>>> 140558031275680:error:2E09D06D:CMS routines:CMS_verify:content verify
>>> error:cms_smime.c:425:
>>>
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Token validation failure.
>>> Traceback (most recent call last):
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 688, in _validate_user_token
>>> verified = self.verify_signed_token(user_token)
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1043, in verify_signed_token
>>> if self.is_signed_token_revoked(signed_text):
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1007, in is_signed_token_revoked
>>> revocation_list = self.token_revocation_list
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1079, in token_revocation_list
>>> self.token_revocation_list = self.fetch_revocation_list()
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1109, in fetch_revocation_list
>>> return self.cms_verify(data['signed'])
>>> File
>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>> line 1038, in cms_verify
>>> raise err
>>> CalledProcessError: Command 'openssl' returned non-zero exit status 4
>>> 2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token]
>>> Marking token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw== as unauthorized in
>>> memcache
>>> 2013-04-30 17:43:07 WARNING [keystoneclient.middleware.auth_token]
>>> Authorization failed for token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw==
>>> 2013-04-30 17:43:07 INFO [keystoneclient.middleware.auth_token]
>>> Invalid user token - rejecting request
>>>
>>> MAYBE... somehow HAproxy is changing something in the header but i don't
>>> think so. This is the haproxy configuration for the cinder API:
>>>
>>> listen nova-api-cinder 172.19.136.1:8776
>>> balance roundrobin
>>> option tcplog
>>> server heladera 172.19.136.245:8776 check
>>>
>>> I don't understand why is the Verification Failure, and why i have
>>> openssl involve in my authentication, I didn't change anything in the
>>> cinder api-paste.ini file, besides the auth_host and service_host.
>>>
>>>
>>> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com>
>>>
>>>> Hi Jay, you are right, i'm trying to balance API calls with HAProxy. I
>>>> installed HAproxy on 172.19.136.1 and configured all the openstack services
>>>> to make the calls to that IP, then i use HAproxy to redirect the API calls
>>>> to the real API servers (172.19.136.10 and 172.19.136.11), this is my
>>>> configuration:
>>>>
>>>> I've these 4 nodes:
>>>>
>>>> 172.19.136.245:
>>>> -Cinder
>>>>
>>>> 172.19.136.10:
>>>> -Keystone
>>>> -Glance (glance, api, registry)
>>>> -Nova (compute, scheduler, etc)
>>>>
>>>> 172.19.136.11:
>>>> -Keystone
>>>> -Glance (glance, api, registry)
>>>> -Nova (compute, scheduler, etc)
>>>>
>>>> 172.19.136.2 / 172.19.136.1:
>>>> -Quantum server
>>>> -RabbitMQ
>>>> -MySQL
>>>> -HAProxy (Listening on 172.19.136.1 for all the API calls, and
>>>> balancing them to either 172.19.136.10 or 172.19.136.11, it also listens
>>>> for cinder api calls and redirects them to 172.19.136.245)
>>>>
>>>> I didn't change all the endpoints yet, but all of them should redirect
>>>> to 172.19.136.1, maybe that's the problem. What do you think?
>>>>
>>>> This configuration might look odd or strange, but i'm trying to build a
>>>> redundant and scalable cloud (like in this article
>>>> http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/).
>>>> Thanks!!!
>>>>
>>>>
>>>> 2013/4/30 Jay Pipes <jaypipes at gmail.com>
>>>>
>>>>> On 04/29/2013 04:56 PM, Juan José Pavlik Salles wrote:
>>>>> > Hi, i have spent the last days trying to solve this problem. I can't
>>>>> > list my cinder volumes from my shell:
>>>>> >
>>>>> > root at locro:~# cinder --os-username=admin --os-tenant-name=admin
>>>>> > --os-password=XXX --os-auth-url=http://172.19.136.1:35357/v2.0--debug list
>>>>> >
>>>>> > REQ: curl -i http://172.19.136.1:35357/v2.0/tokens -X POST -H
>>>>> > "Content-Type: application/json" -H "Accept: application/json" -H
>>>>> > "User-Agent: python-cinderclient" -d '{"auth": {"tenantName":
>>>>> "admin",
>>>>> > "passwordCredentials": {"username": "admin", "password":
>>>>> "zGp05Nsa"}}}'
>>>>> >
>>>>> > RESP: [200] {'date': 'Mon, 29 Apr 2013 17:24:44 GMT', 'content-type':
>>>>> > 'application/json', 'content-length': '7096', 'vary': 'X-Auth-Token'}
>>>>> > RESP BODY: {"access": {"token": {"issued_at":
>>>>> > "2013-04-29T17:24:44.044013", "expires": "2013-04-30T17:24:43Z",
>>>>> "id":
>>>>> > "MIIMaQYJKoZIhvcNAQcC...", "tenant": {"description": null, "enabled":
>>>>> > true, "id": "6aa3bf1ab68040218873a782f90cffa7", "name": "admin"}},
>>>>> > "serviceCatalog": [{"endpoints": [{"adminURL":
>>>>> > "http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
>>>>> > "region": "RegionOne", "internalURL":
>>>>> > "http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
>>>>> "id":
>>>>> > "26178391275a42cfa3b786ab151c8f8a", "publicURL":
>>>>> > "http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7"}],
>>>>> > "endpoints_links": [], "type": "compute", "name": "nova"},
>>>>> {"endpoints":
>>>>> > [{"adminURL": "http://172.19.136.11:9696/", "region": "RegionOne",
>>>>> > "internalURL": "http://172.19.136.11:9696/", "id":
>>>>> > "1d0f394d83804ecaaa5ba708ccf0417b", "publicURL":
>>>>> > "http://172.19.136.11:9696/"}], "endpoints_links": [], "type":
>>>>> > "network", "name": "quantum"}, {"endpoints": [{"adminURL":
>>>>> > "http://172.19.136.10:9292/v2", "region": "RegionOne",
>>>>> "internalURL":
>>>>> > "http://172.19.136.11:9292/v2", "id":
>>>>> > "11f37a313bad47f28b846cb9b94d458c", "publicURL":
>>>>> > "http://172.19.136.11:9292/v2"}], "endpoints_links": [], "type":
>>>>> > "image", "name": "glance"}, {"endpoints": [{"adminURL":
>>>>> > "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
>>>>> > "region": "RegionOne", "internalURL":
>>>>> > "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
>>>>> "id":
>>>>> > "1ebe70478edd45d087263a4dc457f03a", "publicURL":
>>>>> > "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7"}],
>>>>> > "endpoints_links": [], "type": "volume", "name": "cinder"},
>>>>> > {"endpoints": [{"adminURL": "
>>>>> http://172.19.136.11:8773/services/Admin",
>>>>> > "region": "RegionOne", "internalURL":
>>>>> > "http://172.19.136.10:8773/services/Cloud", "id":
>>>>> > "4fd5bcbee3584c2b883b08f22f81de54", "publicURL":
>>>>> > "http://172.19.136.10:8773/services/Cloud"}], "endpoints_links": [],
>>>>> > "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL":
>>>>> > "http://172.19.136.10:8080/v1", "region": "RegionOne",
>>>>> "internalURL":
>>>>> > "http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7
>>>>> ",
>>>>> > "id": "65911114c36341a19006c328c6d0a2ae", "publicURL":
>>>>> > "http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7
>>>>> "}],
>>>>> > "endpoints_links": [], "type": "object-store", "name": "swift"},
>>>>> > {"endpoints": [{"adminURL": "http://172.19.136.11:35357/v2.0",
>>>>> "region":
>>>>> > "RegionOne", "internalURL": "http://172.19.136.10:5000/v2.0", "id":
>>>>> > "0f9389d0485e4f2f9f7874c41181bd28", "publicURL":
>>>>> > "http://172.19.136.10:5000/v2.0"}], "endpoints_links": [], "type":
>>>>> > "identity", "name": "keystone"}], "user": {"username": "admin",
>>>>> > "roles_links": [], "id": "3f82673b5fe0411ab5fd8216bdb693c6", "roles":
>>>>> > [{"name": "KeystoneServiceAdmin"}, {"name": "KeystoneAdmin"},
>>>>> {"name":
>>>>> > "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles":
>>>>> > ["6666fa99078a4f07a070e7e858c32f02",
>>>>> "36bba9ef0178448c8a654b75feb3a0f4",
>>>>> > "a25581dd3470460b91ecaa29eca7205c"]}}}
>>>>> >
>>>>> > REQ: curl -i
>>>>> >
>>>>> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail
>>>>> > -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent:
>>>>> > python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token:
>>>>> > MIIMaQYJKoZIhvcNAQcCo..."
>>>>> >
>>>>> > RESP: [401] {'date': 'Mon, 29 Apr 2013 17:24:44 GMT',
>>>>> 'content-length':
>>>>> > '276', 'content-type': 'text/plain; charset=UTF-8',
>>>>> 'www-authenticate':
>>>>> > "Keystone uri='http://172.19.136.1:35357'"}
>>>>> > RESP BODY: 401 Unauthorized
>>>>>
>>>>> From the above, the authentication URI that you are supplying to
>>>>> cinderclient is http://172.19.136.1:35357, which is not the same as
>>>>> what
>>>>> is returned in the service catalog above, which has the internalURL for
>>>>> the identity endpoint as http://172.19.136.10:5000/v2.0.
>>>>>
>>>>> Is this intended?
>>>>>
>>>>> -jay
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-operators mailing list
>>>>> OpenStack-operators at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pavlik Juan José
>>>>
>>>
>>>
>>>
>>> --
>>> Pavlik Juan José
>>>
>>
>>
>>
>> --
>> Pavlik Juan José
>>
>
>
>
> --
> Pavlik Juan José
>
--
Pavlik Juan José
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130502/5ae44ac5/attachment.html>
More information about the OpenStack-operators
mailing list