[Openstack-operators] Authentication problems with cinder

Jay Pipes jaypipes at gmail.com
Fri May 3 15:58:57 UTC 2013


We saw this exact same error when deploying Keystone +
Cinder/Nova/Glance with PKI in Folsom.

I presume you are using Grizzly, since I see you are also using memcache
with PKI, which does not work in Folsom, AFAIK.

The "solution" to the problem for us was to simply issue a restart of
the cinder-api/nova-api-os-compute/glance-api services, and the service
user would then begin to work again. I believe it has something to do
with the service user not being able to retrieve the token revocation
list from the Keystone server after some time period. For us, it was
usually around 24 hours between requisite restarts.

I've cc'd Adam Donnison to have a look at this as well.

Best,
-jay

On 05/02/2013 03:01 PM, Juan José Pavlik Salles wrote:
> Hi guys, i don't want to be annoying but i'm still having this problem.
> I don't understand this (from /var/log/cinder/cinder-api.log):
> 
> 2013-04-30 20:00:42    DEBUG [keystoneclient.middleware.auth_token]
> Token validation failure.
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 688, in _validate_user_token
>     verified = self.verify_signed_token(user_token)
>   File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1043, in verify_signed_token
>     if self.is_signed_token_revoked(signed_text):
>   File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1007, in is_signed_token_revoked
>     revocation_list = self.token_revocation_list
>   File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1079, in token_revocation_list
>     self.token_revocation_list = self.fetch_revocation_list()
>   File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1109, in fetch_revocation_list
>     return self.cms_verify(data['signed'])
>   File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1038, in cms_verify
>     raise err
> CalledProcessError: Command 'openssl' returned non-zero exit status 4
> *2013-04-30 20:00:42    DEBUG [keystoneclient.middleware.auth_token]
> Marking token
> *MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzY
> ...
> i4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
> *as unauthorized in memcache*
> *2013-04-30 20:00:42  WARNING [keystoneclient.middleware.auth_token]
> Authorization failed for token*
> MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzYjc4NmFiMTUxYzhmOGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTcifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuY
> ...
> dmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MSIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjgwODAvdjEvQVVUSF82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJpZCI6ICI2NTkxMTExNGMzNjM0MWExOTAwNmMzMjhjNmQwYTJhZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MS9BVVRIXzZhYTNiZjFhYjY4MDQwMjE4ODczYTc4MmY5MGNmZmE3In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm9iamVjdC1zdG9yZSIsICJuYW1lIjogInN3aWZ0In0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6MzUzNTcvdjIuMCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjUwMDAvdjIuMCIsICJpZCI6ICIwZjkzODlkMDQ4NWU0ZjJmOWY3ODc0YzQxMTgxYmQyOCIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pb
iIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
> *2013-04-30 20:00:42     INFO [keystoneclient.middleware.auth_token]
> Invalid user token - rejecting request*
> *
> *
> It seems that cinder can't recognise my auth_token so it tries to ban
> it. Does anybody have any idea about this? Thanks!!!
> 
> 
> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
> <mailto:jjpavlik at gmail.com>>
> 
>     I ran tcpdump on my cinder node (172.19.136.245) and this is what i saw:
> 
>     From 172.19.136.10 i ran "cinder --os-username=admin
>     --os-tenant-name=admin --os-password=zGp05Nsa
>     --os-auth-url=http://172.19.136.1:35357/v2.0 list": 
> 
>     After getting a valid token from keystone.
> 
>     -----Request from cinder-client to cinder-api:
> 
>     GET /v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail HTTP/1.1
>     Host: 172.19.136.245:8776 <http://172.19.136.245:8776>
>     X-Auth-Project-Id: admin
>     Accept-Encoding: gzip, deflate, compress
>     Content-Length: 0
>     Accept: application/json
>     User-Agent: python-cinderclient
>     X-Auth-Token: MIIMbwYJKoZIhvcNAQcCoIIMY.....oiRM1nsw==
> 
>     -----Request from cinder-api to keystone:
> 
>     GET /v2.0/tokens/revoked HTTP/1.1
>     Host: 172.19.136.11:35357 <http://172.19.136.11:35357>
>     Accept-Encoding: identity
>     Content-type: application/json
>     Accept: application/json
>     X-Auth-Token:
>     MIIMKAYJKoZIhvcNAQcCoIIMGTCCDBUCAQExCTAHBgUrDgMCGjCCCwEGCS...eufVytyk=
> 
>     -----Answer from keystone to cinder-api:
> 
>     HTTP/1.1 200 OK
>     Vary: X-Auth-Token
>     Content-Type: application/json
>     Content-Length: 612
>     Date: Tue, 30 Apr 2013 19:55:04 GMT
> 
>     {"signed": "-----BEGIN
>     CMS-----\nMIIBkAYJKoZIhvcNAQcCoIIBgTCCAX0CAQExCTAHBgUrDgMCGjBrBgkqhkiG9w0B\nBwGgXgRceyJyZXZva2VkIjogW3siZXhwaXJlcyI6ICIyMDEzLTA0LTMwVDIwOjQy\nOjQ3WiIsICJpZCI6ICJhMDRhMjAwZGZlZTI2NjNkNDNjN2UyNzkzZTU3YWE1OCJ9\nXX0xgf8wgfwCAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVW5zZXQxDjAM\nBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEYMBYGA1UEAxMPd3d3LmV4YW1w\nbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIGAE4mgl+c2wGz0+71j\n5Am0KCI+lKHtYJppPtBvVDJ194J1hgMEMz7Yxlqtn1qMoJm3o5fCTl8pU3IszX/f\nb36zOZCrRXTCqgb32O7HfhPKT+N8kqZxMvtDTzv+3uQOC0xw7cAh+sNPgG1EHrL3\nIO8cMEUJqOkXjhwQPKXSqYVrwg4=\n-----END
>     CMS-----\n"}
> 
> 
>     -----Answer from cinder-api to cinder-client:
> 
>     HTTP/1.1 401 Unauthorized
>     Www-Authenticate: Keystone uri='http://172.19.136.11:35357'
>     Content-Length: 276
>     Content-Type: text/plain; charset=UTF-8
>     Date: Tue, 30 Apr 2013 19:55:04 GMT
> 
>     401 Unauthorized
> 
>     This server could not verify that you are authorized to access the
>     document you requested. Either you supplied the wrong credentials
>     (e.g., bad password), or your browser does not understand how to
>     supply the credentials required.
> 
>      Authentication required  
> 
> 
>     Is there any chance that cinder-api is breaking up my token?? 
> 
> 
> 
>     2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
>     <mailto:jjpavlik at gmail.com>>
> 
>         I can get valid credentials with this line:
> 
>         root at heladera:/etc/cinder# cinder --os-username=admin
>         --os-tenant-name=admin --os-password=XXX
>         --os-auth-url=http://172.19.136.1:35357/v2.0 credentials
>         +------------------+----------------------------------------------------------------------------------------+
>         | User Credentials |                                        
>         Value                                          |
>         +------------------+----------------------------------------------------------------------------------------+
>         |        id        |                          
>          3f82673b5fe0411ab5fd8216bdb693c6                            |
>         |       name       |                                        
>         admin                                          |
>         |      roles       | [{u'name': u'KeystoneServiceAdmin'},
>         {u'name': u'KeystoneAdmin'}, {u'name': u'admin'}] |
>         |   roles_links    |                                          
>         []                                           |
>         |     username     |                                        
>         admin                                          |
>         +------------------+----------------------------------------------------------------------------------------+
>         +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>         |   Token   |                                                  
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                  Value                                  
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                   |
>         +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>         |  expires  |                                                  
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                           2013-05-01T18:47:48Z                          
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                   |
>         |     id    |
>         MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQEx...tcWW6xvpLgWsr3A== |
>         | issued_at |                                                  
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                        2013-04-30T18:47:48.512440                      
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                    |
>         |   tenant  |                                                  
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                 {u'id':
>         u'6aa3bf1ab68040218873a782f90cffa7', u'enabled': True,
>         u'description': None, u'name': u'admin'}                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                        
>                                                                            |
>         +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> 
>         So, it must be something that happens AFTER getting the
>         credentials, something involving the cinder api. I'm not sure
>         how the authentication process work but this is what i think:
> 
>         1-cinder client request for an auth token
>         2-keystone validates the credentials, creates the token and
>         sends it back to the client
>         3-the cinder client uses the received token to connect against
>         the cinder api
>         4-the cinder api validates the token against ¿keystone? Here is
>         where the problem might be.
>         5-somehow the api can't validate the token and rejects me.
> 
>         I'm running out of ideas.
> 
> 
> 
>         2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
>         <mailto:jjpavlik at gmail.com>>
> 
>             When i try to list the volumes this is what i see in the
>             cinder api logs file:
> 
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Authenticating user token
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Removing headers from
>             request environment:
>             X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
>             2013-04-30 17:43:07    ERROR [keystoneclient.common.cms]
>             Verify error: Verification failure
> 
>             140606277047968:error:0407006A:rsa
>             routines:RSA_padding_check_PKCS1_type_1:block type is not
>             01:rsa_pk1.c:100:
>             140606277047968:error:04067072:rsa
>             routines:RSA_EAY_PUBLIC_DECRYPT:padding check
>             failed:rsa_eay.c:721:
>             140606277047968:error:2E09A09E:CMS
>             routines:CMS_SignerInfo_verify_content:verification
>             failure:cms_sd.c:900:
>             140606277047968:error:2E09D06D:CMS
>             routines:CMS_verify:content verify error:cms_smime.c:425:
> 
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Token validation failure.
>             Traceback (most recent call last):
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 688, in _validate_user_token
>                 verified = self.verify_signed_token(user_token)
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1043, in verify_signed_token
>                 if self.is_signed_token_revoked(signed_text):
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1007, in is_signed_token_revoked
>                 revocation_list = self.token_revocation_list
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1079, in token_revocation_list
>                 self.token_revocation_list = self.fetch_revocation_list()
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1109, in fetch_revocation_list
>                 return self.cms_verify(data['signed'])
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1038, in cms_verify
>                 raise err
>             CalledProcessError: Command 'openssl' returned non-zero exit
>             status 4
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Marking token
>             MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw== as unauthorized in memcache
>             2013-04-30 17:43:07  WARNING
>             [keystoneclient.middleware.auth_token] Authorization failed
>             for token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw==
>             2013-04-30 17:43:07     INFO
>             [keystoneclient.middleware.auth_token] Invalid user token -
>             rejecting request
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Authenticating user token
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Removing headers from
>             request environment:
>             X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
>             2013-04-30 17:43:07    ERROR [keystoneclient.common.cms]
>             Verify error: Verification failure
> 
>             140558031275680:error:0407006A:rsa
>             routines:RSA_padding_check_PKCS1_type_1:block type is not
>             01:rsa_pk1.c:100:
>             140558031275680:error:04067072:rsa
>             routines:RSA_EAY_PUBLIC_DECRYPT:padding check
>             failed:rsa_eay.c:721:
>             140558031275680:error:2E09A09E:CMS
>             routines:CMS_SignerInfo_verify_content:verification
>             failure:cms_sd.c:900:
>             140558031275680:error:2E09D06D:CMS
>             routines:CMS_verify:content verify error:cms_smime.c:425:
> 
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Token validation failure.
>             Traceback (most recent call last):
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 688, in _validate_user_token
>                 verified = self.verify_signed_token(user_token)
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1043, in verify_signed_token
>                 if self.is_signed_token_revoked(signed_text):
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1007, in is_signed_token_revoked
>                 revocation_list = self.token_revocation_list
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1079, in token_revocation_list
>                 self.token_revocation_list = self.fetch_revocation_list()
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1109, in fetch_revocation_list
>                 return self.cms_verify(data['signed'])
>               File
>             "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>             line 1038, in cms_verify
>                 raise err
>             CalledProcessError: Command 'openssl' returned non-zero exit
>             status 4
>             2013-04-30 17:43:07    DEBUG
>             [keystoneclient.middleware.auth_token] Marking token
>             MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw== as unauthorized in memcache
>             2013-04-30 17:43:07  WARNING
>             [keystoneclient.middleware.auth_token] Authorization failed
>             for token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw==
>             2013-04-30 17:43:07     INFO
>             [keystoneclient.middleware.auth_token] Invalid user token -
>             rejecting request
> 
>             MAYBE... somehow HAproxy is changing something in the header
>             but i don't think so. This is the haproxy configuration for
>             the cinder API:
> 
>             listen nova-api-cinder 172.19.136.1:8776
>             <http://172.19.136.1:8776>
>                     balance  roundrobin
>                     option  tcplog
>                     server  heladera 172.19.136.245:8776
>             <http://172.19.136.245:8776>  check
> 
>             I don't understand why is the Verification Failure, and why
>             i have openssl involve in my authentication, I didn't change
>             anything in the cinder api-paste.ini file, besides the
>             auth_host and service_host. 
> 
> 
>             2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
>             <mailto:jjpavlik at gmail.com>>
> 
>                 Hi Jay, you are right, i'm trying to balance API calls
>                 with HAProxy. I installed HAproxy on 172.19.136.1 and
>                 configured all the openstack services to make the calls
>                 to that IP, then i use HAproxy to redirect the API calls
>                 to the real API servers (172.19.136.10 and
>                 172.19.136.11), this is my configuration:
> 
>                 I've these 4 nodes: 
> 
>                 172.19.136.245 <http://172.19.136.245>:
>                 -Cinder
> 
>                 172.19.136.10 <http://172.19.136.10>:
>                 -Keystone
>                 -Glance (glance, api, registry)
>                 -Nova (compute, scheduler, etc)
> 
>                 172.19.136.11 <http://172.19.136.11>:
>                 -Keystone
>                 -Glance (glance, api, registry)
>                 -Nova (compute, scheduler, etc)
> 
>                 172.19.136.2 / 172.19.136.1 <http://172.19.136.1>:
>                 -Quantum server
>                 -RabbitMQ
>                 -MySQL
>                 -HAProxy (Listening on 172.19.136.1 for all the API
>                 calls, and balancing them to either 172.19.136.10 or
>                 172.19.136.11, it also listens for cinder api calls and
>                 redirects them to 172.19.136.245)
> 
>                 I didn't change all the endpoints yet, but all of them
>                 should redirect to 172.19.136.1, maybe that's the
>                 problem. What do you think? 
> 
>                 This configuration might look odd or strange, but i'm
>                 trying to build a redundant and scalable cloud (like in
>                 this article
>                 http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/).
>                 Thanks!!!
> 
> 
>                 2013/4/30 Jay Pipes <jaypipes at gmail.com
>                 <mailto:jaypipes at gmail.com>>
> 
>                     On 04/29/2013 04:56 PM, Juan José Pavlik Salles wrote:
>                     > Hi, i have spent the last days trying to solve
>                     this problem. I can't
>                     > list my cinder volumes from my shell:
>                     >
>                     > root at locro:~# cinder --os-username=admin
>                     --os-tenant-name=admin
>                     > --os-password=XXX
>                     --os-auth-url=http://172.19.136.1:35357/v2.0 --debug
>                     list
>                     >
>                     > REQ: curl -i http://172.19.136.1:35357/v2.0/tokens
>                     -X POST -H
>                     > "Content-Type: application/json" -H "Accept:
>                     application/json" -H
>                     > "User-Agent: python-cinderclient" -d '{"auth":
>                     {"tenantName": "admin",
>                     > "passwordCredentials": {"username": "admin",
>                     "password": "zGp05Nsa"}}}'
>                     >
>                     > RESP: [200] {'date': 'Mon, 29 Apr 2013 17:24:44
>                     GMT', 'content-type':
>                     > 'application/json', 'content-length': '7096',
>                     'vary': 'X-Auth-Token'}
>                     > RESP BODY: {"access": {"token": {"issued_at":
>                     > "2013-04-29T17:24:44.044013", "expires":
>                     "2013-04-30T17:24:43Z", "id":
>                     > "MIIMaQYJKoZIhvcNAQcC...", "tenant":
>                     {"description": null, "enabled":
>                     > true, "id": "6aa3bf1ab68040218873a782f90cffa7",
>                     "name": "admin"}},
>                     > "serviceCatalog": [{"endpoints": [{"adminURL":
>                     >
>                     "http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
>                     > "region": "RegionOne", "internalURL":
>                     >
>                     "http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
>                     "id":
>                     > "26178391275a42cfa3b786ab151c8f8a", "publicURL":
>                     >
>                     "http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7"}],
>                     > "endpoints_links": [], "type": "compute", "name":
>                     "nova"}, {"endpoints":
>                     > [{"adminURL": "http://172.19.136.11:9696/",
>                     "region": "RegionOne",
>                     > "internalURL": "http://172.19.136.11:9696/", "id":
>                     > "1d0f394d83804ecaaa5ba708ccf0417b", "publicURL":
>                     > "http://172.19.136.11:9696/"}], "endpoints_links":
>                     [], "type":
>                     > "network", "name": "quantum"}, {"endpoints":
>                     [{"adminURL":
>                     > "http://172.19.136.10:9292/v2", "region":
>                     "RegionOne", "internalURL":
>                     > "http://172.19.136.11:9292/v2", "id":
>                     > "11f37a313bad47f28b846cb9b94d458c", "publicURL":
>                     > "http://172.19.136.11:9292/v2"}],
>                     "endpoints_links": [], "type":
>                     > "image", "name": "glance"}, {"endpoints":
>                     [{"adminURL":
>                     >
>                     "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
>                     > "region": "RegionOne", "internalURL":
>                     >
>                     "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
>                     "id":
>                     > "1ebe70478edd45d087263a4dc457f03a", "publicURL":
>                     >
>                     "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7"}],
>                     > "endpoints_links": [], "type": "volume", "name":
>                     "cinder"},
>                     > {"endpoints": [{"adminURL":
>                     "http://172.19.136.11:8773/services/Admin",
>                     > "region": "RegionOne", "internalURL":
>                     > "http://172.19.136.10:8773/services/Cloud", "id":
>                     > "4fd5bcbee3584c2b883b08f22f81de54", "publicURL":
>                     > "http://172.19.136.10:8773/services/Cloud"}],
>                     "endpoints_links": [],
>                     > "type": "ec2", "name": "ec2"}, {"endpoints":
>                     [{"adminURL":
>                     > "http://172.19.136.10:8080/v1", "region":
>                     "RegionOne", "internalURL":
>                     >
>                     "http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7",
>                     > "id": "65911114c36341a19006c328c6d0a2ae", "publicURL":
>                     >
>                     "http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7"}],
>                     > "endpoints_links": [], "type": "object-store",
>                     "name": "swift"},
>                     > {"endpoints": [{"adminURL":
>                     "http://172.19.136.11:35357/v2.0", "region":
>                     > "RegionOne", "internalURL":
>                     "http://172.19.136.10:5000/v2.0", "id":
>                     > "0f9389d0485e4f2f9f7874c41181bd28", "publicURL":
>                     > "http://172.19.136.10:5000/v2.0"}],
>                     "endpoints_links": [], "type":
>                     > "identity", "name": "keystone"}], "user":
>                     {"username": "admin",
>                     > "roles_links": [], "id":
>                     "3f82673b5fe0411ab5fd8216bdb693c6", "roles":
>                     > [{"name": "KeystoneServiceAdmin"}, {"name":
>                     "KeystoneAdmin"}, {"name":
>                     > "admin"}], "name": "admin"}, "metadata":
>                     {"is_admin": 0, "roles":
>                     > ["6666fa99078a4f07a070e7e858c32f02",
>                     "36bba9ef0178448c8a654b75feb3a0f4",
>                     > "a25581dd3470460b91ecaa29eca7205c"]}}}
>                     >
>                     > REQ: curl -i
>                     >
>                     http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail
>                     > -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent:
>                     > python-cinderclient" -H "Accept: application/json"
>                     -H "X-Auth-Token:
>                     > MIIMaQYJKoZIhvcNAQcCo..."
>                     >
>                     > RESP: [401] {'date': 'Mon, 29 Apr 2013 17:24:44
>                     GMT', 'content-length':
>                     > '276', 'content-type': 'text/plain;
>                     charset=UTF-8', 'www-authenticate':
>                     > "Keystone uri='http://172.19.136.1:35357'"}
>                     > RESP BODY: 401 Unauthorized
> 
>                     From the above, the authentication URI that you are
>                     supplying to
>                     cinderclient is http://172.19.136.1:35357, which is
>                     not the same as what
>                     is returned in the service catalog above, which has
>                     the internalURL for
>                     the identity endpoint as http://172.19.136.10:5000/v2.0.
> 
>                     Is this intended?
> 
>                     -jay
> 
> 
>                     _______________________________________________
>                     OpenStack-operators mailing list
>                     OpenStack-operators at lists.openstack.org
>                     <mailto:OpenStack-operators at lists.openstack.org>
>                     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> 
> 
> 
>                 -- 
>                 Pavlik Juan José
> 
> 
> 
> 
>             -- 
>             Pavlik Juan José
> 
> 
> 
> 
>         -- 
>         Pavlik Juan José
> 
> 
> 
> 
>     -- 
>     Pavlik Juan José
> 
> 
> 
> 
> -- 
> Pavlik Juan José
> 
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 



More information about the OpenStack-operators mailing list