<div dir="ltr">Hi guys, i don't want to be annoying but i'm still having this problem. I don't understand this (from /var/log/cinder/cinder-api.log):<div><br><div><div>2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token] Token validation failure.</div>
<div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 688, in _validate_user_token</div><div> verified = self.verify_signed_token(user_token)</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1043, in verify_signed_token</div><div> if self.is_signed_token_revoked(signed_text):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1007, in is_signed_token_revoked</div>
<div> revocation_list = self.token_revocation_list</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1079, in token_revocation_list</div><div> self.token_revocation_list = self.fetch_revocation_list()</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1109, in fetch_revocation_list</div><div> return self.cms_verify(data['signed'])</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1038, in cms_verify</div>
<div> raise err</div><div>CalledProcessError: Command 'openssl' returned non-zero exit status 4</div></div><div><div><b>2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token] Marking token </b>MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzY ... i4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg== <b>as unauthorized in memcache</b></div>
<div><b>2013-04-30 20:00:42 WARNING [keystoneclient.middleware.auth_token] Authorization failed for token</b> MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzYjc4NmFiMTUxYzhmOGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTcifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuY ... dmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MSIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjgwODAvdjEvQVVUSF82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJpZCI6ICI2NTkxMTExNGMzNjM0MWExOTAwNmMzMjhjNmQwYTJhZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MS9BVVRIXzZhYTNiZjFhYjY4MDQwMjE4ODczYTc4MmY5MGNmZmE3In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm9iamVjdC1zdG9yZSIsICJuYW1lIjogInN3aWZ0In0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6MzUzNTcvdjIuMCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjUwMDAvdjIuMCIsICJpZCI6ICIwZjkzODlkMDQ4NWU0ZjJmOWY3ODc0YzQxMTgxYmQyOCIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==</div>
<div><b>2013-04-30 20:00:42 INFO [keystoneclient.middleware.auth_token] Invalid user token - rejecting request</b></div></div></div><div><b><br></b></div><div style>It seems that cinder can't recognise my auth_token so it tries to ban it. Does anybody have any idea about this? Thanks!!!</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/30 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I ran tcpdump on my cinder node (172.19.136.245) and this is what i saw:<div><br></div><div>From 172.19.136.10 i ran "cinder --os-username=admin --os-tenant-name=admin --os-password=zGp05Nsa --os-auth-url=<a href="http://172.19.136.1:35357/v2.0" target="_blank">http://172.19.136.1:35357/v2.0</a> list": </div>
<div><br></div><div>After getting a valid token from keystone.</div><div><br></div><div>-----Request from cinder-client to cinder-api:</div><div><br></div><div><div>GET /v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail HTTP/1.1</div>
<div>Host: <a href="http://172.19.136.245:8776" target="_blank">172.19.136.245:8776</a></div>
<div>X-Auth-Project-Id: admin</div><div>Accept-Encoding: gzip, deflate, compress</div><div>Content-Length: 0</div><div>Accept: application/json</div><div>User-Agent: python-cinderclient</div><div>X-Auth-Token: MIIMbwYJKoZIhvcNAQcCoIIMY.....oiRM1nsw==</div>
<div><br></div><div>-----Request from cinder-api to keystone:<br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">GET /v2.0/tokens/revoked HTTP/1.1</div><div class="gmail_extra">
Host: <a href="http://172.19.136.11:35357" target="_blank">172.19.136.11:35357</a></div><div class="gmail_extra">Accept-Encoding: identity</div><div class="gmail_extra">Content-type: application/json</div><div class="gmail_extra">
Accept: application/json</div>
<div class="gmail_extra">X-Auth-Token: MIIMKAYJKoZIhvcNAQcCoIIMGTCCDBUCAQExCTAHBgUrDgMCGjCCCwEGCS...eufVytyk=</div></div><div class="gmail_extra"><br></div><div class="gmail_extra">-----Answer from keystone to cinder-api:</div>
<div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">HTTP/1.1 200 OK</div><div class="gmail_extra">Vary: X-Auth-Token</div><div class="gmail_extra">Content-Type: application/json</div><div class="gmail_extra">
Content-Length: 612</div><div class="gmail_extra">Date: Tue, 30 Apr 2013 19:55:04 GMT</div><div class="gmail_extra"><br></div><div class="gmail_extra">{"signed": "-----BEGIN CMS-----\nMIIBkAYJKoZIhvcNAQcCoIIBgTCCAX0CAQExCTAHBgUrDgMCGjBrBgkqhkiG9w0B\nBwGgXgRceyJyZXZva2VkIjogW3siZXhwaXJlcyI6ICIyMDEzLTA0LTMwVDIwOjQy\nOjQ3WiIsICJpZCI6ICJhMDRhMjAwZGZlZTI2NjNkNDNjN2UyNzkzZTU3YWE1OCJ9\nXX0xgf8wgfwCAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVW5zZXQxDjAM\nBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEYMBYGA1UEAxMPd3d3LmV4YW1w\nbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIGAE4mgl+c2wGz0+71j\n5Am0KCI+lKHtYJppPtBvVDJ194J1hgMEMz7Yxlqtn1qMoJm3o5fCTl8pU3IszX/f\nb36zOZCrRXTCqgb32O7HfhPKT+N8kqZxMvtDTzv+3uQOC0xw7cAh+sNPgG1EHrL3\nIO8cMEUJqOkXjhwQPKXSqYVrwg4=\n-----END CMS-----\n"}</div>
<div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><div>-----Answer from cinder-api to cinder-client:</div><div><br></div><div><div>HTTP/1.1 401 Unauthorized</div><div>Www-Authenticate: Keystone uri='<a href="http://172.19.136.11:35357" target="_blank">http://172.19.136.11:35357</a>'</div>
<div>Content-Length: 276</div><div>Content-Type: text/plain; charset=UTF-8</div><div>Date: Tue, 30 Apr 2013 19:55:04 GMT</div><div class="im"><div><br></div><div>401 Unauthorized</div><div><br></div><div>This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.</div>
<div><br></div><div> Authentication required </div></div></div><div><br></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">Is there any chance that cinder-api is breaking up my token?? </div><div><div class="h5">
<div class="gmail_extra">
<br></div><div class="gmail_extra"><br></div><br><div class="gmail_quote">2013/4/30 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>I can get valid credentials with this line:</div>
<div><br></div><div>root@heladera:/etc/cinder# cinder --os-username=admin --os-tenant-name=admin --os-password=XXX --os-auth-url=<a href="http://172.19.136.1:35357/v2.0" target="_blank">http://172.19.136.1:35357/v2.0</a> credentials</div>
<div>+------------------+----------------------------------------------------------------------------------------+</div><div>| User Credentials | Value |</div>
<div>+------------------+----------------------------------------------------------------------------------------+</div><div>| id | 3f82673b5fe0411ab5fd8216bdb693c6 |</div>
<div>| name | admin |</div><div>| roles | [{u'name': u'KeystoneServiceAdmin'}, {u'name': u'KeystoneAdmin'}, {u'name': u'admin'}] |</div>
<div>| roles_links | [] |</div><div>| username | admin |</div>
<div>+------------------+----------------------------------------------------------------------------------------+</div><div>+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+</div>
<div>| Token | Value |</div>
<div>+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+</div>
<div>| expires | 2013-05-01T18:47:48Z |</div>
<div>| id | MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQEx...tcWW6xvpLgWsr3A== |</div><div>| issued_at | 2013-04-30T18:47:48.512440 |</div>
<div>| tenant | {u'id': u'6aa3bf1ab68040218873a782f90cffa7', u'enabled': True, u'description': None, u'name': u'admin'} |</div>
<div>+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+</div>
<div><br></div><div>So, it must be something that happens AFTER getting the credentials, something involving the cinder api. I'm not sure how the authentication process work but this is what i think:</div><div>
<br></div><div>1-cinder client request for an auth token</div><div>2-keystone validates the credentials, creates the token and sends it back to the client</div><div>3-the cinder client uses the received token to connect against the cinder api</div>
<div>4-the cinder api validates the token against ¿keystone? Here is where the problem might be.</div><div>5-somehow the api can't validate the token and rejects me.</div><div><br></div><div>I'm running out of ideas.</div>
<div><br></div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">2013/4/30 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">When i try to list the volumes this is what i see in the cinder api logs file:<div>
<br></div><div><div>
2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Authenticating user token</div>
<div>
2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role</div>
<div>2013-04-30 17:43:07 ERROR [keystoneclient.common.cms] Verify error: Verification failure</div><div><br></div><div>140606277047968:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:</div>
<div>140606277047968:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:</div><div>140606277047968:error:2E09A09E:CMS routines:CMS_SignerInfo_verify_content:verification failure:cms_sd.c:900:</div>
<div>140606277047968:error:2E09D06D:CMS routines:CMS_verify:content verify error:cms_smime.c:425:</div><div><br></div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Token validation failure.</div>
<div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 688, in _validate_user_token</div><div> verified = self.verify_signed_token(user_token)</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1043, in verify_signed_token</div><div> if self.is_signed_token_revoked(signed_text):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1007, in is_signed_token_revoked</div>
<div> revocation_list = self.token_revocation_list</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1079, in token_revocation_list</div><div> self.token_revocation_list = self.fetch_revocation_list()</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1109, in fetch_revocation_list</div><div> return self.cms_verify(data['signed'])</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1038, in cms_verify</div>
<div> raise err</div><div>CalledProcessError: Command 'openssl' returned non-zero exit status 4</div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Marking token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw== as unauthorized in memcache</div>
<div>2013-04-30 17:43:07 WARNING [keystoneclient.middleware.auth_token] Authorization failed for token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw==</div><div>2013-04-30 17:43:07 INFO [keystoneclient.middleware.auth_token] Invalid user token - rejecting request</div>
<div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Authenticating user token</div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role</div>
<div>2013-04-30 17:43:07 ERROR [keystoneclient.common.cms] Verify error: Verification failure</div><div><br></div><div>140558031275680:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:</div>
<div>140558031275680:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:</div><div>140558031275680:error:2E09A09E:CMS routines:CMS_SignerInfo_verify_content:verification failure:cms_sd.c:900:</div>
<div>140558031275680:error:2E09D06D:CMS routines:CMS_verify:content verify error:cms_smime.c:425:</div><div><br></div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Token validation failure.</div>
<div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 688, in _validate_user_token</div><div> verified = self.verify_signed_token(user_token)</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1043, in verify_signed_token</div><div> if self.is_signed_token_revoked(signed_text):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1007, in is_signed_token_revoked</div>
<div> revocation_list = self.token_revocation_list</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1079, in token_revocation_list</div><div> self.token_revocation_list = self.fetch_revocation_list()</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1109, in fetch_revocation_list</div><div> return self.cms_verify(data['signed'])</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1038, in cms_verify</div>
<div> raise err</div><div>CalledProcessError: Command 'openssl' returned non-zero exit status 4</div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Marking token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw== as unauthorized in memcache</div>
<div>2013-04-30 17:43:07 WARNING [keystoneclient.middleware.auth_token] Authorization failed for token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw==</div><div>2013-04-30 17:43:07 INFO [keystoneclient.middleware.auth_token] Invalid user token - rejecting request</div>
</div><div><br></div><div>MAYBE... somehow HAproxy is changing something in the header but i don't think so. This is the haproxy configuration for the cinder API:</div><div><br></div><div><div>listen nova-api-cinder <a href="http://172.19.136.1:8776" target="_blank">172.19.136.1:8776</a></div>
<div> balance roundrobin</div><div> option tcplog</div><div> server heladera <a href="http://172.19.136.245:8776" target="_blank">172.19.136.245:8776</a> check</div><div><br></div><div>I don't understand why is the Verification Failure, and why i have openssl involve in my authentication, I didn't change anything in the cinder api-paste.ini file, besides the auth_host and service_host. </div>
</div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">2013/4/30 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hi Jay, you are right, i'm trying to balance API calls with HAProxy. I installed HAproxy on 172.19.136.1 and configured all the openstack services to make the calls to that IP, then i use HAproxy to redirect the API calls to the real API servers (172.19.136.10 and 172.19.136.11), this is my configuration:<div>
<br></div><div>I've these 4 nodes: </div><div><br></div><div><a href="http://172.19.136.245" target="_blank">172.19.136.245</a>:</div><div>-Cinder<br><div><br></div><div><a href="http://172.19.136.10" target="_blank">172.19.136.10</a>:</div>
<div>
-Keystone</div><div>-Glance (glance, api, registry)</div><div>-Nova (compute, scheduler, etc)</div><div><br></div><div><a href="http://172.19.136.11" target="_blank">172.19.136.11</a>:</div><div><div>-Keystone</div>
<div>-Glance (glance, api, registry)</div><div>-Nova (compute, scheduler, etc)</div><div><br></div><div>172.19.136.2 / <a href="http://172.19.136.1" target="_blank">172.19.136.1</a>:</div><div>-Quantum server</div><div>-RabbitMQ</div>
<div>-MySQL</div><div>-HAProxy (Listening on 172.19.136.1 for all the API calls, and balancing them to either 172.19.136.10 or 172.19.136.11, it also listens for cinder api calls and redirects them to 172.19.136.245)</div>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">I didn't change all the endpoints yet, but all of them should redirect to 172.19.136.1, maybe that's the problem. What do you think? </div><div class="gmail_extra">
<br></div><div class="gmail_extra">This configuration might look odd or strange, but i'm trying to build a redundant and scalable cloud (like in this article <a href="http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/" target="_blank">http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/</a>). Thanks!!!<div>
<div><br>
<br><div class="gmail_quote">2013/4/30 Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div>On 04/29/2013 04:56 PM, Juan José Pavlik Salles wrote:<br>
> Hi, i have spent the last days trying to solve this problem. I can't<br>
> list my cinder volumes from my shell:<br>
><br>
> root@locro:~# cinder --os-username=admin --os-tenant-name=admin<br>
> --os-password=XXX --os-auth-url=<a href="http://172.19.136.1:35357/v2.0" target="_blank">http://172.19.136.1:35357/v2.0</a> --debug list<br>
><br>
> REQ: curl -i <a href="http://172.19.136.1:35357/v2.0/tokens" target="_blank">http://172.19.136.1:35357/v2.0/tokens</a> -X POST -H<br>
> "Content-Type: application/json" -H "Accept: application/json" -H<br>
> "User-Agent: python-cinderclient" -d '{"auth": {"tenantName": "admin",<br>
> "passwordCredentials": {"username": "admin", "password": "zGp05Nsa"}}}'<br>
><br>
> RESP: [200] {'date': 'Mon, 29 Apr 2013 17:24:44 GMT', 'content-type':<br>
> 'application/json', 'content-length': '7096', 'vary': 'X-Auth-Token'}<br>
> RESP BODY: {"access": {"token": {"issued_at":<br>
> "2013-04-29T17:24:44.044013", "expires": "2013-04-30T17:24:43Z", "id":<br>
> "MIIMaQYJKoZIhvcNAQcC...", "tenant": {"description": null, "enabled":<br>
> true, "id": "6aa3bf1ab68040218873a782f90cffa7", "name": "admin"}},<br>
> "serviceCatalog": [{"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7</a>",<br>
> "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7</a>", "id":<br>
> "26178391275a42cfa3b786ab151c8f8a", "publicURL":<br>
> "<a href="http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7</a>"}],<br>
> "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints":<br>
> [{"adminURL": "<a href="http://172.19.136.11:9696/" target="_blank">http://172.19.136.11:9696/</a>", "region": "RegionOne",<br>
> "internalURL": "<a href="http://172.19.136.11:9696/" target="_blank">http://172.19.136.11:9696/</a>", "id":<br>
> "1d0f394d83804ecaaa5ba708ccf0417b", "publicURL":<br>
> "<a href="http://172.19.136.11:9696/" target="_blank">http://172.19.136.11:9696/</a>"}], "endpoints_links": [], "type":<br>
> "network", "name": "quantum"}, {"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.10:9292/v2" target="_blank">http://172.19.136.10:9292/v2</a>", "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.11:9292/v2" target="_blank">http://172.19.136.11:9292/v2</a>", "id":<br>
> "11f37a313bad47f28b846cb9b94d458c", "publicURL":<br>
> "<a href="http://172.19.136.11:9292/v2" target="_blank">http://172.19.136.11:9292/v2</a>"}], "endpoints_links": [], "type":<br>
> "image", "name": "glance"}, {"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7</a>",<br>
> "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7</a>", "id":<br>
> "1ebe70478edd45d087263a4dc457f03a", "publicURL":<br>
> "<a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7</a>"}],<br>
> "endpoints_links": [], "type": "volume", "name": "cinder"},<br>
> {"endpoints": [{"adminURL": "<a href="http://172.19.136.11:8773/services/Admin" target="_blank">http://172.19.136.11:8773/services/Admin</a>",<br>
> "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.10:8773/services/Cloud" target="_blank">http://172.19.136.10:8773/services/Cloud</a>", "id":<br>
> "4fd5bcbee3584c2b883b08f22f81de54", "publicURL":<br>
> "<a href="http://172.19.136.10:8773/services/Cloud" target="_blank">http://172.19.136.10:8773/services/Cloud</a>"}], "endpoints_links": [],<br>
> "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.10:8080/v1" target="_blank">http://172.19.136.10:8080/v1</a>", "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7</a>",<br>
> "id": "65911114c36341a19006c328c6d0a2ae", "publicURL":<br>
> "<a href="http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7</a>"}],<br>
> "endpoints_links": [], "type": "object-store", "name": "swift"},<br>
> {"endpoints": [{"adminURL": "<a href="http://172.19.136.11:35357/v2.0" target="_blank">http://172.19.136.11:35357/v2.0</a>", "region":<br>
> "RegionOne", "internalURL": "<a href="http://172.19.136.10:5000/v2.0" target="_blank">http://172.19.136.10:5000/v2.0</a>", "id":<br>
> "0f9389d0485e4f2f9f7874c41181bd28", "publicURL":<br>
> "<a href="http://172.19.136.10:5000/v2.0" target="_blank">http://172.19.136.10:5000/v2.0</a>"}], "endpoints_links": [], "type":<br>
> "identity", "name": "keystone"}], "user": {"username": "admin",<br>
> "roles_links": [], "id": "3f82673b5fe0411ab5fd8216bdb693c6", "roles":<br>
> [{"name": "KeystoneServiceAdmin"}, {"name": "KeystoneAdmin"}, {"name":<br>
> "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles":<br>
> ["6666fa99078a4f07a070e7e858c32f02", "36bba9ef0178448c8a654b75feb3a0f4",<br>
> "a25581dd3470460b91ecaa29eca7205c"]}}}<br>
><br>
> REQ: curl -i<br>
> <a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail</a><br>
> -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent:<br>
> python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token:<br>
> MIIMaQYJKoZIhvcNAQcCo..."<br>
><br>
> RESP: [401] {'date': 'Mon, 29 Apr 2013 17:24:44 GMT', 'content-length':<br>
> '276', 'content-type': 'text/plain; charset=UTF-8', 'www-authenticate':<br>
> "Keystone uri='<a href="http://172.19.136.1:35357" target="_blank">http://172.19.136.1:35357</a>'"}<br>
> RESP BODY: 401 Unauthorized<br>
<br>
</div></div>From the above, the authentication URI that you are supplying to<br>
cinderclient is <a href="http://172.19.136.1:35357" target="_blank">http://172.19.136.1:35357</a>, which is not the same as what<br>
is returned in the service catalog above, which has the internalURL for<br>
the identity endpoint as <a href="http://172.19.136.10:5000/v2.0" target="_blank">http://172.19.136.10:5000/v2.0</a>.<br>
<br>
Is this intended?<br>
<br>
-jay<br>
<br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Pavlik Juan José
</font></span></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Pavlik Juan José
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Pavlik Juan José
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br>Pavlik Juan José
</font></span></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Pavlik Juan José
</div>