[Openstack-operators] Authentication problems with cinder
Daneyon Hansen (danehans)
danehans at cisco.com
Fri May 3 20:16:40 UTC 2013
It sounds like a bug. I have not seen the behavior myself, as I have always disabled SSL within Keystone.
Regards,
Daneyon Hansen
Software Engineer
Email: danehans at cisco.com
Phone: 303-718-0400
http://about.me/daneyon_hansen
From: Lorin Hochstein <lorin at nimbisservices.com<mailto:lorin at nimbisservices.com>>
Date: Friday, May 3, 2013 2:06 PM
To: Cisco Employee <danehans at cisco.com<mailto:danehans at cisco.com>>
Cc: Juan José Pavlik Salles <jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>>, "openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>" <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: Re: [Openstack-operators] Authentication problems with cinder
Daneyon saves the day! Daneyon, do you know if this is a known bug, or do we need to report it?
Lorin
On Fri, May 3, 2013 at 1:11 PM, Daneyon Hansen (danehans) <danehans at cisco.com<mailto:danehans at cisco.com>> wrote:
Keystone.conf
[ssl]
enable = False
[signing]
token_format = UUID
service keystone restart
Regards,
Daneyon Hansen
Software Engineer
Email: danehans at cisco.com<mailto:danehans at cisco.com>
Phone: 303-718-0400<tel:303-718-0400>
http://about.me/daneyon_hansen
From: Juan José Pavlik Salles <jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>>
Date: Friday, May 3, 2013 10:26 AM
To: "openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>" <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: Re: [Openstack-operators] Authentication problems with cinder
Thanks Jay!!! That makes sense. I'm using Grizzly, is there any way to disable the PKI??? It worked once, but suddenly stopped, and i don't know why. I just installed cinder again but the problem still there...
2013/5/3 Jay Pipes <jaypipes at gmail.com<mailto:jaypipes at gmail.com>>
We saw this exact same error when deploying Keystone +
Cinder/Nova/Glance with PKI in Folsom.
I presume you are using Grizzly, since I see you are also using memcache
with PKI, which does not work in Folsom, AFAIK.
The "solution" to the problem for us was to simply issue a restart of
the cinder-api/nova-api-os-compute/glance-api services, and the service
user would then begin to work again. I believe it has something to do
with the service user not being able to retrieve the token revocation
list from the Keystone server after some time period. For us, it was
usually around 24 hours between requisite restarts.
I've cc'd Adam Donnison to have a look at this as well.
Best,
-jay
On 05/02/2013 03:01 PM, Juan José Pavlik Salles wrote:
> Hi guys, i don't want to be annoying but i'm still having this problem.
> I don't understand this (from /var/log/cinder/cinder-api.log):
>
> 2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
> Token validation failure.
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 688, in _validate_user_token
> verified = self.verify_signed_token(user_token)
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1043, in verify_signed_token
> if self.is_signed_token_revoked(signed_text):
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1007, in is_signed_token_revoked
> revocation_list = self.token_revocation_list
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1079, in token_revocation_list
> self.token_revocation_list = self.fetch_revocation_list()
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1109, in fetch_revocation_list
> return self.cms_verify(data['signed'])
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1038, in cms_verify
> raise err
> CalledProcessError: Command 'openssl' returned non-zero exit status 4
> *2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
> Marking token
> *MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzY
> ...
> i4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
> *as unauthorized in memcache*
> *2013-04-30 20:00:42 WARNING [keystoneclient.middleware.auth_token]
> Authorization failed for token*
> MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzYjc4NmFiMTUxYzhmOGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTcifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuY
> ...
> dmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MSIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjgwODAvdjEvQVVUSF82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJpZCI6ICI2NTkxMTExNGMzNjM0MWExOTAwNmMzMjhjNmQwYTJhZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MS9BVVRIXzZhYTNiZjFhYjY4MDQwMjE4ODczYTc4MmY5MGNmZmE3In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm9iamVjdC1zdG9yZSIsICJuYW1lIjogInN3aWZ0In0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6MzUzNTcvdjIuMCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjUwMDAvdjIuMCIsICJpZCI6ICIwZjkzODlkMDQ4NWU0ZjJmOWY3ODc0YzQxMTgxYmQyOCIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pb
iIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
> *2013-04-30 20:00:42 INFO [keystoneclient.middleware.auth_token]
> Invalid user token - rejecting request*
> *
> *
> It seems that cinder can't recognise my auth_token so it tries to ban
> it. Does anybody have any idea about this? Thanks!!!
>
>
> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>
> <mailto:jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>>>
>
> I ran tcpdump on my cinder node (172.19.136.245) and this is what i saw:
>
> From 172.19.136.10 i ran "cinder --os-username=admin
> --os-tenant-name=admin --os-password=zGp05Nsa
> --os-auth-url=http://172.19.136.1:35357/v2.0 list":
>
> After getting a valid token from keystone.
>
> -----Request from cinder-client to cinder-api:
>
> GET /v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail HTTP/1.1
> Host: 172.19.136.245:8776<http://172.19.136.245:8776> <http://172.19.136.245:8776>
> X-Auth-Project-Id: admin
> Accept-Encoding: gzip, deflate, compress
> Content-Length: 0
> Accept: application/json
> User-Agent: python-cinderclient
> X-Auth-Token: MIIMbwYJKoZIhvcNAQcCoIIMY.....oiRM1nsw==
>
> -----Request from cinder-api to keystone:
>
> GET /v2.0/tokens/revoked HTTP/1.1
> Host: 172.19.136.11:35357<http://172.19.136.11:35357> <http://172.19.136.11:35357>
> Accept-Encoding: identity
> Content-type: application/json
> Accept: application/json
> X-Auth-Token:
> MIIMKAYJKoZIhvcNAQcCoIIMGTCCDBUCAQExCTAHBgUrDgMCGjCCCwEGCS...eufVytyk=
>
> -----Answer from keystone to cinder-api:
>
> HTTP/1.1 200 OK
> Vary: X-Auth-Token
> Content-Type: application/json
> Content-Length: 612
> Date: Tue, 30 Apr 2013 19:55:04 GMT
>
> {"signed": "-----BEGIN
> CMS-----\nMIIBkAYJKoZIhvcNAQcCoIIBgTCCAX0CAQExCTAHBgUrDgMCGjBrBgkqhkiG9w0B\nBwGgXgRceyJyZXZva2VkIjogW3siZXhwaXJlcyI6ICIyMDEzLTA0LTMwVDIwOjQy\nOjQ3WiIsICJpZCI6ICJhMDRhMjAwZGZlZTI2NjNkNDNjN2UyNzkzZTU3YWE1OCJ9\nXX0xgf8wgfwCAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVW5zZXQxDjAM\nBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEYMBYGA1UEAxMPd3d3LmV4YW1w\nbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIGAE4mgl+c2wGz0+71j\n5Am0KCI+lKHtYJppPtBvVDJ194J1hgMEMz7Yxlqtn1qMoJm3o5fCTl8pU3IszX/f\nb36zOZCrRXTCqgb32O7HfhPKT+N8kqZxMvtDTzv+3uQOC0xw7cAh+sNPgG1EHrL3\nIO8cMEUJqOkXjhwQPKXSqYVrwg4=\n-----END
> CMS-----\n"}
>
>
> -----Answer from cinder-api to cinder-client:
>
> HTTP/1.1 401 Unauthorized
> Www-Authenticate: Keystone uri='http://172.19.136.11:35357'
> Content-Length: 276
> Content-Type: text/plain; charset=UTF-8
> Date: Tue, 30 Apr 2013 19:55:04 GMT
>
> 401 Unauthorized
>
> This server could not verify that you are authorized to access the
> document you requested. Either you supplied the wrong credentials
> (e.g., bad password), or your browser does not understand how to
> supply the credentials required.
>
> Authentication required
>
>
> Is there any chance that cinder-api is breaking up my token??
>
>
>
> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>
> <mailto:jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>>>
>
> I can get valid credentials with this line:
>
> root at heladera:/etc/cinder# cinder --os-username=admin
> --os-tenant-name=admin --os-password=XXX
> --os-auth-url=http://172.19.136.1:35357/v2.0 credentials
> +------------------+----------------------------------------------------------------------------------------+
> | User Credentials |
> Value |
> +------------------+----------------------------------------------------------------------------------------+
> | id |
> 3f82673b5fe0411ab5fd8216bdb693c6 |
> | name |
> admin |
> | roles | [{u'name': u'KeystoneServiceAdmin'},
> {u'name': u'KeystoneAdmin'}, {u'name': u'admin'}] |
> | roles_links |
> [] |
> | username |
> admin |
> +------------------+----------------------------------------------------------------------------------------+
> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | Token |
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Value
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> |
> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | expires |
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2013-05-01T18:47:48Z
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> |
> | id |
> MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQEx...tcWW6xvpLgWsr3A== |
> | issued_at |
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2013-04-30T18:47:48.512440
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> |
> | tenant |
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> {u'id':
> u'6aa3bf1ab68040218873a782f90cffa7', u'enabled': True,
> u'description': None, u'name': u'admin'}
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> |
> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>
> So, it must be something that happens AFTER getting the
> credentials, something involving the cinder api. I'm not sure
> how the authentication process work but this is what i think:
>
> 1-cinder client request for an auth token
> 2-keystone validates the credentials, creates the token and
> sends it back to the client
> 3-the cinder client uses the received token to connect against
> the cinder api
> 4-the cinder api validates the token against ¿keystone? Here is
> where the problem might be.
> 5-somehow the api can't validate the token and rejects me.
>
> I'm running out of ideas.
>
>
>
> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>
> <mailto:jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>>>
>
> When i try to list the volumes this is what i see in the
> cinder api logs file:
>
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Authenticating user token
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Removing headers from
> request environment:
> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
> 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms]
> Verify error: Verification failure
>
> 140606277047968:error:0407006A:rsa
> routines:RSA_padding_check_PKCS1_type_1:block type is not
> 01:rsa_pk1.c:100:
> 140606277047968:error:04067072:rsa
> routines:RSA_EAY_PUBLIC_DECRYPT:padding check
> failed:rsa_eay.c:721:
> 140606277047968:error:2E09A09E:CMS
> routines:CMS_SignerInfo_verify_content:verification
> failure:cms_sd.c:900:
> 140606277047968:error:2E09D06D:CMS
> routines:CMS_verify:content verify error:cms_smime.c:425:
>
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Token validation failure.
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 688, in _validate_user_token
> verified = self.verify_signed_token(user_token)
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1043, in verify_signed_token
> if self.is_signed_token_revoked(signed_text):
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1007, in is_signed_token_revoked
> revocation_list = self.token_revocation_list
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1079, in token_revocation_list
> self.token_revocation_list = self.fetch_revocation_list()
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1109, in fetch_revocation_list
> return self.cms_verify(data['signed'])
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1038, in cms_verify
> raise err
> CalledProcessError: Command 'openssl' returned non-zero exit
> status 4
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Marking token
> MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw== as unauthorized in memcache
> 2013-04-30 17:43:07 WARNING
> [keystoneclient.middleware.auth_token] Authorization failed
> for token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw==
> 2013-04-30 17:43:07 INFO
> [keystoneclient.middleware.auth_token] Invalid user token -
> rejecting request
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Authenticating user token
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Removing headers from
> request environment:
> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
> 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms]
> Verify error: Verification failure
>
> 140558031275680:error:0407006A:rsa
> routines:RSA_padding_check_PKCS1_type_1:block type is not
> 01:rsa_pk1.c:100:
> 140558031275680:error:04067072:rsa
> routines:RSA_EAY_PUBLIC_DECRYPT:padding check
> failed:rsa_eay.c:721:
> 140558031275680:error:2E09A09E:CMS
> routines:CMS_SignerInfo_verify_content:verification
> failure:cms_sd.c:900:
> 140558031275680:error:2E09D06D:CMS
> routines:CMS_verify:content verify error:cms_smime.c:425:
>
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Token validation failure.
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 688, in _validate_user_token
> verified = self.verify_signed_token(user_token)
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1043, in verify_signed_token
> if self.is_signed_token_revoked(signed_text):
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1007, in is_signed_token_revoked
> revocation_list = self.token_revocation_list
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1079, in token_revocation_list
> self.token_revocation_list = self.fetch_revocation_list()
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1109, in fetch_revocation_list
> return self.cms_verify(data['signed'])
> File
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> line 1038, in cms_verify
> raise err
> CalledProcessError: Command 'openssl' returned non-zero exit
> status 4
> 2013-04-30 17:43:07 DEBUG
> [keystoneclient.middleware.auth_token] Marking token
> MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw== as unauthorized in memcache
> 2013-04-30 17:43:07 WARNING
> [keystoneclient.middleware.auth_token] Authorization failed
> for token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw==
> 2013-04-30 17:43:07 INFO
> [keystoneclient.middleware.auth_token] Invalid user token -
> rejecting request
>
> MAYBE... somehow HAproxy is changing something in the header
> but i don't think so. This is the haproxy configuration for
> the cinder API:
>
> listen nova-api-cinder 172.19.136.1:8776<http://172.19.136.1:8776>
> <http://172.19.136.1:8776>
> balance roundrobin
> option tcplog
> server heladera 172.19.136.245:8776<http://172.19.136.245:8776>
> <http://172.19.136.245:8776> check
>
> I don't understand why is the Verification Failure, and why
> i have openssl involve in my authentication, I didn't change
> anything in the cinder api-paste.ini file, besides the
> auth_host and service_host.
>
>
> 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>
> <mailto:jjpavlik at gmail.com<mailto:jjpavlik at gmail.com>>>
>
> Hi Jay, you are right, i'm trying to balance API calls
> with HAProxy. I installed HAproxy on 172.19.136.1 and
> configured all the openstack services to make the calls
> to that IP, then i use HAproxy to redirect the API calls
> to the real API servers (172.19.136.10 and
> 172.19.136.11), this is my configuration:
>
> I've these 4 nodes:
>
> 172.19.136.245 <http://172.19.136.245>:
> -Cinder
>
> 172.19.136.10 <http://172.19.136.10>:
> -Keystone
> -Glance (glance, api, registry)
> -Nova (compute, scheduler, etc)
>
> 172.19.136.11 <http://172.19.136.11>:
> -Keystone
> -Glance (glance, api, registry)
> -Nova (compute, scheduler, etc)
>
> 172.19.136.2 / 172.19.136.1 <http://172.19.136.1>:
> -Quantum server
> -RabbitMQ
> -MySQL
> -HAProxy (Listening on 172.19.136.1 for all the API
> calls, and balancing them to either 172.19.136.10 or
> 172.19.136.11, it also listens for cinder api calls and
> redirects them to 172.19.136.245)
>
> I didn't change all the endpoints yet, but all of them
> should redirect to 172.19.136.1, maybe that's the
> problem. What do you think?
>
> This configuration might look odd or strange, but i'm
> trying to build a redundant and scalable cloud (like in
> this article
> http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/).
> Thanks!!!
>
>
> 2013/4/30 Jay Pipes <jaypipes at gmail.com<mailto:jaypipes at gmail.com>
> <mailto:jaypipes at gmail.com<mailto:jaypipes at gmail.com>>>
>
> On 04/29/2013 04:56 PM, Juan José Pavlik Salles wrote:
> > Hi, i have spent the last days trying to solve
> this problem. I can't
> > list my cinder volumes from my shell:
> >
> > root at locro:~# cinder --os-username=admin
> --os-tenant-name=admin
> > --os-password=XXX
> --os-auth-url=http://172.19.136.1:35357/v2.0 --debug
> list
> >
> > REQ: curl -i http://172.19.136.1:35357/v2.0/tokens
> -X POST -H
> > "Content-Type: application/json" -H "Accept:
> application/json" -H
> > "User-Agent: python-cinderclient" -d '{"auth":
> {"tenantName": "admin",
> > "passwordCredentials": {"username": "admin",
> "password": "zGp05Nsa"}}}'
> >
> > RESP: [200] {'date': 'Mon, 29 Apr 2013 17:24:44
> GMT', 'content-type':
> > 'application/json', 'content-length': '7096',
> 'vary': 'X-Auth-Token'}
> > RESP BODY: {"access": {"token": {"issued_at":
> > "2013-04-29T17:24:44.044013", "expires":
> "2013-04-30T17:24:43Z", "id":
> > "MIIMaQYJKoZIhvcNAQcC...", "tenant":
> {"description": null, "enabled":
> > true, "id": "6aa3bf1ab68040218873a782f90cffa7",
> "name": "admin"}},
> > "serviceCatalog": [{"endpoints": [{"adminURL":
> >
> "http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
> > "region": "RegionOne", "internalURL":
> >
> "http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
> "id":
> > "26178391275a42cfa3b786ab151c8f8a", "publicURL":
> >
> "http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7"}],
> > "endpoints_links": [], "type": "compute", "name":
> "nova"}, {"endpoints":
> > [{"adminURL": "http://172.19.136.11:9696/",
> "region": "RegionOne",
> > "internalURL": "http://172.19.136.11:9696/", "id":
> > "1d0f394d83804ecaaa5ba708ccf0417b", "publicURL":
> > "http://172.19.136.11:9696/"}], "endpoints_links":
> [], "type":
> > "network", "name": "quantum"}, {"endpoints":
> [{"adminURL":
> > "http://172.19.136.10:9292/v2", "region":
> "RegionOne", "internalURL":
> > "http://172.19.136.11:9292/v2", "id":
> > "11f37a313bad47f28b846cb9b94d458c", "publicURL":
> > "http://172.19.136.11:9292/v2"}],
> "endpoints_links": [], "type":
> > "image", "name": "glance"}, {"endpoints":
> [{"adminURL":
> >
> "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
> > "region": "RegionOne", "internalURL":
> >
> "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
> "id":
> > "1ebe70478edd45d087263a4dc457f03a", "publicURL":
> >
> "http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7"}],
> > "endpoints_links": [], "type": "volume", "name":
> "cinder"},
> > {"endpoints": [{"adminURL":
> "http://172.19.136.11:8773/services/Admin",
> > "region": "RegionOne", "internalURL":
> > "http://172.19.136.10:8773/services/Cloud", "id":
> > "4fd5bcbee3584c2b883b08f22f81de54", "publicURL":
> > "http://172.19.136.10:8773/services/Cloud"}],
> "endpoints_links": [],
> > "type": "ec2", "name": "ec2"}, {"endpoints":
> [{"adminURL":
> > "http://172.19.136.10:8080/v1", "region":
> "RegionOne", "internalURL":
> >
> "http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7",
> > "id": "65911114c36341a19006c328c6d0a2ae", "publicURL":
> >
> "http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7"}],
> > "endpoints_links": [], "type": "object-store",
> "name": "swift"},
> > {"endpoints": [{"adminURL":
> "http://172.19.136.11:35357/v2.0", "region":
> > "RegionOne", "internalURL":
> "http://172.19.136.10:5000/v2.0", "id":
> > "0f9389d0485e4f2f9f7874c41181bd28", "publicURL":
> > "http://172.19.136.10:5000/v2.0"}],
> "endpoints_links": [], "type":
> > "identity", "name": "keystone"}], "user":
> {"username": "admin",
> > "roles_links": [], "id":
> "3f82673b5fe0411ab5fd8216bdb693c6", "roles":
> > [{"name": "KeystoneServiceAdmin"}, {"name":
> "KeystoneAdmin"}, {"name":
> > "admin"}], "name": "admin"}, "metadata":
> {"is_admin": 0, "roles":
> > ["6666fa99078a4f07a070e7e858c32f02",
> "36bba9ef0178448c8a654b75feb3a0f4",
> > "a25581dd3470460b91ecaa29eca7205c"]}}}
> >
> > REQ: curl -i
> >
> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail
> > -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent:
> > python-cinderclient" -H "Accept: application/json"
> -H "X-Auth-Token:
> > MIIMaQYJKoZIhvcNAQcCo..."
> >
> > RESP: [401] {'date': 'Mon, 29 Apr 2013 17:24:44
> GMT', 'content-length':
> > '276', 'content-type': 'text/plain;
> charset=UTF-8', 'www-authenticate':
> > "Keystone uri='http://172.19.136.1:35357'"}
> > RESP BODY: 401 Unauthorized
>
> From the above, the authentication URI that you are
> supplying to
> cinderclient is http://172.19.136.1:35357, which is
> not the same as what
> is returned in the service catalog above, which has
> the internalURL for
> the identity endpoint as http://172.19.136.10:5000/v2.0.
>
> Is this intended?
>
> -jay
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>
> <mailto:OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
>
> --
> Pavlik Juan José
>
>
>
>
> --
> Pavlik Juan José
>
>
>
>
> --
> Pavlik Juan José
>
>
>
>
> --
> Pavlik Juan José
>
>
>
>
> --
> Pavlik Juan José
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
--
Pavlik Juan José
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
--
Lorin Hochstein
Lead Architect - Cloud Services
Nimbis Services, Inc.
www.nimbisservices.com<http://www.nimbisservices.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130503/7d4b0215/attachment.html>
More information about the OpenStack-operators
mailing list