[Openstack-operators] Authentication problems with cinder
Juan Suero
juan.suero at gmail.com
Fri May 3 17:26:16 UTC 2013
hey this might not help you but i had similar problems. dashboard would
not work. it would authenticate me but not show me resources and such....
i traced it down to similar authentication problem to cinder
the how to i was following did not put the cinder user as an admin of the
service tenant if i recall correctly.
manually making that association fixed that issue for me.
e.g. putting cinder in the admin role in teh service tenant.
On Fri, May 3, 2013 at 1:11 PM, Daneyon Hansen (danehans) <
danehans at cisco.com> wrote:
>
> Keystone.conf
>
> [ssl]
> enable = False
>
> [signing]
> token_format = UUID
>
> service keystone restart
>
> Regards,
> Daneyon Hansen
> Software Engineer
> Email: danehans at cisco.com
> Phone: 303-718-0400
> http://about.me/daneyon_hansen
>
> From: Juan José Pavlik Salles <jjpavlik at gmail.com>
> Date: Friday, May 3, 2013 10:26 AM
> To: "openstack-operators at lists.openstack.org" <
> openstack-operators at lists.openstack.org>
> Subject: Re: [Openstack-operators] Authentication problems with cinder
>
> Thanks Jay!!! That makes sense. I'm using Grizzly, is there any way to
> disable the PKI??? It worked once, but suddenly stopped, and i don't know
> why. I just installed cinder again but the problem still there...
>
>
> 2013/5/3 Jay Pipes <jaypipes at gmail.com>
>
>> We saw this exact same error when deploying Keystone +
>> Cinder/Nova/Glance with PKI in Folsom.
>>
>> I presume you are using Grizzly, since I see you are also using memcache
>> with PKI, which does not work in Folsom, AFAIK.
>>
>> The "solution" to the problem for us was to simply issue a restart of
>> the cinder-api/nova-api-os-compute/glance-api services, and the service
>> user would then begin to work again. I believe it has something to do
>> with the service user not being able to retrieve the token revocation
>> list from the Keystone server after some time period. For us, it was
>> usually around 24 hours between requisite restarts.
>>
>> I've cc'd Adam Donnison to have a look at this as well.
>>
>> Best,
>> -jay
>>
>> On 05/02/2013 03:01 PM, Juan José Pavlik Salles wrote:
>> > Hi guys, i don't want to be annoying but i'm still having this problem.
>> > I don't understand this (from /var/log/cinder/cinder-api.log):
>> >
>> > 2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
>> > Token validation failure.
>> > Traceback (most recent call last):
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 688, in _validate_user_token
>> > verified = self.verify_signed_token(user_token)
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1043, in verify_signed_token
>> > if self.is_signed_token_revoked(signed_text):
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1007, in is_signed_token_revoked
>> > revocation_list = self.token_revocation_list
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1079, in token_revocation_list
>> > self.token_revocation_list = self.fetch_revocation_list()
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1109, in fetch_revocation_list
>> > return self.cms_verify(data['signed'])
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1038, in cms_verify
>> > raise err
>> > CalledProcessError: Command 'openssl' returned non-zero exit status 4
>> > *2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
>> > Marking token
>> >
>> *MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzY
>> > ...
>> >
>> i4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
>> > *as unauthorized in memcache*
>> > *2013-04-30 20:00:42 WARNING [keystoneclient.middleware.auth_token]
>> > Authorization failed for token*
>> >
>> MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzYjc4NmFiMTUxYzhmOGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTcifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuY
>> > ...
>> >
>> dmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MSIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjgwODAvdjEvQVVUSF82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJpZCI6ICI2NTkxMTExNGMzNjM0MWExOTAwNmMzMjhjNmQwYTJhZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MS9BVVRIXzZhYTNiZjFhYjY4MDQwMjE4ODczYTc4MmY5MGNmZmE3In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm9iamVjdC1zdG9yZSIsICJuYW1lIjogInN3aWZ0In0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6MzUzNTcvdjIuMCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjUwMDAvdjIuMCIsICJpZCI6ICIwZjkzODlkMDQ4NWU0ZjJmOWY3ODc0YzQxMTgxYmQyOCIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pb
>>
>> iIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
>> > *2013-04-30 20:00:42 INFO [keystoneclient.middleware.auth_token]
>> > Invalid user token - rejecting request*
>> > *
>> > *
>> > It seems that cinder can't recognise my auth_token so it tries to ban
>> > it. Does anybody have any idea about this? Thanks!!!
>> >
>> >
>> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
>> > <mailto:jjpavlik at gmail.com>>
>> >
>> > I ran tcpdump on my cinder node (172.19.136.245) and this is what i
>> saw:
>> >
>> > From 172.19.136.10 i ran "cinder --os-username=admin
>> > --os-tenant-name=admin --os-password=zGp05Nsa
>> > --os-auth-url=http://172.19.136.1:35357/v2.0 list":
>> >
>> > After getting a valid token from keystone.
>> >
>> > -----Request from cinder-client to cinder-api:
>> >
>> > GET /v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail HTTP/1.1
>> > Host: 172.19.136.245:8776 <http://172.19.136.245:8776>
>> > X-Auth-Project-Id: admin
>> > Accept-Encoding: gzip, deflate, compress
>> > Content-Length: 0
>> > Accept: application/json
>> > User-Agent: python-cinderclient
>> > X-Auth-Token: MIIMbwYJKoZIhvcNAQcCoIIMY.....oiRM1nsw==
>> >
>> > -----Request from cinder-api to keystone:
>> >
>> > GET /v2.0/tokens/revoked HTTP/1.1
>> > Host: 172.19.136.11:35357 <http://172.19.136.11:35357>
>> > Accept-Encoding: identity
>> > Content-type: application/json
>> > Accept: application/json
>> > X-Auth-Token:
>> >
>> MIIMKAYJKoZIhvcNAQcCoIIMGTCCDBUCAQExCTAHBgUrDgMCGjCCCwEGCS...eufVytyk=
>> >
>> > -----Answer from keystone to cinder-api:
>> >
>> > HTTP/1.1 200 OK
>> > Vary: X-Auth-Token
>> > Content-Type: application/json
>> > Content-Length: 612
>> > Date: Tue, 30 Apr 2013 19:55:04 GMT
>> >
>> > {"signed": "-----BEGIN
>> >
>> CMS-----\nMIIBkAYJKoZIhvcNAQcCoIIBgTCCAX0CAQExCTAHBgUrDgMCGjBrBgkqhkiG9w0B\nBwGgXgRceyJyZXZva2VkIjogW3siZXhwaXJlcyI6ICIyMDEzLTA0LTMwVDIwOjQy\nOjQ3WiIsICJpZCI6ICJhMDRhMjAwZGZlZTI2NjNkNDNjN2UyNzkzZTU3YWE1OCJ9\nXX0xgf8wgfwCAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVW5zZXQxDjAM\nBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEYMBYGA1UEAxMPd3d3LmV4YW1w\nbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIGAE4mgl+c2wGz0+71j\n5Am0KCI+lKHtYJppPtBvVDJ194J1hgMEMz7Yxlqtn1qMoJm3o5fCTl8pU3IszX/f\nb36zOZCrRXTCqgb32O7HfhPKT+N8kqZxMvtDTzv+3uQOC0xw7cAh+sNPgG1EHrL3\nIO8cMEUJqOkXjhwQPKXSqYVrwg4=\n-----END
>> > CMS-----\n"}
>> >
>> >
>> > -----Answer from cinder-api to cinder-client:
>> >
>> > HTTP/1.1 401 Unauthorized
>> > Www-Authenticate: Keystone uri='http://172.19.136.11:35357'
>> > Content-Length: 276
>> > Content-Type: text/plain; charset=UTF-8
>> > Date: Tue, 30 Apr 2013 19:55:04 GMT
>> >
>> > 401 Unauthorized
>> >
>> > This server could not verify that you are authorized to access the
>> > document you requested. Either you supplied the wrong credentials
>> > (e.g., bad password), or your browser does not understand how to
>> > supply the credentials required.
>> >
>> > Authentication required
>> >
>> >
>> > Is there any chance that cinder-api is breaking up my token??
>> >
>> >
>> >
>> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
>> > <mailto:jjpavlik at gmail.com>>
>> >
>> > I can get valid credentials with this line:
>> >
>> > root at heladera:/etc/cinder# cinder --os-username=admin
>> > --os-tenant-name=admin --os-password=XXX
>> > --os-auth-url=http://172.19.136.1:35357/v2.0 credentials
>> >
>> +------------------+----------------------------------------------------------------------------------------+
>> > | User Credentials |
>> > Value |
>> >
>> +------------------+----------------------------------------------------------------------------------------+
>> > | id |
>> > 3f82673b5fe0411ab5fd8216bdb693c6 |
>> > | name |
>> > admin |
>> > | roles | [{u'name': u'KeystoneServiceAdmin'},
>> > {u'name': u'KeystoneAdmin'}, {u'name': u'admin'}] |
>> > | roles_links |
>> > [] |
>> > | username |
>> > admin |
>> >
>> +------------------+----------------------------------------------------------------------------------------+
>> >
>> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>> > | Token |
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Value
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > |
>> >
>> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>> > | expires |
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > 2013-05-01T18:47:48Z
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > |
>> > | id |
>> > MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQEx...tcWW6xvpLgWsr3A== |
>> > | issued_at |
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > 2013-04-30T18:47:48.512440
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > |
>> > | tenant |
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > {u'id':
>> > u'6aa3bf1ab68040218873a782f90cffa7', u'enabled': True,
>> > u'description': None, u'name': u'admin'}
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> |
>> >
>> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>> >
>> > So, it must be something that happens AFTER getting the
>> > credentials, something involving the cinder api. I'm not sure
>> > how the authentication process work but this is what i think:
>> >
>> > 1-cinder client request for an auth token
>> > 2-keystone validates the credentials, creates the token and
>> > sends it back to the client
>> > 3-the cinder client uses the received token to connect against
>> > the cinder api
>> > 4-the cinder api validates the token against ¿keystone? Here is
>> > where the problem might be.
>> > 5-somehow the api can't validate the token and rejects me.
>> >
>> > I'm running out of ideas.
>> >
>> >
>> >
>> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
>> > <mailto:jjpavlik at gmail.com>>
>> >
>> > When i try to list the volumes this is what i see in the
>> > cinder api logs file:
>> >
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Authenticating user
>> token
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Removing headers from
>> > request environment:
>> >
>> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
>> > 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms]
>> > Verify error: Verification failure
>> >
>> > 140606277047968:error:0407006A:rsa
>> > routines:RSA_padding_check_PKCS1_type_1:block type is not
>> > 01:rsa_pk1.c:100:
>> > 140606277047968:error:04067072:rsa
>> > routines:RSA_EAY_PUBLIC_DECRYPT:padding check
>> > failed:rsa_eay.c:721:
>> > 140606277047968:error:2E09A09E:CMS
>> > routines:CMS_SignerInfo_verify_content:verification
>> > failure:cms_sd.c:900:
>> > 140606277047968:error:2E09D06D:CMS
>> > routines:CMS_verify:content verify error:cms_smime.c:425:
>> >
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Token validation
>> failure.
>> > Traceback (most recent call last):
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 688, in _validate_user_token
>> > verified = self.verify_signed_token(user_token)
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1043, in verify_signed_token
>> > if self.is_signed_token_revoked(signed_text):
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1007, in is_signed_token_revoked
>> > revocation_list = self.token_revocation_list
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1079, in token_revocation_list
>> > self.token_revocation_list =
>> self.fetch_revocation_list()
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1109, in fetch_revocation_list
>> > return self.cms_verify(data['signed'])
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1038, in cms_verify
>> > raise err
>> > CalledProcessError: Command 'openssl' returned non-zero exit
>> > status 4
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Marking token
>> > MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw== as unauthorized in
>> memcache
>> > 2013-04-30 17:43:07 WARNING
>> > [keystoneclient.middleware.auth_token] Authorization failed
>> > for token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw==
>> > 2013-04-30 17:43:07 INFO
>> > [keystoneclient.middleware.auth_token] Invalid user token -
>> > rejecting request
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Authenticating user
>> token
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Removing headers from
>> > request environment:
>> >
>> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
>> > 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms]
>> > Verify error: Verification failure
>> >
>> > 140558031275680:error:0407006A:rsa
>> > routines:RSA_padding_check_PKCS1_type_1:block type is not
>> > 01:rsa_pk1.c:100:
>> > 140558031275680:error:04067072:rsa
>> > routines:RSA_EAY_PUBLIC_DECRYPT:padding check
>> > failed:rsa_eay.c:721:
>> > 140558031275680:error:2E09A09E:CMS
>> > routines:CMS_SignerInfo_verify_content:verification
>> > failure:cms_sd.c:900:
>> > 140558031275680:error:2E09D06D:CMS
>> > routines:CMS_verify:content verify error:cms_smime.c:425:
>> >
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Token validation
>> failure.
>> > Traceback (most recent call last):
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 688, in _validate_user_token
>> > verified = self.verify_signed_token(user_token)
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1043, in verify_signed_token
>> > if self.is_signed_token_revoked(signed_text):
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1007, in is_signed_token_revoked
>> > revocation_list = self.token_revocation_list
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1079, in token_revocation_list
>> > self.token_revocation_list =
>> self.fetch_revocation_list()
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1109, in fetch_revocation_list
>> > return self.cms_verify(data['signed'])
>> > File
>> >
>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>> > line 1038, in cms_verify
>> > raise err
>> > CalledProcessError: Command 'openssl' returned non-zero exit
>> > status 4
>> > 2013-04-30 17:43:07 DEBUG
>> > [keystoneclient.middleware.auth_token] Marking token
>> > MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw== as unauthorized in
>> memcache
>> > 2013-04-30 17:43:07 WARNING
>> > [keystoneclient.middleware.auth_token] Authorization failed
>> > for token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw==
>> > 2013-04-30 17:43:07 INFO
>> > [keystoneclient.middleware.auth_token] Invalid user token -
>> > rejecting request
>> >
>> > MAYBE... somehow HAproxy is changing something in the header
>> > but i don't think so. This is the haproxy configuration for
>> > the cinder API:
>> >
>> > listen nova-api-cinder 172.19.136.1:8776
>> > <http://172.19.136.1:8776>
>> > balance roundrobin
>> > option tcplog
>> > server heladera 172.19.136.245:8776
>> > <http://172.19.136.245:8776> check
>> >
>> > I don't understand why is the Verification Failure, and why
>> > i have openssl involve in my authentication, I didn't change
>> > anything in the cinder api-paste.ini file, besides the
>> > auth_host and service_host.
>> >
>> >
>> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
>> > <mailto:jjpavlik at gmail.com>>
>> >
>> > Hi Jay, you are right, i'm trying to balance API calls
>> > with HAProxy. I installed HAproxy on 172.19.136.1 and
>> > configured all the openstack services to make the calls
>> > to that IP, then i use HAproxy to redirect the API calls
>> > to the real API servers (172.19.136.10 and
>> > 172.19.136.11), this is my configuration:
>> >
>> > I've these 4 nodes:
>> >
>> > 172.19.136.245 <http://172.19.136.245>:
>> > -Cinder
>> >
>> > 172.19.136.10 <http://172.19.136.10>:
>> > -Keystone
>> > -Glance (glance, api, registry)
>> > -Nova (compute, scheduler, etc)
>> >
>> > 172.19.136.11 <http://172.19.136.11>:
>> > -Keystone
>> > -Glance (glance, api, registry)
>> > -Nova (compute, scheduler, etc)
>> >
>> > 172.19.136.2 / 172.19.136.1 <http://172.19.136.1>:
>> > -Quantum server
>> > -RabbitMQ
>> > -MySQL
>> > -HAProxy (Listening on 172.19.136.1 for all the API
>> > calls, and balancing them to either 172.19.136.10 or
>> > 172.19.136.11, it also listens for cinder api calls and
>> > redirects them to 172.19.136.245)
>> >
>> > I didn't change all the endpoints yet, but all of them
>> > should redirect to 172.19.136.1, maybe that's the
>> > problem. What do you think?
>> >
>> > This configuration might look odd or strange, but i'm
>> > trying to build a redundant and scalable cloud (like in
>> > this article
>> >
>> http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/
>> ).
>> > Thanks!!!
>> >
>> >
>> > 2013/4/30 Jay Pipes <jaypipes at gmail.com
>> > <mailto:jaypipes at gmail.com>>
>> >
>> > On 04/29/2013 04:56 PM, Juan José Pavlik Salles
>> wrote:
>> > > Hi, i have spent the last days trying to solve
>> > this problem. I can't
>> > > list my cinder volumes from my shell:
>> > >
>> > > root at locro:~# cinder --os-username=admin
>> > --os-tenant-name=admin
>> > > --os-password=XXX
>> > --os-auth-url=http://172.19.136.1:35357/v2.0--debug
>> > list
>> > >
>> > > REQ: curl -i
>> http://172.19.136.1:35357/v2.0/tokens
>> > -X POST -H
>> > > "Content-Type: application/json" -H "Accept:
>> > application/json" -H
>> > > "User-Agent: python-cinderclient" -d '{"auth":
>> > {"tenantName": "admin",
>> > > "passwordCredentials": {"username": "admin",
>> > "password": "zGp05Nsa"}}}'
>> > >
>> > > RESP: [200] {'date': 'Mon, 29 Apr 2013 17:24:44
>> > GMT', 'content-type':
>> > > 'application/json', 'content-length': '7096',
>> > 'vary': 'X-Auth-Token'}
>> > > RESP BODY: {"access": {"token": {"issued_at":
>> > > "2013-04-29T17:24:44.044013", "expires":
>> > "2013-04-30T17:24:43Z", "id":
>> > > "MIIMaQYJKoZIhvcNAQcC...", "tenant":
>> > {"description": null, "enabled":
>> > > true, "id": "6aa3bf1ab68040218873a782f90cffa7",
>> > "name": "admin"}},
>> > > "serviceCatalog": [{"endpoints": [{"adminURL":
>> > >
>> > "
>> http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
>> > > "region": "RegionOne", "internalURL":
>> > >
>> > "
>> http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
>> > "id":
>> > > "26178391275a42cfa3b786ab151c8f8a", "publicURL":
>> > >
>> > "
>> http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7"}],
>> > > "endpoints_links": [], "type": "compute", "name":
>> > "nova"}, {"endpoints":
>> > > [{"adminURL": "http://172.19.136.11:9696/",
>> > "region": "RegionOne",
>> > > "internalURL": "http://172.19.136.11:9696/",
>> "id":
>> > > "1d0f394d83804ecaaa5ba708ccf0417b", "publicURL":
>> > > "http://172.19.136.11:9696/"}],
>> "endpoints_links":
>> > [], "type":
>> > > "network", "name": "quantum"}, {"endpoints":
>> > [{"adminURL":
>> > > "http://172.19.136.10:9292/v2", "region":
>> > "RegionOne", "internalURL":
>> > > "http://172.19.136.11:9292/v2", "id":
>> > > "11f37a313bad47f28b846cb9b94d458c", "publicURL":
>> > > "http://172.19.136.11:9292/v2"}],
>> > "endpoints_links": [], "type":
>> > > "image", "name": "glance"}, {"endpoints":
>> > [{"adminURL":
>> > >
>> > "
>> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
>> > > "region": "RegionOne", "internalURL":
>> > >
>> > "
>> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
>> > "id":
>> > > "1ebe70478edd45d087263a4dc457f03a", "publicURL":
>> > >
>> > "
>> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7"}],
>> > > "endpoints_links": [], "type": "volume", "name":
>> > "cinder"},
>> > > {"endpoints": [{"adminURL":
>> > "http://172.19.136.11:8773/services/Admin",
>> > > "region": "RegionOne", "internalURL":
>> > > "http://172.19.136.10:8773/services/Cloud", "id":
>> > > "4fd5bcbee3584c2b883b08f22f81de54", "publicURL":
>> > > "http://172.19.136.10:8773/services/Cloud"}],
>> > "endpoints_links": [],
>> > > "type": "ec2", "name": "ec2"}, {"endpoints":
>> > [{"adminURL":
>> > > "http://172.19.136.10:8080/v1", "region":
>> > "RegionOne", "internalURL":
>> > >
>> > "
>> http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7",
>> > > "id": "65911114c36341a19006c328c6d0a2ae",
>> "publicURL":
>> > >
>> > "
>> http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7"}],
>> > > "endpoints_links": [], "type": "object-store",
>> > "name": "swift"},
>> > > {"endpoints": [{"adminURL":
>> > "http://172.19.136.11:35357/v2.0", "region":
>> > > "RegionOne", "internalURL":
>> > "http://172.19.136.10:5000/v2.0", "id":
>> > > "0f9389d0485e4f2f9f7874c41181bd28", "publicURL":
>> > > "http://172.19.136.10:5000/v2.0"}],
>> > "endpoints_links": [], "type":
>> > > "identity", "name": "keystone"}], "user":
>> > {"username": "admin",
>> > > "roles_links": [], "id":
>> > "3f82673b5fe0411ab5fd8216bdb693c6", "roles":
>> > > [{"name": "KeystoneServiceAdmin"}, {"name":
>> > "KeystoneAdmin"}, {"name":
>> > > "admin"}], "name": "admin"}, "metadata":
>> > {"is_admin": 0, "roles":
>> > > ["6666fa99078a4f07a070e7e858c32f02",
>> > "36bba9ef0178448c8a654b75feb3a0f4",
>> > > "a25581dd3470460b91ecaa29eca7205c"]}}}
>> > >
>> > > REQ: curl -i
>> > >
>> >
>> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail
>> > > -X GET -H "X-Auth-Project-Id: admin" -H
>> "User-Agent:
>> > > python-cinderclient" -H "Accept: application/json"
>> > -H "X-Auth-Token:
>> > > MIIMaQYJKoZIhvcNAQcCo..."
>> > >
>> > > RESP: [401] {'date': 'Mon, 29 Apr 2013 17:24:44
>> > GMT', 'content-length':
>> > > '276', 'content-type': 'text/plain;
>> > charset=UTF-8', 'www-authenticate':
>> > > "Keystone uri='http://172.19.136.1:35357'"}
>> > > RESP BODY: 401 Unauthorized
>> >
>> > From the above, the authentication URI that you are
>> > supplying to
>> > cinderclient is http://172.19.136.1:35357, which is
>> > not the same as what
>> > is returned in the service catalog above, which has
>> > the internalURL for
>> > the identity endpoint as
>> http://172.19.136.10:5000/v2.0.
>> >
>> > Is this intended?
>> >
>> > -jay
>> >
>> >
>> > _______________________________________________
>> > OpenStack-operators mailing list
>> > OpenStack-operators at lists.openstack.org
>> > <mailto:OpenStack-operators at lists.openstack.org>
>> >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>> >
>> >
>> >
>> > --
>> > Pavlik Juan José
>> >
>> >
>> >
>> >
>> > --
>> > Pavlik Juan José
>> >
>> >
>> >
>> >
>> > --
>> > Pavlik Juan José
>> >
>> >
>> >
>> >
>> > --
>> > Pavlik Juan José
>> >
>> >
>> >
>> >
>> > --
>> > Pavlik Juan José
>> >
>> >
>> > _______________________________________________
>> > OpenStack-operators mailing list
>> > OpenStack-operators at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
>
> --
> Pavlik Juan José
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130503/aa818425/attachment-0001.html>
More information about the OpenStack-operators
mailing list