[Openstack-operators] Authentication problems with cinder
Juan José Pavlik Salles
jjpavlik at gmail.com
Fri May 3 16:26:52 UTC 2013
Thanks Jay!!! That makes sense. I'm using Grizzly, is there any way to
disable the PKI??? It worked once, but suddenly stopped, and i don't know
why. I just installed cinder again but the problem still there...
2013/5/3 Jay Pipes <jaypipes at gmail.com>
> We saw this exact same error when deploying Keystone +
> Cinder/Nova/Glance with PKI in Folsom.
>
> I presume you are using Grizzly, since I see you are also using memcache
> with PKI, which does not work in Folsom, AFAIK.
>
> The "solution" to the problem for us was to simply issue a restart of
> the cinder-api/nova-api-os-compute/glance-api services, and the service
> user would then begin to work again. I believe it has something to do
> with the service user not being able to retrieve the token revocation
> list from the Keystone server after some time period. For us, it was
> usually around 24 hours between requisite restarts.
>
> I've cc'd Adam Donnison to have a look at this as well.
>
> Best,
> -jay
>
> On 05/02/2013 03:01 PM, Juan José Pavlik Salles wrote:
> > Hi guys, i don't want to be annoying but i'm still having this problem.
> > I don't understand this (from /var/log/cinder/cinder-api.log):
> >
> > 2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
> > Token validation failure.
> > Traceback (most recent call last):
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 688, in _validate_user_token
> > verified = self.verify_signed_token(user_token)
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1043, in verify_signed_token
> > if self.is_signed_token_revoked(signed_text):
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1007, in is_signed_token_revoked
> > revocation_list = self.token_revocation_list
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1079, in token_revocation_list
> > self.token_revocation_list = self.fetch_revocation_list()
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1109, in fetch_revocation_list
> > return self.cms_verify(data['signed'])
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1038, in cms_verify
> > raise err
> > CalledProcessError: Command 'openssl' returned non-zero exit status 4
> > *2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
> > Marking token
> >
> *MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzY
> > ...
> >
> i4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
> > *as unauthorized in memcache*
> > *2013-04-30 20:00:42 WARNING [keystoneclient.middleware.auth_token]
> > Authorization failed for token*
> >
> MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzYjc4NmFiMTUxYzhmOGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTcifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuY
> > ...
> >
> dmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MSIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjgwODAvdjEvQVVUSF82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJpZCI6ICI2NTkxMTExNGMzNjM0MWExOTAwNmMzMjhjNmQwYTJhZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MS9BVVRIXzZhYTNiZjFhYjY4MDQwMjE4ODczYTc4MmY5MGNmZmE3In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm9iamVjdC1zdG9yZSIsICJuYW1lIjogInN3aWZ0In0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6MzUzNTcvdjIuMCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjUwMDAvdjIuMCIsICJpZCI6ICIwZjkzODlkMDQ4NWU0ZjJmOWY3ODc0YzQxMTgxYmQyOCIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pb
>
> iIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
> > *2013-04-30 20:00:42 INFO [keystoneclient.middleware.auth_token]
> > Invalid user token - rejecting request*
> > *
> > *
> > It seems that cinder can't recognise my auth_token so it tries to ban
> > it. Does anybody have any idea about this? Thanks!!!
> >
> >
> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
> > <mailto:jjpavlik at gmail.com>>
> >
> > I ran tcpdump on my cinder node (172.19.136.245) and this is what i
> saw:
> >
> > From 172.19.136.10 i ran "cinder --os-username=admin
> > --os-tenant-name=admin --os-password=zGp05Nsa
> > --os-auth-url=http://172.19.136.1:35357/v2.0 list":
> >
> > After getting a valid token from keystone.
> >
> > -----Request from cinder-client to cinder-api:
> >
> > GET /v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail HTTP/1.1
> > Host: 172.19.136.245:8776 <http://172.19.136.245:8776>
> > X-Auth-Project-Id: admin
> > Accept-Encoding: gzip, deflate, compress
> > Content-Length: 0
> > Accept: application/json
> > User-Agent: python-cinderclient
> > X-Auth-Token: MIIMbwYJKoZIhvcNAQcCoIIMY.....oiRM1nsw==
> >
> > -----Request from cinder-api to keystone:
> >
> > GET /v2.0/tokens/revoked HTTP/1.1
> > Host: 172.19.136.11:35357 <http://172.19.136.11:35357>
> > Accept-Encoding: identity
> > Content-type: application/json
> > Accept: application/json
> > X-Auth-Token:
> >
> MIIMKAYJKoZIhvcNAQcCoIIMGTCCDBUCAQExCTAHBgUrDgMCGjCCCwEGCS...eufVytyk=
> >
> > -----Answer from keystone to cinder-api:
> >
> > HTTP/1.1 200 OK
> > Vary: X-Auth-Token
> > Content-Type: application/json
> > Content-Length: 612
> > Date: Tue, 30 Apr 2013 19:55:04 GMT
> >
> > {"signed": "-----BEGIN
> >
> CMS-----\nMIIBkAYJKoZIhvcNAQcCoIIBgTCCAX0CAQExCTAHBgUrDgMCGjBrBgkqhkiG9w0B\nBwGgXgRceyJyZXZva2VkIjogW3siZXhwaXJlcyI6ICIyMDEzLTA0LTMwVDIwOjQy\nOjQ3WiIsICJpZCI6ICJhMDRhMjAwZGZlZTI2NjNkNDNjN2UyNzkzZTU3YWE1OCJ9\nXX0xgf8wgfwCAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVW5zZXQxDjAM\nBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEYMBYGA1UEAxMPd3d3LmV4YW1w\nbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIGAE4mgl+c2wGz0+71j\n5Am0KCI+lKHtYJppPtBvVDJ194J1hgMEMz7Yxlqtn1qMoJm3o5fCTl8pU3IszX/f\nb36zOZCrRXTCqgb32O7HfhPKT+N8kqZxMvtDTzv+3uQOC0xw7cAh+sNPgG1EHrL3\nIO8cMEUJqOkXjhwQPKXSqYVrwg4=\n-----END
> > CMS-----\n"}
> >
> >
> > -----Answer from cinder-api to cinder-client:
> >
> > HTTP/1.1 401 Unauthorized
> > Www-Authenticate: Keystone uri='http://172.19.136.11:35357'
> > Content-Length: 276
> > Content-Type: text/plain; charset=UTF-8
> > Date: Tue, 30 Apr 2013 19:55:04 GMT
> >
> > 401 Unauthorized
> >
> > This server could not verify that you are authorized to access the
> > document you requested. Either you supplied the wrong credentials
> > (e.g., bad password), or your browser does not understand how to
> > supply the credentials required.
> >
> > Authentication required
> >
> >
> > Is there any chance that cinder-api is breaking up my token??
> >
> >
> >
> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
> > <mailto:jjpavlik at gmail.com>>
> >
> > I can get valid credentials with this line:
> >
> > root at heladera:/etc/cinder# cinder --os-username=admin
> > --os-tenant-name=admin --os-password=XXX
> > --os-auth-url=http://172.19.136.1:35357/v2.0 credentials
> >
> +------------------+----------------------------------------------------------------------------------------+
> > | User Credentials |
> > Value |
> >
> +------------------+----------------------------------------------------------------------------------------+
> > | id |
> > 3f82673b5fe0411ab5fd8216bdb693c6 |
> > | name |
> > admin |
> > | roles | [{u'name': u'KeystoneServiceAdmin'},
> > {u'name': u'KeystoneAdmin'}, {u'name': u'admin'}] |
> > | roles_links |
> > [] |
> > | username |
> > admin |
> >
> +------------------+----------------------------------------------------------------------------------------+
> >
> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > | Token |
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Value
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > |
> >
> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > | expires |
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 2013-05-01T18:47:48Z
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > |
> > | id |
> > MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQEx...tcWW6xvpLgWsr3A== |
> > | issued_at |
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 2013-04-30T18:47:48.512440
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > |
> > | tenant |
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > {u'id':
> > u'6aa3bf1ab68040218873a782f90cffa7', u'enabled': True,
> > u'description': None, u'name': u'admin'}
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> |
> >
> +-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> >
> > So, it must be something that happens AFTER getting the
> > credentials, something involving the cinder api. I'm not sure
> > how the authentication process work but this is what i think:
> >
> > 1-cinder client request for an auth token
> > 2-keystone validates the credentials, creates the token and
> > sends it back to the client
> > 3-the cinder client uses the received token to connect against
> > the cinder api
> > 4-the cinder api validates the token against ¿keystone? Here is
> > where the problem might be.
> > 5-somehow the api can't validate the token and rejects me.
> >
> > I'm running out of ideas.
> >
> >
> >
> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
> > <mailto:jjpavlik at gmail.com>>
> >
> > When i try to list the volumes this is what i see in the
> > cinder api logs file:
> >
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Authenticating user
> token
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Removing headers from
> > request environment:
> >
> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
> > 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms]
> > Verify error: Verification failure
> >
> > 140606277047968:error:0407006A:rsa
> > routines:RSA_padding_check_PKCS1_type_1:block type is not
> > 01:rsa_pk1.c:100:
> > 140606277047968:error:04067072:rsa
> > routines:RSA_EAY_PUBLIC_DECRYPT:padding check
> > failed:rsa_eay.c:721:
> > 140606277047968:error:2E09A09E:CMS
> > routines:CMS_SignerInfo_verify_content:verification
> > failure:cms_sd.c:900:
> > 140606277047968:error:2E09D06D:CMS
> > routines:CMS_verify:content verify error:cms_smime.c:425:
> >
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Token validation
> failure.
> > Traceback (most recent call last):
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 688, in _validate_user_token
> > verified = self.verify_signed_token(user_token)
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1043, in verify_signed_token
> > if self.is_signed_token_revoked(signed_text):
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1007, in is_signed_token_revoked
> > revocation_list = self.token_revocation_list
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1079, in token_revocation_list
> > self.token_revocation_list = self.fetch_revocation_list()
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1109, in fetch_revocation_list
> > return self.cms_verify(data['signed'])
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1038, in cms_verify
> > raise err
> > CalledProcessError: Command 'openssl' returned non-zero exit
> > status 4
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Marking token
> > MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw== as unauthorized in memcache
> > 2013-04-30 17:43:07 WARNING
> > [keystoneclient.middleware.auth_token] Authorization failed
> > for token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw==
> > 2013-04-30 17:43:07 INFO
> > [keystoneclient.middleware.auth_token] Invalid user token -
> > rejecting request
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Authenticating user
> token
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Removing headers from
> > request environment:
> >
> X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
> > 2013-04-30 17:43:07 ERROR [keystoneclient.common.cms]
> > Verify error: Verification failure
> >
> > 140558031275680:error:0407006A:rsa
> > routines:RSA_padding_check_PKCS1_type_1:block type is not
> > 01:rsa_pk1.c:100:
> > 140558031275680:error:04067072:rsa
> > routines:RSA_EAY_PUBLIC_DECRYPT:padding check
> > failed:rsa_eay.c:721:
> > 140558031275680:error:2E09A09E:CMS
> > routines:CMS_SignerInfo_verify_content:verification
> > failure:cms_sd.c:900:
> > 140558031275680:error:2E09D06D:CMS
> > routines:CMS_verify:content verify error:cms_smime.c:425:
> >
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Token validation
> failure.
> > Traceback (most recent call last):
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 688, in _validate_user_token
> > verified = self.verify_signed_token(user_token)
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1043, in verify_signed_token
> > if self.is_signed_token_revoked(signed_text):
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1007, in is_signed_token_revoked
> > revocation_list = self.token_revocation_list
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1079, in token_revocation_list
> > self.token_revocation_list = self.fetch_revocation_list()
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1109, in fetch_revocation_list
> > return self.cms_verify(data['signed'])
> > File
> >
> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
> > line 1038, in cms_verify
> > raise err
> > CalledProcessError: Command 'openssl' returned non-zero exit
> > status 4
> > 2013-04-30 17:43:07 DEBUG
> > [keystoneclient.middleware.auth_token] Marking token
> > MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw== as unauthorized in
> memcache
> > 2013-04-30 17:43:07 WARNING
> > [keystoneclient.middleware.auth_token] Authorization failed
> > for token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw==
> > 2013-04-30 17:43:07 INFO
> > [keystoneclient.middleware.auth_token] Invalid user token -
> > rejecting request
> >
> > MAYBE... somehow HAproxy is changing something in the header
> > but i don't think so. This is the haproxy configuration for
> > the cinder API:
> >
> > listen nova-api-cinder 172.19.136.1:8776
> > <http://172.19.136.1:8776>
> > balance roundrobin
> > option tcplog
> > server heladera 172.19.136.245:8776
> > <http://172.19.136.245:8776> check
> >
> > I don't understand why is the Verification Failure, and why
> > i have openssl involve in my authentication, I didn't change
> > anything in the cinder api-paste.ini file, besides the
> > auth_host and service_host.
> >
> >
> > 2013/4/30 Juan José Pavlik Salles <jjpavlik at gmail.com
> > <mailto:jjpavlik at gmail.com>>
> >
> > Hi Jay, you are right, i'm trying to balance API calls
> > with HAProxy. I installed HAproxy on 172.19.136.1 and
> > configured all the openstack services to make the calls
> > to that IP, then i use HAproxy to redirect the API calls
> > to the real API servers (172.19.136.10 and
> > 172.19.136.11), this is my configuration:
> >
> > I've these 4 nodes:
> >
> > 172.19.136.245 <http://172.19.136.245>:
> > -Cinder
> >
> > 172.19.136.10 <http://172.19.136.10>:
> > -Keystone
> > -Glance (glance, api, registry)
> > -Nova (compute, scheduler, etc)
> >
> > 172.19.136.11 <http://172.19.136.11>:
> > -Keystone
> > -Glance (glance, api, registry)
> > -Nova (compute, scheduler, etc)
> >
> > 172.19.136.2 / 172.19.136.1 <http://172.19.136.1>:
> > -Quantum server
> > -RabbitMQ
> > -MySQL
> > -HAProxy (Listening on 172.19.136.1 for all the API
> > calls, and balancing them to either 172.19.136.10 or
> > 172.19.136.11, it also listens for cinder api calls and
> > redirects them to 172.19.136.245)
> >
> > I didn't change all the endpoints yet, but all of them
> > should redirect to 172.19.136.1, maybe that's the
> > problem. What do you think?
> >
> > This configuration might look odd or strange, but i'm
> > trying to build a redundant and scalable cloud (like in
> > this article
> >
> http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/
> ).
> > Thanks!!!
> >
> >
> > 2013/4/30 Jay Pipes <jaypipes at gmail.com
> > <mailto:jaypipes at gmail.com>>
> >
> > On 04/29/2013 04:56 PM, Juan José Pavlik Salles
> wrote:
> > > Hi, i have spent the last days trying to solve
> > this problem. I can't
> > > list my cinder volumes from my shell:
> > >
> > > root at locro:~# cinder --os-username=admin
> > --os-tenant-name=admin
> > > --os-password=XXX
> > --os-auth-url=http://172.19.136.1:35357/v2.0 --debug
> > list
> > >
> > > REQ: curl -i http://172.19.136.1:35357/v2.0/tokens
> > -X POST -H
> > > "Content-Type: application/json" -H "Accept:
> > application/json" -H
> > > "User-Agent: python-cinderclient" -d '{"auth":
> > {"tenantName": "admin",
> > > "passwordCredentials": {"username": "admin",
> > "password": "zGp05Nsa"}}}'
> > >
> > > RESP: [200] {'date': 'Mon, 29 Apr 2013 17:24:44
> > GMT', 'content-type':
> > > 'application/json', 'content-length': '7096',
> > 'vary': 'X-Auth-Token'}
> > > RESP BODY: {"access": {"token": {"issued_at":
> > > "2013-04-29T17:24:44.044013", "expires":
> > "2013-04-30T17:24:43Z", "id":
> > > "MIIMaQYJKoZIhvcNAQcC...", "tenant":
> > {"description": null, "enabled":
> > > true, "id": "6aa3bf1ab68040218873a782f90cffa7",
> > "name": "admin"}},
> > > "serviceCatalog": [{"endpoints": [{"adminURL":
> > >
> > "
> http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
> > > "region": "RegionOne", "internalURL":
> > >
> > "
> http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7",
> > "id":
> > > "26178391275a42cfa3b786ab151c8f8a", "publicURL":
> > >
> > "
> http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7"}],
> > > "endpoints_links": [], "type": "compute", "name":
> > "nova"}, {"endpoints":
> > > [{"adminURL": "http://172.19.136.11:9696/",
> > "region": "RegionOne",
> > > "internalURL": "http://172.19.136.11:9696/", "id":
> > > "1d0f394d83804ecaaa5ba708ccf0417b", "publicURL":
> > > "http://172.19.136.11:9696/"}], "endpoints_links":
> > [], "type":
> > > "network", "name": "quantum"}, {"endpoints":
> > [{"adminURL":
> > > "http://172.19.136.10:9292/v2", "region":
> > "RegionOne", "internalURL":
> > > "http://172.19.136.11:9292/v2", "id":
> > > "11f37a313bad47f28b846cb9b94d458c", "publicURL":
> > > "http://172.19.136.11:9292/v2"}],
> > "endpoints_links": [], "type":
> > > "image", "name": "glance"}, {"endpoints":
> > [{"adminURL":
> > >
> > "
> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
> > > "region": "RegionOne", "internalURL":
> > >
> > "
> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7",
> > "id":
> > > "1ebe70478edd45d087263a4dc457f03a", "publicURL":
> > >
> > "
> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7"}],
> > > "endpoints_links": [], "type": "volume", "name":
> > "cinder"},
> > > {"endpoints": [{"adminURL":
> > "http://172.19.136.11:8773/services/Admin",
> > > "region": "RegionOne", "internalURL":
> > > "http://172.19.136.10:8773/services/Cloud", "id":
> > > "4fd5bcbee3584c2b883b08f22f81de54", "publicURL":
> > > "http://172.19.136.10:8773/services/Cloud"}],
> > "endpoints_links": [],
> > > "type": "ec2", "name": "ec2"}, {"endpoints":
> > [{"adminURL":
> > > "http://172.19.136.10:8080/v1", "region":
> > "RegionOne", "internalURL":
> > >
> > "
> http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7",
> > > "id": "65911114c36341a19006c328c6d0a2ae",
> "publicURL":
> > >
> > "
> http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7"}],
> > > "endpoints_links": [], "type": "object-store",
> > "name": "swift"},
> > > {"endpoints": [{"adminURL":
> > "http://172.19.136.11:35357/v2.0", "region":
> > > "RegionOne", "internalURL":
> > "http://172.19.136.10:5000/v2.0", "id":
> > > "0f9389d0485e4f2f9f7874c41181bd28", "publicURL":
> > > "http://172.19.136.10:5000/v2.0"}],
> > "endpoints_links": [], "type":
> > > "identity", "name": "keystone"}], "user":
> > {"username": "admin",
> > > "roles_links": [], "id":
> > "3f82673b5fe0411ab5fd8216bdb693c6", "roles":
> > > [{"name": "KeystoneServiceAdmin"}, {"name":
> > "KeystoneAdmin"}, {"name":
> > > "admin"}], "name": "admin"}, "metadata":
> > {"is_admin": 0, "roles":
> > > ["6666fa99078a4f07a070e7e858c32f02",
> > "36bba9ef0178448c8a654b75feb3a0f4",
> > > "a25581dd3470460b91ecaa29eca7205c"]}}}
> > >
> > > REQ: curl -i
> > >
> >
> http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail
> > > -X GET -H "X-Auth-Project-Id: admin" -H
> "User-Agent:
> > > python-cinderclient" -H "Accept: application/json"
> > -H "X-Auth-Token:
> > > MIIMaQYJKoZIhvcNAQcCo..."
> > >
> > > RESP: [401] {'date': 'Mon, 29 Apr 2013 17:24:44
> > GMT', 'content-length':
> > > '276', 'content-type': 'text/plain;
> > charset=UTF-8', 'www-authenticate':
> > > "Keystone uri='http://172.19.136.1:35357'"}
> > > RESP BODY: 401 Unauthorized
> >
> > From the above, the authentication URI that you are
> > supplying to
> > cinderclient is http://172.19.136.1:35357, which is
> > not the same as what
> > is returned in the service catalog above, which has
> > the internalURL for
> > the identity endpoint as
> http://172.19.136.10:5000/v2.0.
> >
> > Is this intended?
> >
> > -jay
> >
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > <mailto:OpenStack-operators at lists.openstack.org>
> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> >
> >
> >
> > --
> > Pavlik Juan José
> >
> >
> >
> >
> > --
> > Pavlik Juan José
> >
> >
> >
> >
> > --
> > Pavlik Juan José
> >
> >
> >
> >
> > --
> > Pavlik Juan José
> >
> >
> >
> >
> > --
> > Pavlik Juan José
> >
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
--
Pavlik Juan José
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130503/67781a5e/attachment-0001.html>
More information about the OpenStack-operators
mailing list