<div dir="ltr">When i try to list the volumes this is what i see in the cinder api logs file:<div><br></div><div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Authenticating user token</div><div>
2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role</div>
<div>2013-04-30 17:43:07 ERROR [keystoneclient.common.cms] Verify error: Verification failure</div><div><br></div><div>140606277047968:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:</div>
<div>140606277047968:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:</div><div>140606277047968:error:2E09A09E:CMS routines:CMS_SignerInfo_verify_content:verification failure:cms_sd.c:900:</div>
<div>140606277047968:error:2E09D06D:CMS routines:CMS_verify:content verify error:cms_smime.c:425:</div><div><br></div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Token validation failure.</div>
<div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 688, in _validate_user_token</div><div> verified = self.verify_signed_token(user_token)</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1043, in verify_signed_token</div><div> if self.is_signed_token_revoked(signed_text):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1007, in is_signed_token_revoked</div>
<div> revocation_list = self.token_revocation_list</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1079, in token_revocation_list</div><div> self.token_revocation_list = self.fetch_revocation_list()</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1109, in fetch_revocation_list</div><div> return self.cms_verify(data['signed'])</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1038, in cms_verify</div>
<div> raise err</div><div>CalledProcessError: Command 'openssl' returned non-zero exit status 4</div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Marking token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw== as unauthorized in memcache</div>
<div>2013-04-30 17:43:07 WARNING [keystoneclient.middleware.auth_token] Authorization failed for token MIIMbwYJKoZIhvcNA ... Od7Wrw6Aw==</div><div>2013-04-30 17:43:07 INFO [keystoneclient.middleware.auth_token] Invalid user token - rejecting request</div>
<div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Authenticating user token</div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role</div>
<div>2013-04-30 17:43:07 ERROR [keystoneclient.common.cms] Verify error: Verification failure</div><div><br></div><div>140558031275680:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:</div>
<div>140558031275680:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:</div><div>140558031275680:error:2E09A09E:CMS routines:CMS_SignerInfo_verify_content:verification failure:cms_sd.c:900:</div>
<div>140558031275680:error:2E09D06D:CMS routines:CMS_verify:content verify error:cms_smime.c:425:</div><div><br></div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Token validation failure.</div>
<div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 688, in _validate_user_token</div><div> verified = self.verify_signed_token(user_token)</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1043, in verify_signed_token</div><div> if self.is_signed_token_revoked(signed_text):</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1007, in is_signed_token_revoked</div>
<div> revocation_list = self.token_revocation_list</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1079, in token_revocation_list</div><div> self.token_revocation_list = self.fetch_revocation_list()</div>
<div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1109, in fetch_revocation_list</div><div> return self.cms_verify(data['signed'])</div><div> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1038, in cms_verify</div>
<div> raise err</div><div>CalledProcessError: Command 'openssl' returned non-zero exit status 4</div><div>2013-04-30 17:43:07 DEBUG [keystoneclient.middleware.auth_token] Marking token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw== as unauthorized in memcache</div>
<div>2013-04-30 17:43:07 WARNING [keystoneclient.middleware.auth_token] Authorization failed for token MIIMbwYJKoZIhvcNA ... YAUt8D2KYQw==</div><div>2013-04-30 17:43:07 INFO [keystoneclient.middleware.auth_token] Invalid user token - rejecting request</div>
</div><div><br></div><div style>MAYBE... somehow HAproxy is changing something in the header but i don't think so. This is the haproxy configuration for the cinder API:</div><div style><br></div><div style><div>listen nova-api-cinder <a href="http://172.19.136.1:8776">172.19.136.1:8776</a></div>
<div> balance roundrobin</div><div> option tcplog</div><div> server heladera <a href="http://172.19.136.245:8776">172.19.136.245:8776</a> check</div><div><br></div><div style>I don't understand why is the Verification Failure, and why i have openssl involve in my authentication, I didn't change anything in the cinder api-paste.ini file, besides the auth_host and service_host. </div>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/30 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Jay, you are right, i'm trying to balance API calls with HAProxy. I installed HAproxy on 172.19.136.1 and configured all the openstack services to make the calls to that IP, then i use HAproxy to redirect the API calls to the real API servers (172.19.136.10 and 172.19.136.11), this is my configuration:<div>
<br></div><div>I've these 4 nodes: </div><div><br></div><div><a href="http://172.19.136.245" target="_blank">172.19.136.245</a>:</div><div>-Cinder<br><div><br></div><div><a href="http://172.19.136.10" target="_blank">172.19.136.10</a>:</div>
<div>
-Keystone</div><div>-Glance (glance, api, registry)</div><div>-Nova (compute, scheduler, etc)</div><div><br></div><div><a href="http://172.19.136.11" target="_blank">172.19.136.11</a>:</div><div><div>-Keystone</div>
<div>-Glance (glance, api, registry)</div><div>-Nova (compute, scheduler, etc)</div><div><br></div><div>172.19.136.2 / <a href="http://172.19.136.1" target="_blank">172.19.136.1</a>:</div><div>-Quantum server</div><div>-RabbitMQ</div>
<div>-MySQL</div><div>-HAProxy (Listening on 172.19.136.1 for all the API calls, and balancing them to either 172.19.136.10 or 172.19.136.11, it also listens for cinder api calls and redirects them to 172.19.136.245)</div>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">I didn't change all the endpoints yet, but all of them should redirect to 172.19.136.1, maybe that's the problem. What do you think? </div><div class="gmail_extra">
<br></div><div class="gmail_extra">This configuration might look odd or strange, but i'm trying to build a redundant and scalable cloud (like in this article <a href="http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/" target="_blank">http://www.mirantis.com/blog/software-high-availability-load-balancing-openstack-cloud-api-servic/</a>). Thanks!!!<div>
<div class="h5"><br>
<br><div class="gmail_quote">2013/4/30 Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div>On 04/29/2013 04:56 PM, Juan José Pavlik Salles wrote:<br>
> Hi, i have spent the last days trying to solve this problem. I can't<br>
> list my cinder volumes from my shell:<br>
><br>
> root@locro:~# cinder --os-username=admin --os-tenant-name=admin<br>
> --os-password=XXX --os-auth-url=<a href="http://172.19.136.1:35357/v2.0" target="_blank">http://172.19.136.1:35357/v2.0</a> --debug list<br>
><br>
> REQ: curl -i <a href="http://172.19.136.1:35357/v2.0/tokens" target="_blank">http://172.19.136.1:35357/v2.0/tokens</a> -X POST -H<br>
> "Content-Type: application/json" -H "Accept: application/json" -H<br>
> "User-Agent: python-cinderclient" -d '{"auth": {"tenantName": "admin",<br>
> "passwordCredentials": {"username": "admin", "password": "zGp05Nsa"}}}'<br>
><br>
> RESP: [200] {'date': 'Mon, 29 Apr 2013 17:24:44 GMT', 'content-type':<br>
> 'application/json', 'content-length': '7096', 'vary': 'X-Auth-Token'}<br>
> RESP BODY: {"access": {"token": {"issued_at":<br>
> "2013-04-29T17:24:44.044013", "expires": "2013-04-30T17:24:43Z", "id":<br>
> "MIIMaQYJKoZIhvcNAQcC...", "tenant": {"description": null, "enabled":<br>
> true, "id": "6aa3bf1ab68040218873a782f90cffa7", "name": "admin"}},<br>
> "serviceCatalog": [{"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7</a>",<br>
> "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.10:8774/v2/6aa3bf1ab68040218873a782f90cffa7</a>", "id":<br>
> "26178391275a42cfa3b786ab151c8f8a", "publicURL":<br>
> "<a href="http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.11:8774/v2/6aa3bf1ab68040218873a782f90cffa7</a>"}],<br>
> "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints":<br>
> [{"adminURL": "<a href="http://172.19.136.11:9696/" target="_blank">http://172.19.136.11:9696/</a>", "region": "RegionOne",<br>
> "internalURL": "<a href="http://172.19.136.11:9696/" target="_blank">http://172.19.136.11:9696/</a>", "id":<br>
> "1d0f394d83804ecaaa5ba708ccf0417b", "publicURL":<br>
> "<a href="http://172.19.136.11:9696/" target="_blank">http://172.19.136.11:9696/</a>"}], "endpoints_links": [], "type":<br>
> "network", "name": "quantum"}, {"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.10:9292/v2" target="_blank">http://172.19.136.10:9292/v2</a>", "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.11:9292/v2" target="_blank">http://172.19.136.11:9292/v2</a>", "id":<br>
> "11f37a313bad47f28b846cb9b94d458c", "publicURL":<br>
> "<a href="http://172.19.136.11:9292/v2" target="_blank">http://172.19.136.11:9292/v2</a>"}], "endpoints_links": [], "type":<br>
> "image", "name": "glance"}, {"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7</a>",<br>
> "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7</a>", "id":<br>
> "1ebe70478edd45d087263a4dc457f03a", "publicURL":<br>
> "<a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7</a>"}],<br>
> "endpoints_links": [], "type": "volume", "name": "cinder"},<br>
> {"endpoints": [{"adminURL": "<a href="http://172.19.136.11:8773/services/Admin" target="_blank">http://172.19.136.11:8773/services/Admin</a>",<br>
> "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.10:8773/services/Cloud" target="_blank">http://172.19.136.10:8773/services/Cloud</a>", "id":<br>
> "4fd5bcbee3584c2b883b08f22f81de54", "publicURL":<br>
> "<a href="http://172.19.136.10:8773/services/Cloud" target="_blank">http://172.19.136.10:8773/services/Cloud</a>"}], "endpoints_links": [],<br>
> "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL":<br>
> "<a href="http://172.19.136.10:8080/v1" target="_blank">http://172.19.136.10:8080/v1</a>", "region": "RegionOne", "internalURL":<br>
> "<a href="http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.11:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7</a>",<br>
> "id": "65911114c36341a19006c328c6d0a2ae", "publicURL":<br>
> "<a href="http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7" target="_blank">http://172.19.136.10:8080/v1/AUTH_6aa3bf1ab68040218873a782f90cffa7</a>"}],<br>
> "endpoints_links": [], "type": "object-store", "name": "swift"},<br>
> {"endpoints": [{"adminURL": "<a href="http://172.19.136.11:35357/v2.0" target="_blank">http://172.19.136.11:35357/v2.0</a>", "region":<br>
> "RegionOne", "internalURL": "<a href="http://172.19.136.10:5000/v2.0" target="_blank">http://172.19.136.10:5000/v2.0</a>", "id":<br>
> "0f9389d0485e4f2f9f7874c41181bd28", "publicURL":<br>
> "<a href="http://172.19.136.10:5000/v2.0" target="_blank">http://172.19.136.10:5000/v2.0</a>"}], "endpoints_links": [], "type":<br>
> "identity", "name": "keystone"}], "user": {"username": "admin",<br>
> "roles_links": [], "id": "3f82673b5fe0411ab5fd8216bdb693c6", "roles":<br>
> [{"name": "KeystoneServiceAdmin"}, {"name": "KeystoneAdmin"}, {"name":<br>
> "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles":<br>
> ["6666fa99078a4f07a070e7e858c32f02", "36bba9ef0178448c8a654b75feb3a0f4",<br>
> "a25581dd3470460b91ecaa29eca7205c"]}}}<br>
><br>
> REQ: curl -i<br>
> <a href="http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail" target="_blank">http://172.19.136.1:8776/v1/6aa3bf1ab68040218873a782f90cffa7/volumes/detail</a><br>
> -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent:<br>
> python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token:<br>
> MIIMaQYJKoZIhvcNAQcCo..."<br>
><br>
> RESP: [401] {'date': 'Mon, 29 Apr 2013 17:24:44 GMT', 'content-length':<br>
> '276', 'content-type': 'text/plain; charset=UTF-8', 'www-authenticate':<br>
> "Keystone uri='<a href="http://172.19.136.1:35357" target="_blank">http://172.19.136.1:35357</a>'"}<br>
> RESP BODY: 401 Unauthorized<br>
<br>
</div></div>From the above, the authentication URI that you are supplying to<br>
cinderclient is <a href="http://172.19.136.1:35357" target="_blank">http://172.19.136.1:35357</a>, which is not the same as what<br>
is returned in the service catalog above, which has the internalURL for<br>
the identity endpoint as <a href="http://172.19.136.10:5000/v2.0" target="_blank">http://172.19.136.10:5000/v2.0</a>.<br>
<br>
Is this intended?<br>
<br>
-jay<br>
<br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br>Pavlik Juan José
</font></span></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Pavlik Juan José
</div>