Open Stack

Keystone signs the information in auth token with a certificate that in
most setups was generated for that instance of keystone. Swift will use
auth_token middleware to fetch the certificates of keystone so that it
can verify that the tokens are correct. 

My guess is that the two keystone instances are using different
certificates and you are trying to validate a token with the other
keystone instance (other certificates) and it won't work.

If you are using the same keystone instance then it is possible that the
auth_token middleware in swift has cached the certificates for the other
keystone instance, so even though you have updated the values in swift
it is using the old certificates.

Try deleting the certificates found in the folder specified by
signing_dir in the swift setup and make sure you are issuing the tokens
from the keystone instance you are validating them against.


Jamie

On Wed, 2013-10-30 at 18:47 +0100, thorfinn at poivron.org wrote:
> Hi all.
> 
> * Hypervisor 1 : 192.168.1.120
> 	- Keystone 1	: 192.168.3.141
> 	- Swift-proxy 1	: 192.168.3.111
> * Hypervisor 2 : 192.168.1.122
> 	- Keystone 2	: 192.168.3.241
> 	- Swift-proxy 2	: 192.168.3.211
> 
> Keystone servers have the same mysql server, database and 
> configuration, so it's not a data issue.
> Every server can ping and talk to all the other ones.
> 
> When I talk to Swift-proxy 1, connected to Keystone 1 it works.
> Same to Swift-proxy 2, connected to Keystone 2.
> 
> If I connect Swift-proxy 1 to Keystone 2, it doesn't work anymore.
> Same for Swift-proxy 2 to Keystone 1.
> 
> All the servers are using Ubuntu 12.04.3/Havana and are up-to-date.
> 
> When it works, I have this (keystone 2 connected to swift-proxy 2) :
> # swift -V 2 -v -A http://192.168.3.241:5000/v2.0 -U service:swift -K 
> swift stat
> StorageURL: 
> http://192.168.3.211:8080/v1/AUTH_5becb4a93e7f498bbe83534f4481dc0d
> Auth Token: 
> MIIGTQYJKoZIhvcNAQcCoIIGPjCCBjoCAQExCTAHBgUrDgMCGjCCBKMGCSqGSIb3DQEHAaCCBJQEggSQeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMC0zMFQxNzozMzowMC4zNTI4MzAiLCAiZXhwaXJlcyI6ICIyMDEzLTEwLTMxVDE3OjMzOjAwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIlNlcnZpY2UgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNWJlY2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAibmFtZSI6ICJzZXJ2aWNlIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjIxMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjIxMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAiaWQiOiAiMzdhZWQxODc3NmQ5NDUzZGI3MmE3ODc1ZWM3ZTY5ZGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjExOjgwODAvdjEvQVVUSF81YmVjYjRhOTNlN2Y0OThiYmU4MzUzNGY0NDgxZGMwZCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJvYmplY3Qtc3RvcmUiLCAibmFtZSI6ICJzd2lmdCJ9LCB7ImVuZHBvaW50cyI6IFt7ImF!
 kbW
>  luVVJMIj
> ogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjM1MzU3L3YyLjAiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjI0MTo1MDAwL3YyLjAiLCAiaWQiOiAiMGJmNzIxNjM1MmFjNDE4ZmEzODVkNWZmM2ZmODlmMzAiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidXNlcm5hbWUiOiAic3dpZnQiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogIjYzZWVjNjMyYWI3NTRiYzY5NTgzY2M0YTI5Yjc0MDVmIiwgInJvbGVzIjogW3sibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAic3dpZnQifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiZjUwNjczYjliNDUwNDQyZGI1OGRkYTExYjQ4M2ZkMmUiXX19fTGCAYEwggF9AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASCAQASAXjP5gHiUWfl0e8gfh2O4n7IEoerLDmmTR07tQw1ibqyxyyhdNAVuThrceu0z3-VrHyeiuYcWAlJZzI6okeo02CDc7SRK4CjHLG9m8q80UrLOfT1-PbKes16ULKbGJpsBYykVXTV8ts+wQVAYcS73f2bwp6+Ki0Cygtfqkmvq!
 Net
>  7hDtSsvT
> yAGKiLo0TbOYOpF96NelPwuzGAm2y-bcOhCCdJKo8iFEotTXK0SQzUQ78r3Mh1fsd6asoHRZxKKc0oXWm3KgJy1X-isnqopCMutDPPQCAXABOFb-OSovLMmmOS8ZZbII7RTd1e1z1sFYv3d67b0oc2A4e8DWAaVj
>     Account: AUTH_5becb4a93e7f498bbe83534f4481dc0d
> Containers: 4
>     Objects: 11
>       Bytes: 158989835
> Accept-Ranges: bytes
> X-Timestamp: 1382628587.87452
> Content-Type: text/plain; charset=utf-8
> 
> Oct 30 18:32:59 dev-api-002 proxy-server Verify error: Command 
> 'openssl' returned non-zero exit status 4
> Oct 30 18:32:59 dev-api-002 proxy-server Authorization failed for token 
> MIIGTQYJKoZIhvcNAQcCoIIGPjCCBjoCAQExCTAHBgUrDgMCGjCCBKMGCSqGSIb3DQEHAaCCBJQEggSQeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMj
> AxMy0xMC0zMFQxNzozMjo1OC44NTY3MzEiLCAiZXhwaXJlcyI6ICIyMDEzLTEwLTMxVDE3OjMyOjU4WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIlNlcnZpY2UgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNWJl
> Y2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAibmFtZSI6ICJzZXJ2aWNlIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjIxMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlOD
> M1MzRmNDQ4MWRjMGQiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjIxMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAiaWQiOiAiMzdhZWQxODc3NmQ5NDUzZGI3MmE3ODc1
> ZWM3ZTY5ZGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjExOjgwODAvdjEvQVVUSF81YmVjYjRhOTNlN2Y0OThiYmU4MzUzNGY0NDgxZGMwZCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJvYmplY3Qtc3RvcmUiLCAibmFtZSI6ICJzd2
> lmdCJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjM1MzU3L3YyLjAiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjI0MTo1MDAwL3YyLjAiLCAiaWQiOiAiMGJmNzIx
> NjM1MmFjNDE4ZmEzODVkNWZmM2ZmODlmMzAiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidX
> Nlcm5hbWUiOiAic3dpZnQiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogIjYzZWVjNjMyYWI3NTRiYzY5NTgzY2M0YTI5Yjc0MDVmIiwgInJvbGVzIjogW3sibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAic3dpZnQifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAs
> ICJyb2xlcyI6IFsiZjUwNjczYjliNDUwNDQyZGI1OGRkYTExYjQ4M2ZkMmUiXX19fTGCAYEwggF9AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBAT
> AHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASCAQCzitgoJ4ltBsCNN8xnLy3GlopgV5OlVRBa4fbHXcNT6expAdTYtx4I8q1cIF279NPVJO9T8hsedMSHwOxZvxJKskwFuuwUWT+cTBzkxlrY11Njmg9dGwQiJ1Pbb8oA3YZcgWjz6aY+1RajN-Lq9ugCidsY5tzFrHTwPed1VOcu
> Wq2MKcMIqmt2m5b
> Oct 30 18:32:59 dev-api-002 proxy-server Invalid user token - deferring 
> reject downstream
> 
> Why the error if it works ?
> 
> When it doesn't work, I have this (keystone 2 connected to swift-proxy 
> 1) :
> # swift -V 2 -v -A http://192.168.3.241:5000/v2.0 -U service:swift -K 
> swift stat
> Account HEAD failed: 
> http://192.168.3.111:8080/v1/AUTH_5becb4a93e7f498bbe83534f4481dc0d 401 
> Unauthorized
> 
> Oct 30 18:34:53 dev-api-001 proxy-server Verify error: Command 
> 'openssl' returned non-zero exit status 4
> Oct 30 18:34:53 dev-api-001 proxy-server Authorization failed for token 
> MIIGTQYJKoZIhvcNAQcCoIIGPjCCBjoCAQExCTAHBgUrDgMCGjCCBKMGCSqGSIb3DQEHAaCCBJQEggSQeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMj
> AxMy0xMC0zMFQxNzozNDo1My42NTY0NTMiLCAiZXhwaXJlcyI6ICIyMDEzLTEwLTMxVDE3OjM0OjUzWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIlNlcnZpY2UgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNWJl
> Y2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAibmFtZSI6ICJzZXJ2aWNlIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjExMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlOD
> M1MzRmNDQ4MWRjMGQiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjExMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAiaWQiOiAiMzdhZWQxODc3NmQ5NDUzZGI3MmE3ODc1
> ZWM3ZTY5ZGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMTExOjgwODAvdjEvQVVUSF81YmVjYjRhOTNlN2Y0OThiYmU4MzUzNGY0NDgxZGMwZCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJvYmplY3Qtc3RvcmUiLCAibmFtZSI6ICJzd2
> lmdCJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjM1MzU3L3YyLjAiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjI0MTo1MDAwL3YyLjAiLCAiaWQiOiAiMGJmNzIx
> NjM1MmFjNDE4ZmEzODVkNWZmM2ZmODlmMzAiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidX
> Nlcm5hbWUiOiAic3dpZnQiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogIjYzZWVjNjMyYWI3NTRiYzY5NTgzY2M0YTI5Yjc0MDVmIiwgInJvbGVzIjogW3sibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAic3dpZnQifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAs
> ICJyb2xlcyI6IFsiZjUwNjczYjliNDUwNDQyZGI1OGRkYTExYjQ4M2ZkMmUiXX19fTGCAYEwggF9AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBAT
> AHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASCAQAZRHKSulq-73euRy9HrYxMTU-HtRizyySVYkoo3CTCOgxFZz3CzelBIcp6HJySC6DVAW4Uz-xcTmtp1hju3vx3yAVstWtCczO-YZX4bUy4XFmfNje2ydJl5M2sSAUZ8160Vn3QnajesaRIvnu9w8WcpWsmaYjbx15ou2CzWnvH
> j0V1lLTgA28dh90
> Oct 30 18:34:53 dev-api-001 proxy-server Invalid user token - deferring 
> reject downstream
> Oct 30 18:34:55 dev-api-001 proxy-server Verify error: Command 
> 'openssl' returned non-zero exit status 4
> Oct 30 18:34:55 dev-api-001 proxy-server Authorization failed for token 
> MIIGTQYJKoZIhvcNAQcCoIIGPjCCBjoCAQExCTAHBgUrDgMCGjCCBKMGCSqGSIb3DQEHAaCCBJQEggSQeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMj
> AxMy0xMC0zMFQxNzozNDo1NS4xNTA5MjUiLCAiZXhwaXJlcyI6ICIyMDEzLTEwLTMxVDE3OjM0OjU1WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogIlNlcnZpY2UgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNWJl
> Y2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAibmFtZSI6ICJzZXJ2aWNlIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjExMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlOD
> M1MzRmNDQ4MWRjMGQiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjExMTo4MDgwL3YxL0FVVEhfNWJlY2I0YTkzZTdmNDk4YmJlODM1MzRmNDQ4MWRjMGQiLCAiaWQiOiAiMzdhZWQxODc3NmQ5NDUzZGI3MmE3ODc1
> ZWM3ZTY5ZGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMTExOjgwODAvdjEvQVVUSF81YmVjYjRhOTNlN2Y0OThiYmU4MzUzNGY0NDgxZGMwZCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJvYmplY3Qtc3RvcmUiLCAibmFtZSI6ICJzd2
> lmdCJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjM1MzU3L3YyLjAiLCAicmVnaW9uIjogInJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTkyLjE2OC4zLjI0MTo1MDAwL3YyLjAiLCAiaWQiOiAiMGJmNzIx
> NjM1MmFjNDE4ZmEzODVkNWZmM2ZmODlmMzAiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY4LjMuMjQxOjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidX
> Nlcm5hbWUiOiAic3dpZnQiLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogIjYzZWVjNjMyYWI3NTRiYzY5NTgzY2M0YTI5Yjc0MDVmIiwgInJvbGVzIjogW3sibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAic3dpZnQifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAs
> ICJyb2xlcyI6IFsiZjUwNjczYjliNDUwNDQyZGI1OGRkYTExYjQ4M2ZkMmUiXX19fTGCAYEwggF9AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBAT
> AHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASCAQCXUDhH+Q8xUS6upUb8TtF2Uk-h2m-w84CmeJxKc-n7qGuozrZe7KPcrKp002ojDIY+CmGulWtXQD-IJ6V4hcjaVbmoxMIIVmMulxt1G2dLLIrtQCIUwnNFsOaaBiEZTus8DlpjIHGrLfcBRtzjewQXUA5PuRXC-ebtgE7wphMv
> ETodRWB5zKixqmL
> Oct 30 18:34:55 dev-api-001 proxy-server Invalid user token - deferring 
> reject downstream
> 
> I have the same kind of logs entry than the working example but twice.
> 
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack