Octavia: Could not retrieve certificate when create HTTPS listener using application credentials
Hi,
i try to create a Octavia HTTPS listener by using application credentials but get this error:
Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-088d6eb0-a285-4089-bc11-ff0c3097123e)
# openstack secret list +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-... | cert2 | 2019-07-19T13:42:21+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | opaque | cbc | None | | https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... | cert1 | 2019-07-19T13:42:12+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | opaque | cbc | None | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
# openstack loadbalancer listener create foo-lb1 \ --name foo-lb1-https-listener \ --protocol-port 443 \ --protocol TERMINATED_HTTPS \ --insert-headers X-Forwarded-For=true,X-Forwarded-Proto=true \ --default-tls-container=https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... \ --sni-container-refs https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...
--------------------------------
Starting new HTTPS connection (1): octavia.service.dev.example.com:443 https://octavia.service.dev.example.com:443 "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 779 RESP: [200] Connection: keep-alive Content-Length: 779 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:24 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 RESP BODY: {"loadbalancers": [{"provider": "amphora", "description": "", "admin_state_up": true, "pools": [{"id": "169722d1-0a73-4283-bb42-aee5b662e2e2"}], "created_at": "2019-07-19T13:34:52", "provisioning_status": "ACTIVE", "updated_at": "2019-07-19T13:39:34", "vip_qos_policy_id": null, "vip_network_id": "2064c61c-64a1-466f-983a-af435ae1d51c", "listeners": [{"id": "169a91f9-ef5c-4d38-8449-e24b64cf082d"}], "tenant_id": "9646533a8d834978a868e81c9b9a39cf", "vip_port_id": "dcfc6e44-4092-4f2b-bd50-24e02abb078f", "flavor_id": "", "vip_address": "10.0.1.4", "vip_subnet_id": "787035dc-add4-4227-844a-1cf803625abc", "project_id": "9646533a8d834978a868e81c9b9a39cf", "id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7", "operating_status": "OFFLINE", "name": "foo-lb1"}], "loadbalancers_links": []} GET call to https://octavia.service.dev.example.com/v2.0/lbaas/loadbalancers used request id req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 REQ: curl -g -i -X POST https://octavia.service.dev.example.com/v2.0/lbaas/listeners -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.19.0 keystoneauth1/3.11.1 python-requests/2.20.1 CPython/2.7.15+" -H "X-Auth-Token: {SHA256}6414e14f4e78940902b11c89567689e3cc0d3ea62227b87a1e19361685c83584" -d '{"listener": {"insert_headers": {"X-Forwarded-For": "true", "X-Forwarded-Proto": "true"}, "protocol": "TERMINATED_HTTPS", "name": "foo-lb1-https-listener", "default_tls_container_ref": "https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...", "sni_container_refs": ["https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...", "https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-..."], "admin_state_up": true, "protocol_port": 443, "loadbalancer_id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7"}}' https://octavia.service.dev.example.com:443 "POST /v2.0/lbaas/listeners HTTP/1.1" 400 357 RESP: [400] Connection: keep-alive Content-Length: 357 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:27 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca RESP BODY: {"debuginfo": null, "faultcode": "Client", "faultstring": "Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... POST call to https://octavia.service.dev.example.com/v2.0/lbaas/listeners used request id req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca Request returned failure status: 400 Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action json=body) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) clean_up CreateListener: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last): File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run result = self.run_subcommand(remainder) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action json=body) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) ------------------------------
This issue occurs only when application credentials are used. Creation of HTTP listener with applications credentials works fine, also creation of HTTPS listener when user are authenticated by user / password.
Does somebody know which additional ACLs / permissions are required to fix this?
BR
Pawel
Hi Pawel,
First question is what version of Octavia are you using? Older versions required you to set some ACL permissions on the secrets in Barbican. You can check this by reviewing the load balancing cookbook for the version of Octavia you are running [1]. There is a drop down in the upper right corner of the document that allows you to select a version of the document.
Second question, can you expand on what you mean by "application credentials"? Is this by using a pre-created token instead of having the username/password in your environment?
Third, can you check your octavia.conf settings[2]? Check the following options are either the default (commented out) or set to the same settings as the default. [certificates] cert_manager = barbican_cert_manager barbican_auth = barbican_acl_auth (Note, this is only valid in the newer versions of Octavia as noted above)
Fourth (last one), can you provide the associated log output from the Octavia API process that is handling this request? Debug if you can.
Michael
[1] https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html#de... [2] https://docs.openstack.org/octavia/latest/configuration/configref.html#certi...
On Fri, Jul 19, 2019 at 7:51 AM Pawel Konczalski pawel.konczalski@everyware.ch wrote:
Hi,
i try to create a Octavia HTTPS listener by using application credentials but get this error:
Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-088d6eb0-a285-4089-bc11-ff0c3097123e)
# openstack secret list +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-... | cert2 | 2019-07-19T13:42:21+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | opaque | cbc | None | | https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... | cert1 | 2019-07-19T13:42:12+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | opaque | cbc | None | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
# openstack loadbalancer listener create foo-lb1 \ --name foo-lb1-https-listener \ --protocol-port 443 \ --protocol TERMINATED_HTTPS \ --insert-headers X-Forwarded-For=true,X-Forwarded-Proto=true \ --default-tls-container=https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... \ --sni-container-refs https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...
Starting new HTTPS connection (1): octavia.service.dev.example.com:443 https://octavia.service.dev.example.com:443 "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 779 RESP: [200] Connection: keep-alive Content-Length: 779 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:24 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 RESP BODY: {"loadbalancers": [{"provider": "amphora", "description": "", "admin_state_up": true, "pools": [{"id": "169722d1-0a73-4283-bb42-aee5b662e2e2"}], "created_at": "2019-07-19T13:34:52", "provisioning_status": "ACTIVE", "updated_at": "2019-07-19T13:39:34", "vip_qos_policy_id": null, "vip_network_id": "2064c61c-64a1-466f-983a-af435ae1d51c", "listeners": [{"id": "169a91f9-ef5c-4d38-8449-e24b64cf082d"}], "tenant_id": "9646533a8d834978a868e81c9b9a39cf", "vip_port_id": "dcfc6e44-4092-4f2b-bd50-24e02abb078f", "flavor_id": "", "vip_address": "10.0.1.4", "vip_subnet_id": "787035dc-add4-4227-844a-1cf803625abc", "project_id": "9646533a8d834978a868e81c9b9a39cf", "id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7", "operating_status": "OFFLINE", "name": "foo-lb1"}], "loadbalancers_links": []} GET call to https://octavia.service.dev.example.com/v2.0/lbaas/loadbalancers used request id req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 REQ: curl -g -i -X POST https://octavia.service.dev.example.com/v2.0/lbaas/listeners -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.19.0 keystoneauth1/3.11.1 python-requests/2.20.1 CPython/2.7.15+" -H "X-Auth-Token: {SHA256}6414e14f4e78940902b11c89567689e3cc0d3ea62227b87a1e19361685c83584" -d '{"listener": {"insert_headers": {"X-Forwarded-For": "true", "X-Forwarded-Proto": "true"}, "protocol": "TERMINATED_HTTPS", "name": "foo-lb1-https-listener", "default_tls_container_ref": "https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...", "sni_container_refs": ["https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...", "https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-..."], "admin_state_up": true, "protocol_port": 443, "loadbalancer_id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7"}}' https://octavia.service.dev.example.com:443 "POST /v2.0/lbaas/listeners HTTP/1.1" 400 357 RESP: [400] Connection: keep-alive Content-Length: 357 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:27 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca RESP BODY: {"debuginfo": null, "faultcode": "Client", "faultstring": "Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-... POST call to https://octavia.service.dev.example.com/v2.0/lbaas/listeners used request id req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca Request returned failure status: 400 Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action json=body) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) clean_up CreateListener: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last): File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run result = self.run_subcommand(remainder) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action json=body) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-...', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-...'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca)
This issue occurs only when application credentials are used. Creation of HTTP listener with applications credentials works fine, also creation of HTTPS listener when user are authenticated by user / password.
Does somebody know which additional ACLs / permissions are required to fix this?
BR
Pawel
participants (2)
-
Michael Johnson
-
Pawel Konczalski