Open Stack

Hello Magnum users and devs,

I was wondering if any may be able to provide some clues on a mystifying
problem we've been seeing with some Magnum-deployed k8s clusters, of older
deployment date (v1.17, deployed ~6 months ago) and Keystone trusts used to
manage OpenStack cloud resources.

I've attached a copy of the Magnum template for the k8s cluster to give an
idea of its initial environment.

The issue we've been seeing is that the Keystone trust generated at Magnum
cluster creation time seems no longer be usable to successfully
authenticate to the OpenStack APIs, and so operations such as creation of
Manila shares used as k8s Persistent Volume Claims then get stuck in
Pending state, forever.

The strange thing is that this doesn't happen for newly created Magnum
clusters of the same template (in attached file, I use trust credentials
for a newly deployed and old cluster; OpenStack API calls for new
credentials succeed, whereas old cluster trust fails).

Unfortunately following debug-level Keystone logging for rejected trust
auth attempts hasn't led me much further to understanding the root issue;
the Magnum generated trust exists and has no expiry, but it still seems to
be rejected. Keystone debug logging doesn't seem to give much indication as
to *why* it is being rejected or what policy violations may be involved,
but perhaps there's details hidden in there I'm not seeing yet.

Does anyone have any clues as to why this may be happening, or any advice
on how we may be able to replace or refresh the trust that the
Magnum-created k8s cluster is expecting to use in calling OpenStack APIs?

Many thanks,
Paul

-- 
*******************
Paul Browne
Research Computing Platforms
University Information Services
Roger Needham Building
JJ Thompson Avenue
University of Cambridge
Cambridge
United Kingdom
E-Mail: pfb29 at cam.ac.uk
Tel: 0044-1223-746548
*******************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20201030/b5949caf/attachment-0001.html>
-------------- next part --------------
[root at clc-er13-u38 ~]# grep b47f80eb-93dc-4099-afe7-8ac60dc58bd4 /var/log/kolla/* -R

/var/log/kolla/keystone/keystone.log:2020-10-30 14:15:21.113 23 INFO oslo.messaging.notification.identity.authenticate [req-b47f80eb-93dc-4099-afe7-8ac60dc58bd4 - - - - -] {"event_type": "identity.authenticate", "timestamp": "2020-10-30 14:15:21.113010", "payload": {"typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "initiator": {"username": "563cea4f-230a-4932-96d5-5933906419d6_fd39e5b6c53d48a281c2988e83bfee8f", "typeURI": "service/security/account/user", "user_id": "fe2f78fa308a48e19b065b818cec7827", "host": {"agent": "openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.23.0 CPython/3.6.9", "address": "X.X.X.X"}, "request_id": "req-b47f80eb-93dc-4099-afe7-8ac60dc58bd4", "id": "fe2f78fa308a48e19b065b818cec7827"}, "target": {"typeURI": "service/security/account/user", "id": "44542564-00c6-5d3c-b8b2-a2f70bb903db"}, "observer": {"typeURI": "service/security", "id": "34d389203c31493cabce6d4a4660951b"}, "eventType": "activity", "eventTime": "2020-10-30T14:15:21.112380+0000", "action": "authenticate", "outcome": "success", "id": "fe2600cd-96a5-5b98-85af-38ffce428ca5"}, "priority": "INFO", "publisher_id": "identity.clc-er13-u38.mgt.cluster", "message_id": "36dd5ab0-bafd-4412-9f42-fe76cfa2f1d6"}

/var/log/kolla/keystone/keystone.log:2020-10-30 14:15:21.141 23 WARNING keystone.server.flask.application [req-b47f80eb-93dc-4099-afe7-8ac60dc58bd4 - - - - -] You are not authorized to perform the requested action.: Forbidden: You are not authorized to perform the requested action.

/var/log/kolla/keystone/keystone-apache-public-error.log:2020-10-30 14:15:21.114564 2020-10-30 14:15:21.113 23 INFO oslo.messaging.notification.identity.authenticate [req-b47f80eb-93dc-4099-afe7-8ac60dc58bd4 - - - - -] {"event_type": "identity.authenticate", "timestamp": "2020-10-30 14:15:21.113010", "payload": {"typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "initiator": {"username": "563cea4f-230a-4932-96d5-5933906419d6_fd39e5b6c53d48a281c2988e83bfee8f", "typeURI": "service/security/account/user", "user_id": "fe2f78fa308a48e19b065b818cec7827", "host": {"agent": "openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.23.0 CPython/3.6.9", "address": "X.X.X.X"}, "request_id": "req-b47f80eb-93dc-4099-afe7-8ac60dc58bd4", "id": "fe2f78fa308a48e19b065b818cec7827"}, "target": {"typeURI": "service/security/account/user", "id": "44542564-00c6-5d3c-b8b2-a2f70bb903db"}, "observer": {"typeURI": "service/security", "id": "34d389203c31493cabce6d4a4660951b"}, "eventType": "activity", "eventTime": "2020-10-30T14:15:21.112380+0000", "action": "authenticate", "outcome": "success", "id": "fe2600cd-96a5-5b98-85af-38ffce428ca5"}, "priority": "INFO", "publisher_id": "identity.clc-er13-u38.mgt.cluster", "message_id": "36dd5ab0-bafd-4412-9f42-fe76cfa2f1d6"}\x1b[00m

/var/log/kolla/keystone/keystone-apache-public-error.log:2020-10-30 14:15:21.142409 2020-10-30 14:15:21.141 23 WARNING keystone.server.flask.application [req-b47f80eb-93dc-4099-afe7-8ac60dc58bd4 - - - - -] You are not authorized to perform the requested action.: Forbidden: You are not authorized to perform the requested action.\x1b[00m
-------------- next part --------------
A non-text attachment was scrubbed...
Name: magnum_k8s_1.17.3.yml
Type: application/octet-stream
Size: 1219 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20201030/b5949caf/attachment-0001.obj>
-------------- next part --------------
# FOR NEWLY CREATED MAGNUM CLUSTER
auth-url=https://mycluster.example.com:5000/v3
user-id=7f3e2a58ffe343378134c31fa183c8c6
user-name=fb65c16b-f50b-44e9-acf9-6fbcacef3967_402d56070d5c486fa1f4e7c01038dd51
password=*REMOVED*
trust-id=d3c5a1cc633549af9764e9af68099936

(oscli) paul at ubuntu-vm:~/Desktop$ openstack --os-user-domain-name magnum --os-trust-id d3c5a1cc633549af9764e9af68099936 --os-username fb65c16b-f50b-44e9-acf9-6fbcacef3967_402d56070d5c486fa1f4e7c01038dd51 --os-password *REMOVED* --os-auth-url https://mycluster.example.com:5000/v3 --os-region-name RegionOne server list -f value
949e079b-cb1e-414e-b615-957cbadd1499 pfb29-test-sytr5fdazfcp-gw-0 ACTIVE private=10.0.0.47, 128.232.222.179 CentOS7-1907 vm.v1.tiny
ab8bb214-71b4-45ec-9ce4-2a4de9852f33 pfb29-test-sytr5fdazfcp-node-0 ACTIVE private=10.0.0.222 FedoraAtomic29-20191126 vm.v1.small
d4baae3b-94b9-4321-ab65-04a977116dcc pfb29-test-sytr5fdazfcp-master-0 ACTIVE private=10.0.0.7 FedoraAtomic29-20191126 vm.v1.small
a8f0dd31-fb5d-44c0-ab4e-1fed02bb889c vm-teste ACTIVE WCDC-Prov-43=10.43.102.71 csd3-slurm-0.0.10 vm.v1.small
9428cc14-22db-46ea-8bf7-7230b1bf36f2 csd3-packages-0-0-10 SHUTOFF WCDC-Prov-43=10.43.102.219 csd3-upgrade-0.0.10 vm.v1.small
1b8ae16b-f159-416b-b74b-523549e71035 jg-part-test ACTIVE WCDC-Prov-43=10.43.102.40  vm.v1.small
33aaaa58-e8e1-4fd9-8e91-aa3e1e95a773 ofed-build SHUTOFF lab-net=10.0.0.99, 128.232.222.75 CentOS7-1907 vm.v1.tiny
37136ff4-23a7-49d6-b66d-9c7a5a0dc4e2 training-gw SHUTOFF lab-net=10.0.0.17; opencb-hdp=10.21.3.40  vm.v1.small

### FOR AN OLDER DEPLOYED MAGNUM CLUSTER
auth-url=https://mycluster.example.com:5000/v3
user-id=fe2f78fa308a48e19b065b818cec7827
user-name=563cea4f-230a-4932-96d5-5933906419d6_fd39e5b6c53d48a281c2988e83bfee8f
password=*REMOVED*
trust-id=215495bc20a845f18c1d2fa7a0d78e26

(oscli) paul at ubuntu-vm:~/Desktop$ openstack --os-user-domain-name magnum --os-trust-id 215495bc20a845f18c1d2fa7a0d78e26 --os-username 563cea4f-230a-4932-96d5-5933906419d6_fd39e5b6c53d48a281c2988e83bfee8f --os-password *REMOVED* --os-auth-url https://mycluster.example.com:5000/v3 --os-region-name RegionOne server list -f value
You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-b47f80eb-93dc-4099-afe7-8ac60dc58bd4)

### Details of failing trust-id
(oscli) paul at ubuntu-vm:~/Desktop$ openstack trust show 215495bc20a845f18c1d2fa7a0d78e26

+----------------------+----------------------------------+
| Field                | Value                            |
+----------------------+----------------------------------+
| delegation_depth     | 0                                |
| deleted_at           | None                             |
| expires_at           | None                             |
| id                   | 215495bc20a845f18c1d2fa7a0d78e26 |
| impersonation        | True                             |
| project_id           | fd39e5b6c53d48a281c2988e83bfee8f |
| redelegated_trust_id | None                             |
| redelegation_count   | 0                                |
| remaining_uses       | None                             |
| roles                | heat_stack_owner reader member   |
| trustee_user_id      | fe2f78fa308a48e19b065b818cec7827 |
| trustor_user_id      | 147366f69fa74674ba2dc541893e8f2f |
+----------------------+----------------------------------+