[openstack helm] [Cinder] [Ceph] keyring not getting decode when doing cinder-storage-init
Hello today i was testing to deploy openstack with helm in my kubernetes cluster and i encounter an error when trying to deploy cinder Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred: * job cinder-storage-init failed: BackoffLimitExceeded when i print the log of the cinder-storage-init pod it shows 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error setting modifier for [client.admin] type=key val=: Malformed input [buffer:3] 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 monclient: keyring not found [errno 5] RADOS I/O error (error connecting to the cluster) and the keyring secret is use looks like this: apiVersion: v1 data: key: AQDtmCFnh3STCBAAGKQIyIGuZjnmX3L4mPCQXw== kind: Secret metadata: name: pvc-ceph-client-key namespace: openstack after checking secret and the rook-ceph cluster works fine i change the template and makes the pod not executing the command so i can look inside the pod and it's keyrings the keyring looks like this: [client.admin] key = �!g�tȁ�f9�_r���_ so apparently the keyring secert or when it mount into the pod got some problem and change the key into the normal one fix the problem is there a reason why it cause this ? because the secret and ceph-cluster are all works fine
It is not so easy to see from the description what is wrong. Have you deployed the ceph-adapter-rook chart? It is supposed to get the admin key from rook-ceph-tools pod using the command `ceph auth ls` and then it puts the base64 encoded key to the secret pvc-ceph-client-key. In your case it looks like you have this secret pvc-ceph-client-key but it contains invalid data. On Wed, Oct 30, 2024 at 12:51 AM <daniel890723@gmail.com> wrote:
Hello today i was testing to deploy openstack with helm in my kubernetes cluster and i encounter an error when trying to deploy cinder Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred: * job cinder-storage-init failed: BackoffLimitExceeded when i print the log of the cinder-storage-init pod it shows
2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error setting modifier for [client.admin] type=key val=: Malformed input [buffer:3] 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 monclient: keyring not found [errno 5] RADOS I/O error (error connecting to the cluster)
and the keyring secret is use looks like this: apiVersion: v1 data: key: AQDtmCFnh3STCBAAGKQIyIGuZjnmX3L4mPCQXw== kind: Secret metadata: name: pvc-ceph-client-key namespace: openstack
after checking secret and the rook-ceph cluster works fine i change the template and makes the pod not executing the command so i can look inside the pod and it's keyrings the keyring looks like this: [client.admin] key = �!g�tȁ�f9�_r���_
so apparently the keyring secert or when it mount into the pod got some problem and change the key into the normal one fix the problem
is there a reason why it cause this ? because the secret and ceph-cluster are all works fine
-- Best regards, Kozhukalov Vladimir
yes i did deploy the ceph-adapter-rook but it seems like i didn't have the rook-ceph-tools Pods i only deploy the rook-ceph-operator and rook cluster following the guide from rook documentation Did i miss some step so i don't get the rook-ceph-tools pod ? Also does the rook-ceph-operator has to be in the namespace ceph ? because i only put the rook-ceph-cluster in namespace ceph and the operator are still in the rook-ceph namespace
In Openstack-Helm we use this script [1] to deploy the Rook operator and Ceph cluster. As you can see it uses official Rook helm charts with some custom values. It creates the rook-ceph-tools Deployment which creates the pod rook-ceph-tools-xxxxxxxx-yyyyy that contains some CLI tools for managing Ceph clusters. So you can run `ceph auth ls` inside this pod. If you don't have this tools pod, then you can run the command manually in a pod where this tool is available. Here is the command from the ceph-adapter-rook chart that is used to get the key. [1] https://opendev.org/openstack/openstack-helm-infra/src/branch/master/tools/d... [2] https://opendev.org/openstack/openstack-helm-infra/src/branch/master/ceph-ad... On Wed, Oct 30, 2024 at 7:44 PM <daniel890723@gmail.com> wrote:
yes i did deploy the ceph-adapter-rook but it seems like i didn't have the rook-ceph-tools Pods i only deploy the rook-ceph-operator and rook cluster following the guide from rook documentation
Did i miss some step so i don't get the rook-ceph-tools pod ?
Also does the rook-ceph-operator has to be in the namespace ceph ? because i only put the rook-ceph-cluster in namespace ceph and the operator are still in the rook-ceph namespace
-- Best regards, Kozhukalov Vladimir
Thank you for replying after checking the script i notice i was using the wrong value.yaml when deploying cluster i will test with the right value.yaml to see if it fix the error Thank you
participants (2)
-
daniel890723＠gmail.com
-
Vladimir Kozhukalov