<div dir="ltr"><div>Hi,</div><div>@Oliver, thanks to you for your blog, it was simple yet it helped me a lot. I am a newbie in the kubernetes world.</div><div><br></div><div>@Nguyen, yes I do have <span style="font-family:monospace">enable_cluster_user_trust enabled in my globals.yml</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">From these two threads (<a href="https://github.com/rook/rook/issues/6457">https://github.com/rook/rook/issues/6457</a>, <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1769693">https://bugzilla.redhat.com/show_bug.cgi?id=1769693</a>), I think it's an access right problem, a missing access right, what I don't know, is should I add this access right manually? should I update the rest of the images in the cluster, maybe one of them contains the missing right?<br></span></div><div><br></div><div>In the first thread it is said : <br><p dir="auto">Solution:-</p>
<ul dir="auto"><li>apiGroups: ["<a href="http://storage.k8s.io">storage.k8s.io</a>"]<br>
resources: ["volumeattachments/status"]<br>
verbs: ["patch"]<br>
need to be added to rbd-external-provisioner-runner and cephfs-external-provisioner-runner ClusterRole</li></ul><div>In the second thread :</div><div><pre class="gmail-bz_comment_text" id="gmail-comment_text_2">csi-external-attacher has changed in 4.3</pre></div><div><pre class="gmail-bz_comment_text" id="gmail-comment_text_2">external attacher needs extra privileges to patch various API objects.
</pre></div></div><div><br></div><div><br></div><div>Regards.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le lun. 20 mars 2023 à 03:34, Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com">nguyenhuukhoinw@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello.<div>Are you enable <span style="color:rgb(110,191,38);font-weight:bold;background-color:rgb(51,51,51);font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:13px">enable_cluster_user_trust?</span><div><div dir="ltr"><div dir="ltr">Nguyen Huu Khoi<br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 20, 2023 at 12:42 AM wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>I am trying to attach a cinder volume to my pod, but it does not work.</div><div><br></div><div>The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got :</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> Image: <a href="http://quay.io/k8scsi/csi-attacher:v2.0.0" target="_blank">quay.io/k8scsi/csi-attacher:v2.0.0</a><br> Image: <a href="http://quay.io/k8scsi/csi-provisioner:v1.4.0" target="_blank">quay.io/k8scsi/csi-provisioner:v1.4.0</a><br> Image: <a href="http://quay.io/k8scsi/csi-snapshotter:v1.2.2" target="_blank">quay.io/k8scsi/csi-snapshotter:v1.2.2</a><br> Image: <a href="http://quay.io/k8scsi/csi-resizer:v0.3.0" target="_blank">quay.io/k8scsi/csi-resizer:v0.3.0</a><br> Image: <a href="http://docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0" target="_blank">docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0</a><br> Image: <a href="http://quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" target="_blank">quay.io/k8scsi/csi-node-driver-registrar:v1.1.0</a></div><div> Image: <a href="http://docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1" target="_blank">docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1</a></div></blockquote><div><br></div><div>Which <br></div><div>1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga :</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> Image: <a href="http://10.0.0.165:4000/csi-attacher:v3.3.0" target="_blank">10.0.0.165:4000/csi-attacher:v3.3.0</a><br> Image: <a href="http://10.0.0.165:4000/csi-provisioner:v3.0.0" target="_blank">10.0.0.165:4000/csi-provisioner:v3.0.0</a><br> Image: <a href="http://10.0.0.165:4000/csi-snapshotter:v4.2.1" target="_blank">10.0.0.165:4000/csi-snapshotter:v4.2.1</a><br> Image: <a href="http://10.0.0.165:4000/csi-resizer:v1.3.0" target="_blank">10.0.0.165:4000/csi-resizer:v1.3.0</a><br> Image: <a href="http://10.0.0.165:4000/cinder-csi-plugin:v1.26.2" target="_blank">10.0.0.165:4000/cinder-csi-plugin:v1.26.2</a><br> Image: <a href="http://10.0.0.165:4000/csi-node-driver-registrar:v2.4.0" target="_blank">10.0.0.165:4000/csi-node-driver-registrar:v2.4.0</a><br> Image: <a href="http://10.0.0.165:4000/cinder-csi-plugin:v1.26.2" target="_blank">10.0.0.165:4000/cinder-csi-plugin:v1.26.2</a> (cinder-csi-plugin:v1.23.0 which does not exists anymore)<br></div></blockquote><div><br></div><div>2 - And does not work, csi-cinder-controllerplugin keeps crashing.</div><div><br></div><div>I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got :</div><div><br></div><div>Volumes:<br> html-volume:<br> Type: Cinder (a Persistent Disk resource in OpenStack)<br> VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1<br> FSType: ext4<br> ReadOnly: false<br> SecretRef: nil<br> kube-api-access-slqf4:<br> Type: Projected (a volume that contains injected data from multiple sources)<br> TokenExpirationSeconds: 3607<br> ConfigMapName: kube-root-ca.crt<br> ConfigMapOptional: <nil><br> DownwardAPI: true<br>QoS Class: Burstable<br>Node-Selectors: <none><br>Tolerations: <a href="http://node.kubernetes.io/not-ready:NoExecute" target="_blank">node.kubernetes.io/not-ready:NoExecute</a> op=Exists for 300s<br> <a href="http://node.kubernetes.io/unreachable:NoExecute" target="_blank">node.kubernetes.io/unreachable:NoExecute</a> op=Exists for 300s<br>Events:<br> Type Reason Age From Message<br> ---- ------ ---- ---- -------<br><b> Warning FailedMount 26m (x10 over 135m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting for the condition<br> Warning FailedAttachVolume 3m39s (x40 over 146m) attachdetach-controller AttachVolume.Attach failed for volume "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1<br> Warning FailedMount 104s (x54 over 146m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the condition</b></div><div><br></div><div><br></div><div>"volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]}<br>I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil><br>I0319 12:36:56.925658 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:56.925687 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0<br>I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:56.947632 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume<br>I0319 12:36:56.947646 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"}<br>I0319 12:36:58.343821 1 connection.go:200] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}}<br>I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil><br>I0319 12:36:58.343841 1 csi_handler.go:264] Attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:58.343848 1 util.go:38] Marking as attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br><b>I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: <a href="http://volumeattachments.storage.k8s.io" target="_blank">volumeattachments.storage.k8s.io</a> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "<a href="http://storage.k8s.io" target="_blank">storage.k8s.io</a>" at the cluster scope</b><br>I0319 12:36:58.348503 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:58.348513 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0<br>I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"<br>I0319 12:36:58.348564 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume<br>I0319 12:36:58.348567 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext:</div><div><br></div><div><br></div><div>The I tried even the most updated images :</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> <a href="http://10.0.0.165:4000/cinder-csi-plugin:v1.26.2" target="_blank">10.0.0.165:4000/cinder-csi-plugin:v1.26.2</a><br> <a href="http://10.0.0.165:4000/csi-provisioner:v3.4.0" target="_blank">10.0.0.165:4000/csi-provisioner:v3.4.0</a><br> <a href="http://10.0.0.165:4000/csi-resizer:v1.7.0" target="_blank">10.0.0.165:4000/csi-resizer:v1.7.0</a><br> <a href="http://10.0.0.165:4000/csi-snapshotter:v6.2.1" target="_blank">10.0.0.165:4000/csi-snapshotter:v6.2.1</a><br> <a href="http://10.0.0.165:4000/csi-attacher:v4.2.0" target="_blank">10.0.0.165:4000/csi-attacher:v4.2.0</a><br> <a href="http://10.0.0.165:4000/csi-node-driver-registrar:v2.7.0" target="_blank">10.0.0.165:4000/csi-node-driver-registrar:v2.7.0</a></div></blockquote><div><br></div><div>I had the same problem. <br></div><div><br></div><div>Then I tried to use an older version of kubernetes : 1.21.11 with the older images shown above (following this link <a href="https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minutes-with-magnum/" target="_blank">https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minutes-with-magnum/</a>), and it worked, the cinder volume was successfully mounted inside my nginx pod.</div><div><br></div><div><br></div><div><br></div><div>- What is the meaning of the error I am having?</div><div>- Is it magnum related or kubernetes related or both?<br></div><div><br></div><div><br></div><div>Regards.<br></div></div>
</blockquote></div>
</blockquote></div>