[kolla-ansible][yoga][Magnum] Cannot attach cinder volume to pod
Hi,
I am trying to attach a cinder volume to my pod, but it does not work.
The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got :
Image: quay.io/k8scsi/csi-attacher:v2.0.0 Image: quay.io/k8scsi/csi-provisioner:v1.4.0 Image: quay.io/k8scsi/csi-snapshotter:v1.2.2 Image: quay.io/k8scsi/csi-resizer:v0.3.0 Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0 Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0 Image:docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1
Which 1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga :
Image: 10.0.0.165:4000/csi-attacher:v3.3.0 Image: 10.0.0.165:4000/csi-provisioner:v3.0.0 Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1 Image: 10.0.0.165:4000/csi-resizer:v1.3.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2(cinder-csi-plugin:v1.23.0 which does not exists anymore)
2 - And does not work, csi-cinder-controllerplugin keeps crashing.
I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got :
Volumes: html-volume: Type: Cinder (a Persistent Disk resource in OpenStack) VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1 FSType: ext4 ReadOnly: false SecretRef: nil kube-api-access-slqf4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- -------
* Warning FailedMount 26m (x10 over 135m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting for the condition Warning FailedAttachVolume 3m39s (x40 over 146m) attachdetach-controller AttachVolume.Attach failed for volume "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning FailedMount 104s (x54 over 146m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the condition*
"volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]} I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil> I0319 12:36:56.925658 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925687 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.947632 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:56.947646 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"} I0319 12:36:58.343821 1 connection.go:200] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}} I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil> I0319 12:36:58.343841 1 csi_handler.go:264] Attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.343848 1 util.go:38] Marking as attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: volumeattachments.storage.k8s.io http://volumeattachments.storage.k8s.io "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io http://storage.k8s.io" at the cluster scope* I0319 12:36:58.348503 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348513 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348564 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:58.348567 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext:
The I tried even the most updated images :
10.0.0.165:4000/cinder-csi-plugin:v1.26.2 10.0.0.165:4000/csi-provisioner:v3.4.0 10.0.0.165:4000/csi-resizer:v1.7.0 10.0.0.165:4000/csi-snapshotter:v6.2.1 10.0.0.165:4000/csi-attacher:v4.2.0 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0
I had the same problem.
Then I tried to use an older version of kubernetes : 1.21.11 with the older images shown above (following this link https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minut...), and it worked, the cinder volume was successfully mounted inside my nginx pod.
- What is the meaning of the error I am having? - Is it magnum related or kubernetes related or both?
Regards.
Hello. Are you enable enable_cluster_user_trust? Nguyen Huu Khoi
On Mon, Mar 20, 2023 at 12:42 AM wodel youchi wodel.youchi@gmail.com wrote:
Hi,
I am trying to attach a cinder volume to my pod, but it does not work.
The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got :
Image: quay.io/k8scsi/csi-attacher:v2.0.0 Image: quay.io/k8scsi/csi-provisioner:v1.4.0 Image: quay.io/k8scsi/csi-snapshotter:v1.2.2 Image: quay.io/k8scsi/csi-resizer:v0.3.0 Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0 Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0 Image:docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1
Which 1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga :
Image: 10.0.0.165:4000/csi-attacher:v3.3.0 Image: 10.0.0.165:4000/csi-provisioner:v3.0.0 Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1 Image: 10.0.0.165:4000/csi-resizer:v1.3.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2(cinder-csi-plugin:v1.23.0 which does not exists anymore)
2 - And does not work, csi-cinder-controllerplugin keeps crashing.
I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got :
Volumes: html-volume: Type: Cinder (a Persistent Disk resource in OpenStack) VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1 FSType: ext4 ReadOnly: false SecretRef: nil kube-api-access-slqf4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message
-------
- Warning FailedMount 26m (x10 over 135m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume],
unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting for the condition Warning FailedAttachVolume 3m39s (x40 over 146m) attachdetach-controller AttachVolume.Attach failed for volume "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning FailedMount 104s (x54 over 146m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the condition*
"volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]} I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil> I0319 12:36:56.925658 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925687 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.947632 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:56.947646 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"} I0319 12:36:58.343821 1 connection.go:200] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}} I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil> I0319 12:36:58.343841 1 csi_handler.go:264] Attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.343848 1 util.go:38] Marking as attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: volumeattachments.storage.k8s.io http://volumeattachments.storage.k8s.io "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io http://storage.k8s.io" at the cluster scope* I0319 12:36:58.348503 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348513 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348564 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:58.348567 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext:
The I tried even the most updated images :
10.0.0.165:4000/cinder-csi-plugin:v1.26.2 10.0.0.165:4000/csi-provisioner:v3.4.0 10.0.0.165:4000/csi-resizer:v1.7.0 10.0.0.165:4000/csi-snapshotter:v6.2.1 10.0.0.165:4000/csi-attacher:v4.2.0 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0
I had the same problem.
Then I tried to use an older version of kubernetes : 1.21.11 with the older images shown above (following this link https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minut...), and it worked, the cinder volume was successfully mounted inside my nginx pod.
- What is the meaning of the error I am having?
- Is it magnum related or kubernetes related or both?
Regards.
Hi, @Oliver, thanks to you for your blog, it was simple yet it helped me a lot. I am a newbie in the kubernetes world.
@Nguyen, yes I do have enable_cluster_user_trust enabled in my globals.yml
From these two threads (https://github.com/rook/rook/issues/6457,
https://bugzilla.redhat.com/show_bug.cgi?id=1769693), I think it's an access right problem, a missing access right, what I don't know, is should I add this access right manually? should I update the rest of the images in the cluster, maybe one of them contains the missing right?
In the first thread it is said :
Solution:-
- apiGroups: ["storage.k8s.io"] resources: ["volumeattachments/status"] verbs: ["patch"] need to be added to rbd-external-provisioner-runner and cephfs-external-provisioner-runner ClusterRole
In the second thread :
csi-external-attacher has changed in 4.3
external attacher needs extra privileges to patch various API objects.
Regards.
Le lun. 20 mars 2023 à 03:34, Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com a écrit :
Hello. Are you enable enable_cluster_user_trust? Nguyen Huu Khoi
On Mon, Mar 20, 2023 at 12:42 AM wodel youchi wodel.youchi@gmail.com wrote:
Hi,
I am trying to attach a cinder volume to my pod, but it does not work.
The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got :
Image: quay.io/k8scsi/csi-attacher:v2.0.0 Image: quay.io/k8scsi/csi-provisioner:v1.4.0 Image: quay.io/k8scsi/csi-snapshotter:v1.2.2 Image: quay.io/k8scsi/csi-resizer:v0.3.0 Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0 Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0 Image:docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1
Which 1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga :
Image: 10.0.0.165:4000/csi-attacher:v3.3.0 Image: 10.0.0.165:4000/csi-provisioner:v3.0.0 Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1 Image: 10.0.0.165:4000/csi-resizer:v1.3.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2(cinder-csi-plugin:v1.23.0 which does not exists anymore)
2 - And does not work, csi-cinder-controllerplugin keeps crashing.
I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got :
Volumes: html-volume: Type: Cinder (a Persistent Disk resource in OpenStack) VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1 FSType: ext4 ReadOnly: false SecretRef: nil kube-api-access-slqf4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message
-------
- Warning FailedMount 26m (x10 over 135m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume],
unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting for the condition Warning FailedAttachVolume 3m39s (x40 over 146m) attachdetach-controller AttachVolume.Attach failed for volume "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning FailedMount 104s (x54 over 146m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the condition*
"volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]} I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil> I0319 12:36:56.925658 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925687 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.947632 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:56.947646 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"} I0319 12:36:58.343821 1 connection.go:200] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}} I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil> I0319 12:36:58.343841 1 csi_handler.go:264] Attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.343848 1 util.go:38] Marking as attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: volumeattachments.storage.k8s.io http://volumeattachments.storage.k8s.io "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io http://storage.k8s.io" at the cluster scope* I0319 12:36:58.348503 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348513 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348564 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:58.348567 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext:
The I tried even the most updated images :
10.0.0.165:4000/cinder-csi-plugin:v1.26.2 10.0.0.165:4000/csi-provisioner:v3.4.0 10.0.0.165:4000/csi-resizer:v1.7.0 10.0.0.165:4000/csi-snapshotter:v6.2.1 10.0.0.165:4000/csi-attacher:v4.2.0 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0
I had the same problem.
Then I tried to use an older version of kubernetes : 1.21.11 with the older images shown above (following this link https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minut...), and it worked, the cinder volume was successfully mounted inside my nginx pod.
- What is the meaning of the error I am having?
- Is it magnum related or kubernetes related or both?
Regards.
Hi,
As stated by @Jake, there is some code lingering in https://review.opendev.org/c/openstack/magnum/+/833354 but it has not been merged, it does not exist even in the master branch of Magnum. It looks like we have no choice but the 1.21 version of kubernetes for now.
Regards.
Le lun. 20 mars 2023 à 15:53, Oliver Weinmann oliver.weinmann@me.com a écrit :
Hi,
Good point:
external attacher needs extra privileges to patch various API objects
I remember that I played around with this an tried to apply some yaml files but couldn’t make it work.
Cheers,
Oliver
Von meinem iPhone gesendet
Am 20.03.2023 um 11:43 schrieb wodel youchi wodel.youchi@gmail.com:
Hi, @Oliver, thanks to you for your blog, it was simple yet it helped me a lot. I am a newbie in the kubernetes world.
@Nguyen, yes I do have enable_cluster_user_trust enabled in my globals.yml
From these two threads (https://github.com/rook/rook/issues/6457, https://bugzilla.redhat.com/show_bug.cgi?id=1769693), I think it's an access right problem, a missing access right, what I don't know, is should I add this access right manually? should I update the rest of the images in the cluster, maybe one of them contains the missing right?
In the first thread it is said :
Solution:-
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"] verbs: ["patch"] need to be added to rbd-external-provisioner-runner and cephfs-external-provisioner-runner ClusterRole
In the second thread :
csi-external-attacher has changed in 4.3
external attacher needs extra privileges to patch various API objects.
Regards.
Le lun. 20 mars 2023 à 03:34, Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com a écrit :
Hello. Are you enable enable_cluster_user_trust? Nguyen Huu Khoi
On Mon, Mar 20, 2023 at 12:42 AM wodel youchi wodel.youchi@gmail.com wrote:
Hi,
I am trying to attach a cinder volume to my pod, but it does not work.
The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got :
Image: quay.io/k8scsi/csi-attacher:v2.0.0 Image: quay.io/k8scsi/csi-provisioner:v1.4.0 Image: quay.io/k8scsi/csi-snapshotter:v1.2.2 Image: quay.io/k8scsi/csi-resizer:v0.3.0 Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0 Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0 Image:docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1
Which 1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga :
Image: 10.0.0.165:4000/csi-attacher:v3.3.0 Image: 10.0.0.165:4000/csi-provisioner:v3.0.0 Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1 Image: 10.0.0.165:4000/csi-resizer:v1.3.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0 Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2(cinder-csi-plugin:v1.23.0 which does not exists anymore)
2 - And does not work, csi-cinder-controllerplugin keeps crashing.
I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got :
Volumes: html-volume: Type: Cinder (a Persistent Disk resource in OpenStack) VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1 FSType: ext4 ReadOnly: false SecretRef: nil kube-api-access-slqf4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message
-------
- Warning FailedMount 26m (x10 over 135m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume],
unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting for the condition Warning FailedAttachVolume 3m39s (x40 over 146m) attachdetach-controller AttachVolume.Attach failed for volume "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning FailedMount 104s (x54 over 146m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the condition*
"volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]} I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil> I0319 12:36:56.925658 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925687 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.947632 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:56.947646 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"} I0319 12:36:58.343821 1 connection.go:200] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}} I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil> I0319 12:36:58.343841 1 csi_handler.go:264] Attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.343848 1 util.go:38] Marking as attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: volumeattachments.storage.k8s.io http://volumeattachments.storage.k8s.io "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io http://storage.k8s.io" at the cluster scope* I0319 12:36:58.348503 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348513 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348564 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:58.348567 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext:
The I tried even the most updated images :
10.0.0.165:4000/cinder-csi-plugin:v1.26.2 10.0.0.165:4000/csi-provisioner:v3.4.0 10.0.0.165:4000/csi-resizer:v1.7.0 10.0.0.165:4000/csi-snapshotter:v6.2.1 10.0.0.165:4000/csi-attacher:v4.2.0 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0
I had the same problem.
Then I tried to use an older version of kubernetes : 1.21.11 with the older images shown above (following this link https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minut...), and it worked, the cinder volume was successfully mounted inside my nginx pod.
- What is the meaning of the error I am having?
- Is it magnum related or kubernetes related or both?
Regards.
Hi,
I had a feeling the below two issues are due to a missing backport[1] to Yoga.
I tried to backport it locally but it failed devstack, so it might take a while before we have something.
Regards, Jake
[1] https://review.opendev.org/c/openstack/magnum/+/833354
On 20/3/2023 4:36 am, wodel youchi wrote:
Hi,
I am trying to attach a cinder volume to my pod, but it does not work.
The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got :
Image: quay.io/k8scsi/csi-attacher:v2.0.0 <http://quay.io/k8scsi/csi-attacher:v2.0.0>...
Which 1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga :
Image: 10.0.0.165:4000/csi-attacher:v3.3.0 <http://10.0.0.165:4000/csi-attacher:v3.3.0>...
I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got :
... *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: volumeattachments.storage.k8s.io http://volumeattachments.storage.k8s.io "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io http://storage.k8s.io" at the cluster scope*
participants (4)
-
Jake Yip
-
Nguyễn Hữu Khôi
-
Oliver Weinmann
-
wodel youchi