[kolla-ansible][yoga][Magnum] Cannot attach cinder volume to pod
wodel youchi
wodel.youchi at gmail.com
Mon Mar 20 10:40:35 UTC 2023
Hi,
@Oliver, thanks to you for your blog, it was simple yet it helped me a lot.
I am a newbie in the kubernetes world.
@Nguyen, yes I do have enable_cluster_user_trust enabled in my globals.yml
>From these two threads (https://github.com/rook/rook/issues/6457,
https://bugzilla.redhat.com/show_bug.cgi?id=1769693), I think it's an
access right problem, a missing access right, what I don't know, is should
I add this access right manually? should I update the rest of the images in
the cluster, maybe one of them contains the missing right?
In the first thread it is said :
Solution:-
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"]
verbs: ["patch"]
need to be added to rbd-external-provisioner-runner and
cephfs-external-provisioner-runner ClusterRole
In the second thread :
csi-external-attacher has changed in 4.3
external attacher needs extra privileges to patch various API objects.
Regards.
Le lun. 20 mars 2023 à 03:34, Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com> a
écrit :
> Hello.
> Are you enable enable_cluster_user_trust?
> Nguyen Huu Khoi
>
>
> On Mon, Mar 20, 2023 at 12:42 AM wodel youchi <wodel.youchi at gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying to attach a cinder volume to my pod, but it does not work.
>>
>> The long story, the default version of kubernetes used in Yoga is 1.23.3
>> fcore35. When creating a default kubernetes cluster we got :
>>
>>> Image: quay.io/k8scsi/csi-attacher:v2.0.0
>>> Image: quay.io/k8scsi/csi-provisioner:v1.4.0
>>> Image: quay.io/k8scsi/csi-snapshotter:v1.2.2
>>> Image: quay.io/k8scsi/csi-resizer:v0.3.0
>>> Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0
>>> Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
>>> Image:
>>> docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1
>>>
>>
>> Which
>> 1 - Does not correspond to the documentation of Magnum, the documentation
>> states these defaults for yoga :
>>
>>> Image: 10.0.0.165:4000/csi-attacher:v3.3.0
>>> Image: 10.0.0.165:4000/csi-provisioner:v3.0.0
>>> Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1
>>> Image: 10.0.0.165:4000/csi-resizer:v1.3.0
>>> Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2
>>> Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0
>>> Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2
>>> (cinder-csi-plugin:v1.23.0 which does not exists anymore)
>>>
>>
>> 2 - And does not work, csi-cinder-controllerplugin keeps crashing.
>>
>> I tried to use the updates images (using a local registry), but I
>> couldn't attach the cinder-volume, I got :
>>
>> Volumes:
>> html-volume:
>> Type: Cinder (a Persistent Disk resource in OpenStack)
>> VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1
>> FSType: ext4
>> ReadOnly: false
>> SecretRef: nil
>> kube-api-access-slqf4:
>> Type: Projected (a volume that contains injected
>> data from multiple sources)
>> TokenExpirationSeconds: 3607
>> ConfigMapName: kube-root-ca.crt
>> ConfigMapOptional: <nil>
>> DownwardAPI: true
>> QoS Class: Burstable
>> Node-Selectors: <none>
>> Tolerations: node.kubernetes.io/not-ready:NoExecute
>> op=Exists for 300s
>> node.kubernetes.io/unreachable:NoExecute
>> op=Exists for 300s
>> Events:
>> Type Reason Age From
>> Message
>> ---- ------ ---- ----
>> -------
>>
>>
>> * Warning FailedMount 26m (x10 over 135m) kubelet
>> Unable to attach or mount volumes: unmounted volumes=[html-volume],
>> unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting
>> for the condition Warning FailedAttachVolume 3m39s (x40 over 146m)
>> attachdetach-controller AttachVolume.Attach failed for volume
>> "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach
>> timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning
>> FailedMount 104s (x54 over 146m) kubelet Unable
>> to attach or mount volumes: unmounted volumes=[html-volume], unattached
>> volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the
>> condition*
>>
>>
>>
>> "volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]}
>> I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil>
>> I0319 12:36:56.925658 1 controller.go:210] Started VA processing
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing
>> VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:56.925687 1 csi_handler.go:251] Attaching
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach
>> operation for
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID
>> 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode
>> k8intcalnewer-56bgom6jntbm-node-0
>> I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added
>> to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:56.947632 1 connection.go:193] GRPC call:
>> /csi.v1.Controller/ControllerPublishVolume
>> I0319 12:36:56.947646 1 connection.go:194] GRPC request:
>> {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"}
>> I0319 12:36:58.343821 1 connection.go:200] GRPC response:
>> {"publish_context":{"DevicePath":"/dev/vdc"}}
>> I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil>
>> I0319 12:36:58.343841 1 csi_handler.go:264] Attached
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:58.343848 1 util.go:38] Marking as attached
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7":
>> failed to mark as attached: volumeattachments.storage.k8s.io
>> <http://volumeattachments.storage.k8s.io>
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is
>> forbidden: User
>> "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch
>> resource "volumeattachments/status" in API group "storage.k8s.io
>> <http://storage.k8s.io>" at the cluster scope*
>> I0319 12:36:58.348503 1 controller.go:210] Started VA processing
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing
>> VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:58.348513 1 csi_handler.go:251] Attaching
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach
>> operation for
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID
>> 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode
>> k8intcalnewer-56bgom6jntbm-node-0
>> I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already
>> set on
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is
>> already set on
>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>> I0319 12:36:58.348564 1 connection.go:193] GRPC call:
>> /csi.v1.Controller/ControllerPublishVolume
>> I0319 12:36:58.348567 1 connection.go:194] GRPC request:
>> {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext:
>>
>>
>> The I tried even the most updated images :
>>
>>> 10.0.0.165:4000/cinder-csi-plugin:v1.26.2
>>> 10.0.0.165:4000/csi-provisioner:v3.4.0
>>> 10.0.0.165:4000/csi-resizer:v1.7.0
>>> 10.0.0.165:4000/csi-snapshotter:v6.2.1
>>> 10.0.0.165:4000/csi-attacher:v4.2.0
>>> 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0
>>>
>>
>> I had the same problem.
>>
>> Then I tried to use an older version of kubernetes : 1.21.11 with the
>> older images shown above (following this link
>> https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minutes-with-magnum/),
>> and it worked, the cinder volume was successfully mounted inside my nginx
>> pod.
>>
>>
>>
>> - What is the meaning of the error I am having?
>> - Is it magnum related or kubernetes related or both?
>>
>>
>> Regards.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230320/66347007/attachment-0001.htm>
More information about the openstack-discuss
mailing list