[kolla-ansible][yoga][Magnum] Cannot attach cinder volume to pod
wodel youchi
wodel.youchi at gmail.com
Mon Mar 20 15:50:03 UTC 2023
Hi,
As stated by @Jake, there is some code lingering in
https://review.opendev.org/c/openstack/magnum/+/833354 but it has not been
merged, it does not exist even in the master branch of Magnum.
It looks like we have no choice but the 1.21 version of kubernetes for now.
Regards.
Le lun. 20 mars 2023 à 15:53, Oliver Weinmann <oliver.weinmann at me.com> a
écrit :
> Hi,
>
> Good point:
>
> external attacher needs extra privileges to patch various API objects
>
>
> I remember that I played around with this an tried to apply some yaml files but couldn’t make it work.
>
>
> Cheers,
>
> Oliver
>
>
> Von meinem iPhone gesendet
>
> Am 20.03.2023 um 11:43 schrieb wodel youchi <wodel.youchi at gmail.com>:
>
>
> Hi,
> @Oliver, thanks to you for your blog, it was simple yet it helped me a
> lot. I am a newbie in the kubernetes world.
>
> @Nguyen, yes I do have enable_cluster_user_trust enabled in my globals.yml
>
> From these two threads (https://github.com/rook/rook/issues/6457,
> https://bugzilla.redhat.com/show_bug.cgi?id=1769693), I think it's an
> access right problem, a missing access right, what I don't know, is should
> I add this access right manually? should I update the rest of the images in
> the cluster, maybe one of them contains the missing right?
>
> In the first thread it is said :
>
> Solution:-
>
> - apiGroups: ["storage.k8s.io"]
> resources: ["volumeattachments/status"]
> verbs: ["patch"]
> need to be added to rbd-external-provisioner-runner and
> cephfs-external-provisioner-runner ClusterRole
>
> In the second thread :
>
> csi-external-attacher has changed in 4.3
>
> external attacher needs extra privileges to patch various API objects.
>
>
>
> Regards.
>
> Le lun. 20 mars 2023 à 03:34, Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com>
> a écrit :
>
>> Hello.
>> Are you enable enable_cluster_user_trust?
>> Nguyen Huu Khoi
>>
>>
>> On Mon, Mar 20, 2023 at 12:42 AM wodel youchi <wodel.youchi at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to attach a cinder volume to my pod, but it does not work.
>>>
>>> The long story, the default version of kubernetes used in Yoga is 1.23.3
>>> fcore35. When creating a default kubernetes cluster we got :
>>>
>>>> Image: quay.io/k8scsi/csi-attacher:v2.0.0
>>>> Image: quay.io/k8scsi/csi-provisioner:v1.4.0
>>>> Image: quay.io/k8scsi/csi-snapshotter:v1.2.2
>>>> Image: quay.io/k8scsi/csi-resizer:v0.3.0
>>>> Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0
>>>> Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
>>>> Image:
>>>> docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1
>>>>
>>>
>>> Which
>>> 1 - Does not correspond to the documentation of Magnum, the
>>> documentation states these defaults for yoga :
>>>
>>>> Image: 10.0.0.165:4000/csi-attacher:v3.3.0
>>>> Image: 10.0.0.165:4000/csi-provisioner:v3.0.0
>>>> Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1
>>>> Image: 10.0.0.165:4000/csi-resizer:v1.3.0
>>>> Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2
>>>> Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0
>>>> Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2
>>>> (cinder-csi-plugin:v1.23.0 which does not exists anymore)
>>>>
>>>
>>> 2 - And does not work, csi-cinder-controllerplugin keeps crashing.
>>>
>>> I tried to use the updates images (using a local registry), but I
>>> couldn't attach the cinder-volume, I got :
>>>
>>> Volumes:
>>> html-volume:
>>> Type: Cinder (a Persistent Disk resource in OpenStack)
>>> VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1
>>> FSType: ext4
>>> ReadOnly: false
>>> SecretRef: nil
>>> kube-api-access-slqf4:
>>> Type: Projected (a volume that contains injected
>>> data from multiple sources)
>>> TokenExpirationSeconds: 3607
>>> ConfigMapName: kube-root-ca.crt
>>> ConfigMapOptional: <nil>
>>> DownwardAPI: true
>>> QoS Class: Burstable
>>> Node-Selectors: <none>
>>> Tolerations: node.kubernetes.io/not-ready:NoExecute
>>> op=Exists for 300s
>>> node.kubernetes.io/unreachable:NoExecute
>>> op=Exists for 300s
>>> Events:
>>> Type Reason Age From
>>> Message
>>> ---- ------ ---- ----
>>> -------
>>>
>>>
>>> * Warning FailedMount 26m (x10 over 135m) kubelet
>>> Unable to attach or mount volumes: unmounted volumes=[html-volume],
>>> unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting
>>> for the condition Warning FailedAttachVolume 3m39s (x40 over 146m)
>>> attachdetach-controller AttachVolume.Attach failed for volume
>>> "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach
>>> timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning
>>> FailedMount 104s (x54 over 146m) kubelet Unable
>>> to attach or mount volumes: unmounted volumes=[html-volume], unattached
>>> volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the
>>> condition*
>>>
>>>
>>>
>>> "volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]}
>>> I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil>
>>> I0319 12:36:56.925658 1 controller.go:210] Started VA processing
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing
>>> VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:56.925687 1 csi_handler.go:251] Attaching
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach
>>> operation for
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID
>>> 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode
>>> k8intcalnewer-56bgom6jntbm-node-0
>>> I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation
>>> added to
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:56.947632 1 connection.go:193] GRPC call:
>>> /csi.v1.Controller/ControllerPublishVolume
>>> I0319 12:36:56.947646 1 connection.go:194] GRPC request:
>>> {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"}
>>> I0319 12:36:58.343821 1 connection.go:200] GRPC response:
>>> {"publish_context":{"DevicePath":"/dev/vdc"}}
>>> I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil>
>>> I0319 12:36:58.343841 1 csi_handler.go:264] Attached
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:58.343848 1 util.go:38] Marking as attached
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7":
>>> failed to mark as attached: volumeattachments.storage.k8s.io
>>> <http://volumeattachments.storage.k8s.io>
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is
>>> forbidden: User
>>> "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch
>>> resource "volumeattachments/status" in API group "storage.k8s.io
>>> <http://storage.k8s.io>" at the cluster scope*
>>> I0319 12:36:58.348503 1 controller.go:210] Started VA processing
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing
>>> VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:58.348513 1 csi_handler.go:251] Attaching
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach
>>> operation for
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID
>>> 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode
>>> k8intcalnewer-56bgom6jntbm-node-0
>>> I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is
>>> already set on
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is
>>> already set on
>>> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7"
>>> I0319 12:36:58.348564 1 connection.go:193] GRPC call:
>>> /csi.v1.Controller/ControllerPublishVolume
>>> I0319 12:36:58.348567 1 connection.go:194] GRPC request:
>>> {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext:
>>>
>>>
>>> The I tried even the most updated images :
>>>
>>>> 10.0.0.165:4000/cinder-csi-plugin:v1.26.2
>>>> 10.0.0.165:4000/csi-provisioner:v3.4.0
>>>> 10.0.0.165:4000/csi-resizer:v1.7.0
>>>> 10.0.0.165:4000/csi-snapshotter:v6.2.1
>>>> 10.0.0.165:4000/csi-attacher:v4.2.0
>>>> 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0
>>>>
>>>
>>> I had the same problem.
>>>
>>> Then I tried to use an older version of kubernetes : 1.21.11 with the
>>> older images shown above (following this link
>>> https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minutes-with-magnum/),
>>> and it worked, the cinder volume was successfully mounted inside my nginx
>>> pod.
>>>
>>>
>>>
>>> - What is the meaning of the error I am having?
>>> - Is it magnum related or kubernetes related or both?
>>>
>>>
>>> Regards.
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230320/8cfa4d09/attachment-0001.htm>
More information about the openstack-discuss
mailing list