[kolla-ansible] [yoga] [magnum] [k8s] cannot attach persistent volume to pod
Hello,
When I try to attach a persistent cinder volume to a pod, I get FailedMount and FailedAttachVolume timeout events. I also get these errors in the log of the csi-cinder-controllerplugin-0 pod: E1020 13:38:41.747511 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.VolumeAttachment: the server could not find the requested resource E1020 13:38:41.748187 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSINode: the server could not find the requested resource I fixed a CrashLoopBackoff error with the csi-snapshotter container in the csi-cinder-controllerplugin-0 pod by providing the label "csi_snapshotter_tag=v4.0.0" when I created the cluster template. I found that suggestion in an issue on the GitHub cloud-provider-openstack project.
I'm not finding any help with this error on Google.
Thanks, -Stu _____________________________________ The information contained in this e-mail and any attachments from Group W may contain confidential and/or proprietary information and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, be aware that any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately of that fact by return e-mail and permanently delete the e-mail and any attachments to it.
I fixed this by replacing the csi-cinder-controllerplugin and the csi-nodeplugin using the manifests found in this project: https://github.com/kubernetes/cloud-provider-openstack.
I used kubectl to make the changes. Does anyone know if end-users can configure these kinds of changes when kolla-ansible installs magnum? Or when creating the cluster template?
Thanks, -Stu ________________________________ From: Stuart Whitman Sent: Thursday, October 20, 2022 10:09 AM To: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: [kolla-ansible] [yoga] [magnum] [k8s] cannot attach persistent volume to pod
Hello,
When I try to attach a persistent cinder volume to a pod, I get FailedMount and FailedAttachVolume timeout events. I also get these errors in the log of the csi-cinder-controllerplugin-0 pod: E1020 13:38:41.747511 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.VolumeAttachment: the server could not find the requested resource E1020 13:38:41.748187 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSINode: the server could not find the requested resource I fixed a CrashLoopBackoff error with the csi-snapshotter container in the csi-cinder-controllerplugin-0 pod by providing the label "csi_snapshotter_tag=v4.0.0" when I created the cluster template. I found that suggestion in an issue on the GitHub cloud-provider-openstack project.
I'm not finding any help with this error on Google.
Thanks, -Stu _____________________________________ The information contained in this e-mail and any attachments from Group W may contain confidential and/or proprietary information and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, be aware that any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately of that fact by return e-mail and permanently delete the e-mail and any attachments to it.
Hi,
sorry to pick up this old post, but I faced the very same issue on multiple ocassions, but I remembered that I had this issue before and how I fixed it. But the questions still remains. I have a fresh kolla-ansible yoga magnum deployment with latest Fedora CoreOS 35 image and I always fail to deploy a K8s cluster:
kubectl get pods -A
kube-system csi-cinder-controllerplugin-0 4/5 CrashLoopBackOff 6 (4m54s ago) 14m
I can fix it by specifying a label of:
csi_snapshot_tag=v4.0.1
But this only works If I use a local registry where I download plenty of container images from multipe sources.
It failed when not using a local registry since magnum tries to download the csi_snapshotter docker image from a different source:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 13m default-scheduler Successfully assigned kube-system/csi-cinder-controllerplugin-0 to k8s-flannel-small-35-mi6fcba4mpdc-master-0 Warning NetworkNotReady 13m (x24 over 13m) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Normal Pulling 13m kubelet Pulling image "quay.io/k8scsi/csi-attacher:v2.0.0" Normal BackOff 3m43s (x33 over 11m) kubelet Back-off pulling image "quay.io/k8scsi/csi-snapshotter:v4.0.1"
BTW for the local registry, It seems that the list of images needed described here:
https://docs.openstack.org/magnum/yoga/user/
is incomplete and outdated. I had to pull a couple more. No complaint, just mentioning it. :)
I wonder if the issue is known and if it is if there is note for it in the release notes for magnum under yoga?
I have not tried ZED yet since my OS is Rocky Linux 8.x and there seems to be now upgrade path available yet.
Cheers and a nice weekend
Oliver
Am 21.10.2022 um 22:11 schrieb Stuart Whitman:
[kolla-ansible] [yoga] [magnum] [k8s]
On 11/2/2023 5:57 am, Oliver Weinmann wrote:
BTW for the local registry, It seems that the list of images needed described here:
https://docs.openstack.org/magnum/yoga/user/
is incomplete and outdated. I had to pull a couple more. No complaint, just mentioning it. :)
Yeah that may be incomplete. Maybe you can reference the following which works for us.
Labels
[ "kube_tag=v1.23.8", "flannel_tag=v0.18.1", "master_lb_floating_ip_enabled=true", "cinder_csi_enabled=true", "docker_volume_type=standard", "ingress_controller=octavia", "container_runtime=containerd", "containerd_version=1.6.6",
"containerd_tarball_sha256=a64568c8ce792dd73859ce5f336d5485fcbceab15dc3e06d5d1bc1c3353fa20f", "cloud_provider_tag=v1.23.4", "cinder_csi_plugin_tag=v1.23.4", "k8s_keystone_auth_tag=v1.23.4", "magnum_auto_healer_tag=v1.23.4", "octavia_ingress_controller_tag=v1.23.4", "autoscaler_tag=v1.23.0", "coredns_tag=1.9.3", "csi_snapshotter_tag=v4.2.1", "csi_attacher_tag=v3.3.0", "csi_resizer_tag=v1.3.0", "csi_provisioner_tag=v3.0.0", "csi_node_driver_registrar_tag=v2.4.0", ]
Containers
k8scloudprovider/openstack-cloud-controller-manager:v1.23.4 k8scloudprovider/k8s-keystone-auth:v1.23.4 k8scloudprovider/cinder-csi-plugin:v1.23.4 k8scloudprovider/magnum-auto-healer:v1.23.4 k8scloudprovider/octavia-ingress-controller:v1.23.4 k8s.gcr.io/sig-storage/csi-attacher:v3.3.0 k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0 k8s.gcr.io/sig-storage/csi-resizer:v1.3.0 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0 k8s.gcr.io/sig-storage/livenessprobe:v2.5.0 openstackmagnum/cluster-autoscaler:v1.22.0
Regards, Jake
Hi Jake,
Many thanks for the list. Using my local registry I can at least deploy a K8s cluster, but there is still an issue I think with the default version of the csi_snapshotter container in stable/yoga and stable/zed. I had a look at the magnum source code and v4.2.1 is only defined in master:
type: string
description: tag of csi snapshotter
default: v4.2.1 csi_snapshotter_tag:
That would explain why it used to be working in my staging openstack kolla-ansible cluster. Because I deployed latest and not a specific release. At least in my case I'm not able to deploy a fully functional K8s cluster without specifying the csi_snapshotter tag version of v4.2.1 and using a local registry. By just specifying the csi snapshotter tag it will try to download the container image from the wrong source. Can someone from the magnum team please check and clarify?
Cheers,
Oliver
Am 14.02.2023 um 13:57 schrieb Jake Yip:
On 11/2/2023 5:57 am, Oliver Weinmann wrote:
BTW for the local registry, It seems that the list of images needed described here:
https://docs.openstack.org/magnum/yoga/user/
is incomplete and outdated. I had to pull a couple more. No complaint, just mentioning it. :)
Yeah that may be incomplete. Maybe you can reference the following which works for us.
Labels
[ "kube_tag=v1.23.8", "flannel_tag=v0.18.1", "master_lb_floating_ip_enabled=true", "cinder_csi_enabled=true", "docker_volume_type=standard", "ingress_controller=octavia", "container_runtime=containerd", "containerd_version=1.6.6",
"containerd_tarball_sha256=a64568c8ce792dd73859ce5f336d5485fcbceab15dc3e06d5d1bc1c3353fa20f",
"cloud_provider_tag=v1.23.4", "cinder_csi_plugin_tag=v1.23.4", "k8s_keystone_auth_tag=v1.23.4", "magnum_auto_healer_tag=v1.23.4", "octavia_ingress_controller_tag=v1.23.4", "autoscaler_tag=v1.23.0", "coredns_tag=1.9.3", "csi_snapshotter_tag=v4.2.1", "csi_attacher_tag=v3.3.0", "csi_resizer_tag=v1.3.0", "csi_provisioner_tag=v3.0.0", "csi_node_driver_registrar_tag=v2.4.0", ]
Containers
k8scloudprovider/openstack-cloud-controller-manager:v1.23.4 k8scloudprovider/k8s-keystone-auth:v1.23.4 k8scloudprovider/cinder-csi-plugin:v1.23.4 k8scloudprovider/magnum-auto-healer:v1.23.4 k8scloudprovider/octavia-ingress-controller:v1.23.4 k8s.gcr.io/sig-storage/csi-attacher:v3.3.0 k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0 k8s.gcr.io/sig-storage/csi-resizer:v1.3.0 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0 k8s.gcr.io/sig-storage/livenessprobe:v2.5.0 openstackmagnum/cluster-autoscaler:v1.22.0
Regards, Jake
Hi Jake,I stepped over some more issues now that I have solved the snapshottter issue in Yoga. When creating a pvc I can see that a volume is created by cinder but it is not binding to the pod. I checked the logs of csi-cinder-controllerplugin-0 and I see this:I0224 10:48:43.073514 1 controller.go:208] Started VA processing "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.073981 1 csi_handler.go:221] CSIHandler: processing VA "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.074231 1 csi_handler.go:248] Attaching "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.074246 1 csi_handler.go:427] Starting attach operation for "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.074663 1 csi_handler.go:341] PV finalizer is already set on "pvc-09ce4510-5c63-41f4-9a49-b9ff759a7c24"I0224 10:48:43.074940 1 csi_handler.go:746] Found NodeID 84ffe846-a06f-40fb-a91d-f9762f8fc1bd in CSINode simple-ig2ayexh5trl-node-0I0224 10:48:43.075170 1 csi_handler.go:301] VA finalizer is already set on "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.075446 1 csi_handler.go:315] NodeID annotation is already set on "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.075701 1 connection.go:183] GRPC call: /csi.v1.Controller/ControllerPublishVolumeI0224 10:48:43.075924 1 connection.go:184] GRPC request: {"node_id":"84ffe846-a06f-40fb-a91d-f9762f8fc1bd","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"storage.kubernetes.io/csiProvisionerIdentity":"1677232452553-8081-cinder.csi.openstack.org"},"volume_id":"32a290d1-1d77-43a0-83ee-3f00e3789459"}I0224 10:48:43.475947 1 connection.go:186] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}}I0224 10:48:43.476804 1 connection.go:187] GRPC error: <nil>I0224 10:48:43.477403 1 csi_handler.go:261] Attached "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.477735 1 util.go:38] Marking as attached "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"I0224 10:48:43.488629 1 csi_handler.go:231] Error processing "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6": failed to mark as attached: volumeattachments.storage.k8s.io "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io" at the cluster scopeI'm not sure if this is because I'm not using containerd. I can't use containerd since this doesn't like my local unsecure registry.Also what is the proper way to report these sightings as bugs? Storyboard.openstack.org?Best Regards,OliverOliver Weinmann oliver.weinmann@me.com schrieb am 14. Feb. 2023 um 21:34:Hi Jake,Many thanks for the list. Using my local registry I can at least deploy a K8s cluster, but there is still an issue I think with the default version of the csi_snapshotter container in stable/yoga and stable/zed. I had a look at the magnum source code and v4.2.1 is only defined in master:type: stringdescription: tag of csi snapshotterdefault: v4.2.1 csi_snapshotter_tag:That would explain why it used to be working in my staging openstack kolla-ansible cluster. Because I deployed latest and not a specific release. At least in my case I'm not able to deploy a fully functional K8s cluster without specifying the csi_snapshotter tag version of v4.2.1 and using a local registry. By just specifying the csi snapshotter tag it will try to download the container image from the wrong source. Can someone from the magnum team please check and clarify?Cheers,OliverAm 14.02.2023 um 13:57 schrieb Jake Yip:On 11/2/2023 5:57 am, Oliver Weinmann wrote: BTW for the local registry, It seems that the list of images needed described here: https://docs.openstack.org/magnum/yoga/user/ is incomplete and outdated. I had to pull a couple more. No complaint, just mentioning it. :) Yeah that may be incomplete. Maybe you can reference the following which works for us. Labels [ "kube_tag=v1.23.8", "flannel_tag=v0.18.1", "master_lb_floating_ip_enabled=true", "cinder_csi_enabled=true", "docker_volume_type=standard", "ingress_controller=octavia", "container_runtime=containerd", "containerd_version=1.6.6", "containerd_tarball_sha256=a64568c8ce792dd73859ce5f336d5485fcbceab15dc3e06d5d1bc1c3353fa20f", "cloud_provider_tag=v1.23.4", "cinder_csi_plugin_tag=v1.23.4", "k8s_keystone_auth_tag=v1.23.4", "magnum_auto_healer_tag=v1.23.4", "octavia_ingress_controller_tag=v1.23.4", "autoscaler_tag=v1.23.0", "coredns_tag=1.9.3", "csi_snapshotter_tag=v4.2.1", "csi_attacher_tag=v3.3.0", "csi_resizer_tag=v1.3.0", "csi_provisioner_tag=v3.0.0", "csi_node_driver_registrar_tag=v2.4.0", ] Containers k8scloudprovider/openstack-cloud-controller-manager:v1.23.4 k8scloudprovider/k8s-keystone-auth:v1.23.4 k8scloudprovider/cinder-csi-plugin:v1.23.4 k8scloudprovider/magnum-auto-healer:v1.23.4 k8scloudprovider/octavia-ingress-controller:v1.23.4 k8s.gcr.io/sig-storage/csi-attacher:v3.3.0 k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0 k8s.gcr.io/sig-storage/csi-resizer:v1.3.0 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0 k8s.gcr.io/sig-storage/livenessprobe:v2.5.0 openstackmagnum/cluster-autoscaler:v1.22.0 Regards, Jake
participants (3)
-
Jake Yip
-
Oliver Weinmann
-
Stuart Whitman