Hi Jake,

I stepped over some more issues now that I have solved the snapshottter issue in Yoga. When creating a pvc I can see that a volume is created by cinder but it is not binding to the pod. I checked the logs of csi-cinder-controllerplugin-0 and I see this:

I0224 10:48:43.073514       1 controller.go:208] Started VA processing "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.073981       1 csi_handler.go:221] CSIHandler: processing VA "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.074231       1 csi_handler.go:248] Attaching "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.074246       1 csi_handler.go:427] Starting attach operation for "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.074663       1 csi_handler.go:341] PV finalizer is already set on "pvc-09ce4510-5c63-41f4-9a49-b9ff759a7c24"
I0224 10:48:43.074940       1 csi_handler.go:746] Found NodeID 84ffe846-a06f-40fb-a91d-f9762f8fc1bd in CSINode simple-ig2ayexh5trl-node-0
I0224 10:48:43.075170       1 csi_handler.go:301] VA finalizer is already set on "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.075446       1 csi_handler.go:315] NodeID annotation is already set on "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.075701       1 connection.go:183] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0224 10:48:43.075924       1 connection.go:184] GRPC request: {"node_id":"84ffe846-a06f-40fb-a91d-f9762f8fc1bd","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"storage.kubernetes.io/csiProvisionerIdentity":"1677232452553-8081-cinder.csi.openstack.org"},"volume_id":"32a290d1-1d77-43a0-83ee-3f00e3789459"}
I0224 10:48:43.475947       1 connection.go:186] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}}
I0224 10:48:43.476804       1 connection.go:187] GRPC error: <nil>
I0224 10:48:43.477403       1 csi_handler.go:261] Attached "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.477735       1 util.go:38] Marking as attached "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"
I0224 10:48:43.488629       1 csi_handler.go:231] Error processing "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6": failed to mark as attached: volumeattachments.storage.k8s.io "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io" at the cluster scope

I'm not sure if this is because I'm not using containerd. I can't use containerd since this doesn't like my local unsecure registry.

Also what is the proper way to report these sightings as bugs? Storyboard.openstack.org?

Best Regards,
Oliver

Oliver Weinmann <oliver.weinmann@me.com> schrieb am 14. Feb. 2023 um 21:34:


Hi Jake,

Many thanks for the list. Using my local registry I can at least deploy a K8s cluster, but there is still an issue I think with the default version of the csi_snapshotter container in stable/yoga and stable/zed. I had a look at the magnum source code and v4.2.1 is only defined in master:


type: string

description: tag of csi snapshotter
default: v4.2.1

csi_snapshotter_tag:

That would explain why it used to be working in my staging openstack kolla-ansible cluster. Because I deployed latest and not a specific release. At least in my case I'm not able to deploy a fully functional K8s cluster without specifying the csi_snapshotter tag version of v4.2.1 and using a local registry. By just specifying the csi snapshotter tag it will try to download the container image from the wrong source. Can someone from the magnum team please check and clarify?

Cheers,

Oliver

Am 14.02.2023 um 13:57 schrieb Jake Yip:
On 11/2/2023 5:57 am, Oliver Weinmann wrote:


BTW for the local registry, It seems that the list of images needed described here:


is incomplete and outdated. I had to pull a couple more. No complaint, just mentioning it. :)


Yeah that may be incomplete. Maybe you can reference the following which works for us.

Labels

[
              "kube_tag=v1.23.8",
              "flannel_tag=v0.18.1",
              "master_lb_floating_ip_enabled=true",
              "cinder_csi_enabled=true",
              "docker_volume_type=standard",
              "ingress_controller=octavia",
              "container_runtime=containerd",
              "containerd_version=1.6.6",

"containerd_tarball_sha256=a64568c8ce792dd73859ce5f336d5485fcbceab15dc3e06d5d1bc1c3353fa20f",
              "cloud_provider_tag=v1.23.4",
              "cinder_csi_plugin_tag=v1.23.4",
              "k8s_keystone_auth_tag=v1.23.4",
              "magnum_auto_healer_tag=v1.23.4",
              "octavia_ingress_controller_tag=v1.23.4",
              "autoscaler_tag=v1.23.0",
              "coredns_tag=1.9.3",
              "csi_snapshotter_tag=v4.2.1",
              "csi_attacher_tag=v3.3.0",
              "csi_resizer_tag=v1.3.0",
              "csi_provisioner_tag=v3.0.0",
              "csi_node_driver_registrar_tag=v2.4.0",
             ]


Containers

k8scloudprovider/openstack-cloud-controller-manager:v1.23.4
k8scloudprovider/k8s-keystone-auth:v1.23.4
k8scloudprovider/cinder-csi-plugin:v1.23.4
k8scloudprovider/magnum-auto-healer:v1.23.4
k8scloudprovider/octavia-ingress-controller:v1.23.4
k8s.gcr.io/sig-storage/csi-attacher:v3.3.0
k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
k8s.gcr.io/sig-storage/csi-resizer:v1.3.0
k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1
k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
k8s.gcr.io/sig-storage/livenessprobe:v2.5.0
openstackmagnum/cluster-autoscaler:v1.22.0


Regards,
Jake