<html><body><div><div><br></div><div>Hi Jake,<br></div><div><br></div><div>I stepped over some more issues now that I have solved the snapshottter issue in Yoga. When creating a pvc I can see that a volume is created by cinder but it is not binding to the pod. I checked the logs of csi-cinder-controllerplugin-0 and I see this:<br></div><div><br></div><div>I0224 10:48:43.073514 1 controller.go:208] Started VA processing "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div></div><div>I0224 10:48:43.073981 1 csi_handler.go:221] CSIHandler: processing VA "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div><div>I0224 10:48:43.074231 1 csi_handler.go:248] Attaching "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div><div>I0224 10:48:43.074246 1 csi_handler.go:427] Starting attach operation for "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div><div>I0224 10:48:43.074663 1 csi_handler.go:341] PV finalizer is already set on "pvc-09ce4510-5c63-41f4-9a49-b9ff759a7c24"<br></div><div>I0224 10:48:43.074940 1 csi_handler.go:746] Found NodeID 84ffe846-a06f-40fb-a91d-f9762f8fc1bd in CSINode simple-ig2ayexh5trl-node-0<br></div><div>I0224 10:48:43.075170 1 csi_handler.go:301] VA finalizer is already set on "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div><div>I0224 10:48:43.075446 1 csi_handler.go:315] NodeID annotation is already set on "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div><div>I0224 10:48:43.075701 1 connection.go:183] GRPC call: /<a rel="noopener noreferrer" href="http://csi.v1.Controller/ControllerPublishVolume">csi.v1.Controller/ControllerPublishVolume</a><br></div><div>I0224 10:48:43.075924 1 connection.go:184] GRPC request: {"node_id":"84ffe846-a06f-40fb-a91d-f9762f8fc1bd","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"<a rel="noopener noreferrer" href="http://storage.kubernetes.io/csiProvisionerIdentity":"1677232452553-8081-cinder.csi.openstack.org"},"volume_id":"32a290d1-1d77-43a0-83ee-3f00e3789459">storage.kubernetes.io/csiProvisionerIdentity":"1677232452553-8081-cinder.csi.openstack.org"},"volume_id":"32a290d1-1d77-43a0-83ee-3f00e3789459</a>"}<br></div><div>I0224 10:48:43.475947 1 connection.go:186] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}}<br></div><div>I0224 10:48:43.476804 1 connection.go:187] GRPC error: <nil><br></div><div>I0224 10:48:43.477403 1 csi_handler.go:261] Attached "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div><div>I0224 10:48:43.477735 1 util.go:38] Marking as attached "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6"<br></div><div>I0224 10:48:43.488629 1 csi_handler.go:231] Error processing "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6": failed to mark as attached: volumeattachments.storage.k8s.io "csi-68dcbe9863b1c836bcac55d4d62e3af5864ae45eb515f31e3eea2a29e6ff84b6" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io" at the cluster scope<br></div><div><div><br></div><div>I'm not sure if this is because I'm not using containerd. I can't use containerd since this doesn't like my local unsecure registry.<br></div><div><br></div><div>Also what is the proper way to report these sightings as bugs? Storyboard.openstack.org?<br></div><div><br></div><div>Best Regards,<br></div><div>Oliver<br></div><div><br></div><blockquote type="cite"><div>Oliver Weinmann <oliver.weinmann@me.com> schrieb am 14. Feb. 2023 um 21:34:<br></div><div><br></div><div><br></div><div><p>Hi Jake,<br></p><p>Many thanks for the list. Using my local registry I can at least
deploy a K8s cluster, but there is still an issue I think with the
default version of the csi_snapshotter container in stable/yoga
and stable/zed. I had a look at the magnum source code and v4.2.1
is only defined in master:<br></p><br><table data-tagsearch-path="magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml" data-tagsearch-lang="YAML" data-paste-markdown-skip="" data-tab-size="8" class="highlight tab-size js-file-line-container
js-code-nav-container js-tagsearch-file" data-hpc=""><tbody><tr><td class="blob-code blob-code-inner js-file-line" id="LC882"><span class="pl-ent">type</span>: <span class="pl-s">string</span><br></td></tr><tr></tr></tbody></table><br><table data-tagsearch-path="magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml" data-tagsearch-lang="YAML" data-paste-markdown-skip="" data-tab-size="8" class="highlight tab-size js-file-line-container
js-code-nav-container js-tagsearch-file" data-hpc=""><tbody><tr><td class="blob-code blob-code-inner js-file-line" id="LC883"><span class="pl-ent">description</span>: <span class="pl-s">tag
of csi snapshotter</span><br></td></tr><tr></tr></tbody></table><div><span class="pl-ent">default</span>: <span class="pl-s">v4.2.1</span> <br></div><br><table data-tagsearch-path="magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml" data-tagsearch-lang="YAML" data-paste-markdown-skip="" data-tab-size="8" class="highlight tab-size js-file-line-container
js-code-nav-container js-tagsearch-file" data-hpc=""><tbody><tr><td class="blob-code blob-code-inner js-file-line" id="LC881"><span class="pl-ent">csi_snapshotter_tag</span>:<br></td></tr><tr></tr></tbody></table><p>That would explain why it used to be working in my staging
openstack kolla-ansible cluster. Because I deployed latest and not
a specific release. At least in my case I'm not able to deploy a
fully functional K8s cluster without specifying the
csi_snapshotter tag version of v4.2.1 and using a local registry.
By just specifying the csi snapshotter tag it will try to download
the container image from the wrong source. Can someone from the
magnum team please check and clarify?<br></p><p>Cheers,<br></p><p>Oliver<br></p><div class="moz-cite-prefix">Am 14.02.2023 um 13:57 schrieb Jake
Yip:<br></div><blockquote type="cite"><div>On
11/2/2023 5:57 am, Oliver Weinmann wrote: <br></div><div> <br></div><blockquote type="cite"><div><br></div><div>BTW for the local registry, It seems that the list of images
needed described here:<br></div><div> <br></div><div> <a rel="noopener noreferrer" href="https://docs.openstack.org/magnum/yoga/user/" class="moz-txt-link-freetext">https://docs.openstack.org/magnum/yoga/user/</a> <br></div><div> <br></div><div> is incomplete and outdated. I had to pull a couple more. No
complaint, just mentioning it. :) <br></div><div> <br></div></blockquote><div><br></div><div>Yeah that may be incomplete. Maybe you can reference the following
which works for us.<br></div><div> <br></div><div> Labels <br></div><div> <br></div><div> [ <br></div><div> "kube_tag=v1.23.8", <br></div><div> "flannel_tag=v0.18.1", <br></div><div> "master_lb_floating_ip_enabled=true", <br></div><div> "cinder_csi_enabled=true", <br></div><div> "docker_volume_type=standard", <br></div><div> "ingress_controller=octavia", <br></div><div> "container_runtime=containerd", <br></div><div> "containerd_version=1.6.6", <br></div><div> <br></div><div> "containerd_tarball_sha256=a64568c8ce792dd73859ce5f336d5485fcbceab15dc3e06d5d1bc1c3353fa20f", <br></div><div> "cloud_provider_tag=v1.23.4", <br></div><div> "cinder_csi_plugin_tag=v1.23.4", <br></div><div> "k8s_keystone_auth_tag=v1.23.4", <br></div><div> "magnum_auto_healer_tag=v1.23.4", <br></div><div> "octavia_ingress_controller_tag=v1.23.4", <br></div><div> "autoscaler_tag=v1.23.0", <br></div><div> "coredns_tag=1.9.3", <br></div><div> "csi_snapshotter_tag=v4.2.1", <br></div><div> "csi_attacher_tag=v3.3.0", <br></div><div> "csi_resizer_tag=v1.3.0", <br></div><div> "csi_provisioner_tag=v3.0.0", <br></div><div> "csi_node_driver_registrar_tag=v2.4.0", <br></div><div> ] <br></div><div> <br></div><div> <br></div><div> Containers <br></div><div> <br></div><div> k8scloudprovider/openstack-cloud-controller-manager:v1.23.4 <br></div><div> k8scloudprovider/k8s-keystone-auth:v1.23.4 <br></div><div> k8scloudprovider/cinder-csi-plugin:v1.23.4 <br></div><div> k8scloudprovider/magnum-auto-healer:v1.23.4 <br></div><div> k8scloudprovider/octavia-ingress-controller:v1.23.4 <br></div><div> k8s.gcr.io/sig-storage/csi-attacher:v3.3.0 <br></div><div> k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0 <br></div><div> k8s.gcr.io/sig-storage/csi-resizer:v1.3.0 <br></div><div> k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1 <br></div><div> k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0 <br></div><div> k8s.gcr.io/sig-storage/livenessprobe:v2.5.0 <br></div><div> openstackmagnum/cluster-autoscaler:v1.22.0 <br></div><div> <br></div><div> <br></div><div> Regards, <br></div><div> Jake <br></div></blockquote></div></blockquote><div><br></div></div><div><br></div></body></html>