Hi,

For the record, it seems I managed to fix the CSI issue to create K8s 1.23 clusters with Yoga Magnum (K8s 1.23 being the supported version, according to the release notes). With the information provided by Oliver and a few others posts, I understood that I should try to run much newer CSI sidecars than those provided by Yoga Magnum as sidecar versions tend to be bound more to K8s versions than Magnum ones.

As I  didn't have a local registry already setup, I decided to patch /usr/lib/python3.6/site-packages/magnum/drivers/common/templates/kubernetes/fragments/enable-cinder-csi.sh to use "registry.k8s.io/sig-storage/" as the default registry source for sidecars to be able to use recent versions. The version I use are (defined as labels in cluster template):

With these CSI sidecar versions, I was able to get the CSI-related pods but had to fix the cluster role csi-snapshotter-role (in the .sh file mentioned above) to add the right to patch "volumeattachments/status" (without this the error message is explicit in the pod csi-cinder-controllerplugin-0 log.

With these mods, I have been able to create several ephemeral and persistent volumes so it seems it is working. There may be additional details I have not seen but I'm confident that it will possible to solve them, should they arise...

If it helps, I attach the patch for enable-cinder-csi.sh without any warranty that it doesn't hurt in some situations (in particular it probably breaks templates using older versions of K8s without the appropriate labels as the old versions of the sidecars will not be found in registry.k8s.io)... If you plan to use it, be aware that it is at your own risk... Note that the only required part of the patch is the one about the clusterrole: a cleaner solution for installing the last version of the sidecars would be to have a local registry and copy the required versions in it.

Best regards,

Michel

Hi Michel,

Yes a local registry can be helpful. I highly recommend upgrading to a newer Openstack release or try if you can patch magnum in Yoga to use CAPI. It is so much better and just works.

I was able to patch it in 2023.1 and 2023.2.

* https://www.roksblog.de/openstack-magnum-cluster-api-driver/

This will not help you with existing clusters but it is absolutely helpful for any new cluster that you deploy.

You can easily migrate your pods from the old clusters to the new ones deployed with CAPI using Velero.

* https://www.roksblog.de/kubernetes-backup-with-velero/

Have a nice weekend

Von meinem iPhone gesendet

Am 16.03.2024 um 11:36 schrieb Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:



Oliver,

Thanks for this helpful information. We always delayed having a local registry but I think you are right, we probably need to do it to make easier  working around problems like the ones we have with Magnum.

Michel

Le 16/03/2024 à 07:21, Oliver Weinmann a écrit :

Hi Michel,

I found my old post again:

* https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032149.html

I think you need to create a local registry where you pull all the docker images upfront, since magnum tried to pull the csi snapshotter image from a wrong source. Also your k8s cluster might need containerd.

For the local registry you can check my other blogposts:

* https://www.roksblog.de/openstack-magnum-insecure-registry/

Hope this helps. Unfortunately I don’t have a yoga cluster up and running at the moment to test it.

Von meinem iPhone gesendet

Am 15.03.2024 um 18:38 schrieb Oliver Weinmann <oliver.weinmann@me.com>:

 Hi Michel,

Sorry just read my post again. I never managed to fix 1.23 under yoga. I remember asking the same question to the mailing list back then and the solutions provided didn’t work.

I think you need to change a lot more tags than just snapshotter.

Von meinem iPhone gesendet

Am 15.03.2024 um 18:32 schrieb Oliver Weinmann <oliver.weinmann@me.com>:

 Hi Michel,

It’s been quite some time but as far as I can remember I only changed the snapshotter tag.

What FedoraCoreOs version and k8s are you using?

Cheers,

Oliver

Von meinem iPhone gesendet

Am 15.03.2024 um 18:05 schrieb Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:



Hi Oliver,

Thanks for your excellent post! You described very well everything that need to be done... and all the steps I went through... But I have not seen how you fixed the CSI problem. Is it enough to define the csi_snapshotter_tag? I tried this this morning but was not able to find the version I was supposed to use.

BTW, I see that you are using flannel as the network driver. I'm using calico, not sure it makes any difference for this problem anyway.

Cheers,

Michel

Le 15/03/2024 à 17:09, Oliver Weinmann a écrit :

Hi Michel,

Maybe my old blogpost can help you:

* https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minutes-with-magnum/

Best regards,

Oliver

Von meinem iPhone gesendet

Am 15.03.2024 um 16:04 schrieb Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

Hi,

We recently upgraded our cluster to Yoga and since then we cannot successfully start pods in clusters using K8s 1.23 that require a volume. The volume is properly created but attachment fails because it is trying to use v1beta1.CSINode and v1beta1.VolumeAttachment that no longer exists. I found a reference to this in https://github.com/kubernetes/cloud-provider-openstack/issues/1845 but the way to fix it is unclear. I tried to use last version of CSI-related stuff from registry.k8s.io (playing with labels and source), but I then got another problem which may be related (it is my guess) to the fact that I'm using too recent versions.

Is somebody using sucessfully Magnum Yoga/K8bs 1.23 combination sucessfully and what is the trick to do it?

Thanks in advance for any help. Best regards,

Michel