Hi all,

a couple of weeks ago I updated the containers in my 2023.2 release.

kolla-ansible pull

kolla-ansible deploy

since then the Vexxhost driver in Magnum is no longer working.

I have clusters stuck in provisioning and deleting:

$ kubectl get cluster -A
NAMESPACE       NAME         CLUSTERCLASS     PHASE          AGE   VERSION
magnum-system   kube-6r0du   magnum-v0.26.0   Provisioning   20d   v1.26.7
magnum-system   kube-7ljg3   magnum-v0.26.0   Provisioning   18d   v1.26.7
magnum-system   kube-csf6s   magnum-v0.13.4   Deleting       71d   v1.26.7

I believe this is due to a versionn mismatch. Many different tools are involved here.

All pods in the KIND Management cluster are running:

$ kubectl -n kube-system get pods
NAME                                               READY   STATUS    RESTARTS         AGE
coredns-76f75df574-lfckv                           1/1     Running   5 (2d23h ago)    71d
coredns-76f75df574-n85fv                           1/1     Running   5 (2d23h ago)    71d
etcd-magnumcapi-control-plane                      1/1     Running   5 (2d23h ago)    71d
kindnet-jtdtm                                      1/1     Running   5 (2d23h ago)    71d
kube-apiserver-magnumcapi-control-plane            1/1     Running   5 (2d23h ago)    71d
kube-controller-manager-magnumcapi-control-plane   1/1     Running   10 (2d23h ago)   71d
kube-proxy-qwjc8                                   1/1     Running   5 (2d23h ago)    71d
kube-scheduler-magnumcapi-control-plane            1/1     Running   9 (2d23h ago)    71d

There is nothing obvious in the capi or capo logs.

Is there someone that is currently running 2023.2 or newer with the Vexxhost CAPI driver?

Would  be great if you could share the versions of:

Management Cluster

$ kind --version
kind version 0.22.0
$ export KUBECONFIG=~/config
$ kubectl get nodes -o wide
NAME                       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                 CONTAINER-RUNTIME
magnumcapi-control-plane   Ready    control-plane   71d   v1.29.2   172.18.0.2    <none>        Debian GNU/Linux 12 (bookworm)   5.14.0-503.38.1.el9_5.x86_64   containerd://1.7.13
$ kubectl get provider -A
NAMESPACE                           NAME                       AGE   TYPE                     PROVIDER      VERSION
capi-kubeadm-bootstrap-system       bootstrap-kubeadm          71d   BootstrapProvider        kubeadm       v1.9.5
capi-kubeadm-control-plane-system   control-plane-kubeadm      71d   ControlPlaneProvider     kubeadm       v1.9.5
capi-system                         cluster-api                71d   CoreProvider             cluster-api   v1.9.5
capo-system                         infrastructure-openstack   71d   InfrastructureProvider   openstack     v0.10.8

The magnum pods:

~]# podman exec -it magnum_conductor bash
()[ /]$ pip list | grep api
magnum-cluster-api      0.26.0
()[ /]$ helm version
version.BuildInfo{Version:"v3.13.2", GitCommit:"2a2fb3b98829f1e0be6fb18af2f6599e0f4e8243", GitTreeState:"clean", GoVersion:"go1.20.10"}

~]# podman exec -it magnum_api bash

()[ /]$ pip list | grep api
magnum-cluster-api      0.26.0
()[ /]$ helm version
version.BuildInfo{Version:"v3.13.2", GitCommit:"2a2fb3b98829f1e0be6fb18af2f6599e0f4e8243", GitTreeState:"clean", GoVersion:"go1.20.10"}
()[ /]$

Cheers,

Oliver