Hi Bharat / Feilong, The problem look like in ubuntu wallaby repo's magnum DEB package. The package does not have all the changes that a wallaby release should contain. Like heat_container_agent_tag that should be wallaby-stable-1. In that package it was still pointing to victoria-dev. I have cloned stable/wallaby from https://opendev.org/openstack/magnum.git and replaced magnum directory in /lib/python3/dist-packages with downloaded files in stable/magnum branch. Now everything is working a expected. [root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-cluster-2zcsd5n6qnre-master-0 Ready master 5m43s v1.19.10 k8s-cluster-2zcsd5n6qnre-node-0 Ready <none> 39s v1.19.10 [root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-57999d5467-2l46z 1/1 Running 0 5m43s coredns-57999d5467-rn8d9 1/1 Running 0 5m43s csi-cinder-controllerplugin-0 5/5 Running 0 5m36s csi-cinder-nodeplugin-nmjnb 2/2 Running 0 30s dashboard-metrics-scraper-7b59f7d4df-mzmb6 1/1 Running 0 5m41s k8s-keystone-auth-knhk7 1/1 Running 0 5m39s kube-dns-autoscaler-f57cd985f-7dqj5 1/1 Running 0 5m43s kube-flannel-ds-8d7pc 1/1 Running 0 5m42s kube-flannel-ds-pcncq 1/1 Running 0 60s kubernetes-dashboard-7fb447bf79-x6kvj 1/1 Running 0 5m41s npd-hcw25 1/1 Running 0 30s openstack-cloud-controller-manager-x759f 1/1 Running 0 5m45s - Ammad On Mon, May 3, 2021 at 11:13 PM Bharat Kunwar <bharat@stackhpc.com> wrote:

Can you try the calico plugin? The flannel plug-in has been unmaintained for a while.

Sent from my iPhone

On 3 May 2021, at 18:25, feilong <feilong@catalyst.net.nz> wrote:



Hi Ammad,

What's the error of your kubelet? If the node is in not ready, then you should be able to see some errors from the kubelet log.

On 3/05/21 8:55 pm, Ammad Syed wrote:

Hi,

I have upgraded my magnum environment from victoria to wallaby. The upgrade went successfully. When I am trying to deploy a cluster from template, the status of cluster shows UNHEALTHY but create complete.

I have logged into the master nodes and found no error message in heat logs. The nodes status still sees NotReady.

[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get nodes --all-namespaces NAME STATUS ROLES AGE VERSION k8s-cluster-iomfrpuadezp-master-0 NotReady master 14m v1.18.16 k8s-cluster-iomfrpuadezp-node-0 NotReady <none> 9m51s v1.18.16

Also there is no pods running in kube-system namespace.

[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get pods --all-namespaces No resources found

I have checked the logs, the flannel was deployed.

+ printf 'Starting to run calico-service\n' + set -e + set +x + '[' flannel = calico ']' + printf 'Finished running calico-service\n' + set -e + set +x Finished running calico-service + '[' flannel = flannel ']' + _prefix=quay.io/coreos/ + FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml + '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ']' + echo 'Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml' Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ++ dirname /srv/magnum/kubernetes/manifests/flannel-deploy.yaml + mkdir -p /srv/magnum/kubernetes/manifests + set +x + '[' '' = 0 ']' + /usr/bin/kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created

I tried to deploy the flannel again, but it showing unchanged.

[root@k8s-cluster-iomfrpuadezp-master-0 heat-config-script]# kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged configured clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged serviceaccount/flannel unchanged configmap/kube-flannel-cfg unchanged daemonset.apps/kube-flannel-ds unchanged

The other thing I have noticed that cluster deployment still uses old parameters of victoria like heat_agent_tag and others. Its not using latest default tags of wallaby release.

I am using magnum on ubuntu 20.04. The other components in stack are already upgraded to wallaby release.

-- Regards, Ammad Ali

-- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------

-- Regards, Syed Ammad Ali