Hi Bharat / Feilong,

The problem look like in ubuntu wallaby repo's magnum DEB package. The package does not have all the changes that a wallaby release should contain. Like heat_container_agent_tag that should be wallaby-stable-1. In that package it was still pointing to victoria-dev. 

I have cloned stable/wallaby from https://opendev.org/openstack/magnum.git and replaced magnum directory in /lib/python3/dist-packages with downloaded files in stable/magnum branch. Now everything is working a expected.  

[root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get nodes
NAME                                STATUS   ROLES    AGE     VERSION
k8s-cluster-2zcsd5n6qnre-master-0   Ready    master   5m43s   v1.19.10
k8s-cluster-2zcsd5n6qnre-node-0     Ready    <none>   39s     v1.19.10
[root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get pods -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-57999d5467-2l46z                     1/1     Running   0          5m43s
coredns-57999d5467-rn8d9                     1/1     Running   0          5m43s
csi-cinder-controllerplugin-0                5/5     Running   0          5m36s
csi-cinder-nodeplugin-nmjnb                  2/2     Running   0          30s
dashboard-metrics-scraper-7b59f7d4df-mzmb6   1/1     Running   0          5m41s
k8s-keystone-auth-knhk7                      1/1     Running   0          5m39s
kube-dns-autoscaler-f57cd985f-7dqj5          1/1     Running   0          5m43s
kube-flannel-ds-8d7pc                        1/1     Running   0          5m42s
kube-flannel-ds-pcncq                        1/1     Running   0          60s
kubernetes-dashboard-7fb447bf79-x6kvj        1/1     Running   0          5m41s
npd-hcw25                                    1/1     Running   0          30s
openstack-cloud-controller-manager-x759f     1/1     Running   0          5m45s

- Ammad

On Mon, May 3, 2021 at 11:13 PM Bharat Kunwar <bharat@stackhpc.com> wrote:
Can you try the calico plugin? The flannel plug-in has been unmaintained for a while.

Sent from my iPhone

On 3 May 2021, at 18:25, feilong <feilong@catalyst.net.nz> wrote:



Hi Ammad,

What's the error of your kubelet? If the node is in not ready, then you should be able to see some errors from the kubelet log.


On 3/05/21 8:55 pm, Ammad Syed wrote:
Hi,

I have upgraded my magnum environment from victoria to wallaby. The upgrade went successfully. When I am trying to deploy a cluster from template, the status of cluster shows UNHEALTHY but create complete. 

I have logged into the master nodes and found no error message in heat logs. The nodes status still sees NotReady.

[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get nodes --all-namespaces
NAME                                STATUS     ROLES    AGE     VERSION
k8s-cluster-iomfrpuadezp-master-0   NotReady   master   14m     v1.18.16
k8s-cluster-iomfrpuadezp-node-0     NotReady   <none>   9m51s   v1.18.16

Also there is no pods running in kube-system namespace. 

[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get pods --all-namespaces
No resources found

I have checked the logs, the flannel was deployed.

+ printf 'Starting to run calico-service\n'
+ set -e
+ set +x
+ '[' flannel = calico ']'
+ printf 'Finished running calico-service\n'
+ set -e
+ set +x
Finished running calico-service
+ '[' flannel = flannel ']'
+ _prefix=quay.io/coreos/
+ FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml
+ '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ']'
+ echo 'Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml'
Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
++ dirname /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
+ mkdir -p /srv/magnum/kubernetes/manifests
+ set +x
+ '[' '' = 0 ']'
+ /usr/bin/kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

I tried to deploy the flannel again, but it showing unchanged.

[root@k8s-cluster-iomfrpuadezp-master-0 heat-config-script]# kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system
podsecuritypolicy.policy/psp.flannel.unprivileged configured
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.apps/kube-flannel-ds unchanged

The other thing I have noticed that cluster deployment still uses old parameters of victoria like heat_agent_tag and others. Its not using latest default tags of wallaby release. 

I am using magnum on ubuntu 20.04. The other components in stack are already upgraded to wallaby release. 

--
Regards,
Ammad Ali
-- 
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: flwang@catalyst.net.nz
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
------------------------------------------------------


--
Regards,


Syed Ammad Ali