Hi Ammad,

What's the error of your kubelet? If the node is in not ready, then you should be able to see some errors from the kubelet log.


On 3/05/21 8:55 pm, Ammad Syed wrote:
Hi,

I have upgraded my magnum environment from victoria to wallaby. The upgrade went successfully. When I am trying to deploy a cluster from template, the status of cluster shows UNHEALTHY but create complete. 

I have logged into the master nodes and found no error message in heat logs. The nodes status still sees NotReady.

[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get nodes --all-namespaces
NAME                                STATUS     ROLES    AGE     VERSION
k8s-cluster-iomfrpuadezp-master-0   NotReady   master   14m     v1.18.16
k8s-cluster-iomfrpuadezp-node-0     NotReady   <none>   9m51s   v1.18.16

Also there is no pods running in kube-system namespace. 

[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get pods --all-namespaces
No resources found

I have checked the logs, the flannel was deployed.

+ printf 'Starting to run calico-service\n'
+ set -e
+ set +x
+ '[' flannel = calico ']'
+ printf 'Finished running calico-service\n'
+ set -e
+ set +x
Finished running calico-service
+ '[' flannel = flannel ']'
+ _prefix=quay.io/coreos/
+ FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml
+ '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ']'
+ echo 'Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml'
Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
++ dirname /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
+ mkdir -p /srv/magnum/kubernetes/manifests
+ set +x
+ '[' '' = 0 ']'
+ /usr/bin/kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

I tried to deploy the flannel again, but it showing unchanged.

[root@k8s-cluster-iomfrpuadezp-master-0 heat-config-script]# kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system
podsecuritypolicy.policy/psp.flannel.unprivileged configured
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.apps/kube-flannel-ds unchanged

The other thing I have noticed that cluster deployment still uses old parameters of victoria like heat_agent_tag and others. Its not using latest default tags of wallaby release. 

I am using magnum on ubuntu 20.04. The other components in stack are already upgraded to wallaby release. 

--
Regards,
Ammad Ali
-- 
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: flwang@catalyst.net.nz
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
------------------------------------------------------