<div dir="ltr">Hi Bharat / Feilong,<div><br></div><div>The problem look like in ubuntu wallaby repo's magnum DEB package. The package does not have all the changes that a wallaby release should contain. Like <span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px;font-weight:700">heat_container_agent_tag </span>that should be wallaby-stable-1. In that package it was still pointing to victoria-dev. </div><div><br></div><div>I have cloned stable/wallaby from <a href="https://opendev.org/openstack/magnum.git">https://opendev.org/openstack/magnum.git</a> and replaced magnum directory in /lib/python3/dist-packages with downloaded files in stable/magnum branch. Now everything is working a expected. </div><div><br></div><div>[root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get nodes<br>NAME STATUS ROLES AGE VERSION<br>k8s-cluster-2zcsd5n6qnre-master-0 Ready master 5m43s v1.19.10<br>k8s-cluster-2zcsd5n6qnre-node-0 Ready <none> 39s v1.19.10<br>[root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get pods -n kube-system<br>NAME READY STATUS RESTARTS AGE<br>coredns-57999d5467-2l46z 1/1 Running 0 5m43s<br>coredns-57999d5467-rn8d9 1/1 Running 0 5m43s<br>csi-cinder-controllerplugin-0 5/5 Running 0 5m36s<br>csi-cinder-nodeplugin-nmjnb 2/2 Running 0 30s<br>dashboard-metrics-scraper-7b59f7d4df-mzmb6 1/1 Running 0 5m41s<br>k8s-keystone-auth-knhk7 1/1 Running 0 5m39s<br>kube-dns-autoscaler-f57cd985f-7dqj5 1/1 Running 0 5m43s<br>kube-flannel-ds-8d7pc 1/1 Running 0 5m42s<br>kube-flannel-ds-pcncq 1/1 Running 0 60s<br>kubernetes-dashboard-7fb447bf79-x6kvj 1/1 Running 0 5m41s<br>npd-hcw25 1/1 Running 0 30s<br>openstack-cloud-controller-manager-x759f 1/1 Running 0 5m45s</div><div><br></div><div>- Ammad</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 3, 2021 at 11:13 PM Bharat Kunwar <<a href="mailto:bharat@stackhpc.com">bharat@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Can you try the calico plugin? The flannel plug-in has been unmaintained for a while.<br><br><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On 3 May 2021, at 18:25, feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
<p>Hi Ammad,</p>
<p>What's the error of your kubelet? If the node is in not ready,
then you should be able to see some errors from the kubelet log.<br>
</p>
<p><br>
</p>
<div>On 3/05/21 8:55 pm, Ammad Syed wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I have upgraded my magnum environment from victoria to
wallaby. The upgrade went successfully. When I am trying to
deploy a cluster from template, the status of cluster shows
UNHEALTHY but create complete. </div>
<div><br>
</div>
<div>I have logged into the master nodes and found no error
message in heat logs. The nodes status still sees NotReady.</div>
<div><br>
</div>
<div>[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]#
kubectl get nodes --all-namespaces<br>
NAME STATUS ROLES AGE
VERSION<br>
k8s-cluster-iomfrpuadezp-master-0 NotReady master 14m
v1.18.16<br>
k8s-cluster-iomfrpuadezp-node-0 NotReady <none>
9m51s v1.18.16<br>
</div>
<div><br>
</div>
<div>Also there is no pods running in kube-system namespace. </div>
<div><br>
</div>
<div>[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]#
kubectl get pods --all-namespaces<br>
No resources found<br>
</div>
<div><br>
</div>
<div>I have checked the logs, the flannel was deployed.</div>
<div><br>
</div>
<div>+ printf 'Starting to run calico-service\n'<br>
+ set -e<br>
+ set +x<br>
+ '[' flannel = calico ']'<br>
+ printf 'Finished running calico-service\n'<br>
+ set -e<br>
+ set +x<br>
Finished running calico-service<br>
+ '[' flannel = flannel ']'<br>
+ _prefix=<a href="http://quay.io/coreos/" target="_blank">quay.io/coreos/</a><br>
+
FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml<br>
+ '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
']'<br>
+ echo 'Writing File:
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml'<br>
Writing File:
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml<br>
++ dirname
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml<br>
+ mkdir -p /srv/magnum/kubernetes/manifests<br>
+ set +x<br>
+ '[' '' = 0 ']'<br>
+ /usr/bin/kubectl apply -f
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml
--namespace=kube-system<br>
podsecuritypolicy.policy/psp.flannel.unprivileged created<br>
<a href="http://clusterrole.rbac.authorization.k8s.io/flannel" target="_blank">clusterrole.rbac.authorization.k8s.io/flannel</a>
created<br>
<a href="http://clusterrolebinding.rbac.authorization.k8s.io/flannel" target="_blank">clusterrolebinding.rbac.authorization.k8s.io/flannel</a>
created<br>
serviceaccount/flannel created<br>
configmap/kube-flannel-cfg created<br>
daemonset.apps/kube-flannel-ds created<br>
</div>
<div><br>
</div>
<div>I tried to deploy the flannel again, but it showing
unchanged.</div>
<div><br>
</div>
<div>[root@k8s-cluster-iomfrpuadezp-master-0
heat-config-script]# kubectl apply -f
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml
--namespace=kube-system <br>
podsecuritypolicy.policy/psp.flannel.unprivileged configured<br>
<a href="http://clusterrole.rbac.authorization.k8s.io/flannel" target="_blank">clusterrole.rbac.authorization.k8s.io/flannel</a>
unchanged<br>
<a href="http://clusterrolebinding.rbac.authorization.k8s.io/flannel" target="_blank">clusterrolebinding.rbac.authorization.k8s.io/flannel</a>
unchanged<br>
serviceaccount/flannel unchanged<br>
configmap/kube-flannel-cfg unchanged<br>
daemonset.apps/kube-flannel-ds unchanged<br>
</div>
<div><br>
</div>
<div>The other thing I have noticed that cluster
deployment still uses old parameters of victoria like
heat_agent_tag and others. Its not using latest default tags
of wallaby release. </div>
<div><br>
</div>
<div>I am using magnum on ubuntu 20.04. The other components in
stack are already upgraded to wallaby release. </div>
<div>
<div><br>
</div>
-- <br>
<div dir="ltr">Regards,
<div>Ammad Ali</div>
</div>
</div>
</div>
</blockquote>
<pre cols="72">--
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a href="mailto:flwang@catalyst.net.nz" target="_blank">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
------------------------------------------------------ </pre>
</div></blockquote></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Regards,<div><br></div><div><br></div><div>Syed Ammad Ali</div></div>