[wallaby][magnum] Cluster Deployment Unhealthy
Hi,
I have upgraded my magnum environment from victoria to wallaby. The upgrade went successfully. When I am trying to deploy a cluster from template, the status of cluster shows UNHEALTHY but create complete.
I have logged into the master nodes and found no error message in heat logs. The nodes status still sees NotReady.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get nodes --all-namespaces NAME STATUS ROLES AGE VERSION k8s-cluster-iomfrpuadezp-master-0 NotReady master 14m v1.18.16 k8s-cluster-iomfrpuadezp-node-0 NotReady <none> 9m51s v1.18.16
Also there is no pods running in kube-system namespace.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get pods --all-namespaces No resources found
I have checked the logs, the flannel was deployed.
+ printf 'Starting to run calico-service\n' + set -e + set +x + '[' flannel = calico ']' + printf 'Finished running calico-service\n' + set -e + set +x Finished running calico-service + '[' flannel = flannel ']' + _prefix=quay.io/coreos/ + FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml + '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ']' + echo 'Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml' Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ++ dirname /srv/magnum/kubernetes/manifests/flannel-deploy.yaml + mkdir -p /srv/magnum/kubernetes/manifests + set +x + '[' '' = 0 ']' + /usr/bin/kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created
I tried to deploy the flannel again, but it showing unchanged.
[root@k8s-cluster-iomfrpuadezp-master-0 heat-config-script]# kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged configured clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged serviceaccount/flannel unchanged configmap/kube-flannel-cfg unchanged daemonset.apps/kube-flannel-ds unchanged
The other thing I have noticed that cluster deployment still uses old parameters of victoria like heat_agent_tag and others. Its not using latest default tags of wallaby release.
I am using magnum on ubuntu 20.04. The other components in stack are already upgraded to wallaby release.
Hi Ammad,
What's the error of your kubelet? If the node is in not ready, then you should be able to see some errors from the kubelet log.
On 3/05/21 8:55 pm, Ammad Syed wrote:
Hi,
I have upgraded my magnum environment from victoria to wallaby. The upgrade went successfully. When I am trying to deploy a cluster from template, the status of cluster shows UNHEALTHY but create complete.
I have logged into the master nodes and found no error message in heat logs. The nodes status still sees NotReady.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get nodes --all-namespaces NAME STATUS ROLES AGE VERSION k8s-cluster-iomfrpuadezp-master-0 NotReady master 14m v1.18.16 k8s-cluster-iomfrpuadezp-node-0 NotReady <none> 9m51s v1.18.16
Also there is no pods running in kube-system namespace.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get pods --all-namespaces No resources found
I have checked the logs, the flannel was deployed.
- printf 'Starting to run calico-service\n'
- set -e
- set +x
- '[' flannel = calico ']'
- printf 'Finished running calico-service\n'
- set -e
- set +x
Finished running calico-service
- '[' flannel = flannel ']'
- _prefix=quay.io/coreos/ http://quay.io/coreos/
- FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml
- '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ']'
- echo 'Writing File:
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml' Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ++ dirname /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
- mkdir -p /srv/magnum/kubernetes/manifests
- set +x
- '[' '' = 0 ']'
- /usr/bin/kubectl apply -f
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel http://clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel http://clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created
I tried to deploy the flannel again, but it showing unchanged.
[root@k8s-cluster-iomfrpuadezp-master-0 heat-config-script]# kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged configured clusterrole.rbac.authorization.k8s.io/flannel http://clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/flannel http://clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged serviceaccount/flannel unchanged configmap/kube-flannel-cfg unchanged daemonset.apps/kube-flannel-ds unchanged
The other thing I have noticed that cluster deployment still uses old parameters of victoria like heat_agent_tag and others. Its not using latest default tags of wallaby release.
I am using magnum on ubuntu 20.04. The other components in stack are already upgraded to wallaby release.
-- Regards, Ammad Ali
Can you try the calico plugin? The flannel plug-in has been unmaintained for a while.
Sent from my iPhone
On 3 May 2021, at 18:25, feilong feilong@catalyst.net.nz wrote:
Hi Ammad,
What's the error of your kubelet? If the node is in not ready, then you should be able to see some errors from the kubelet log.
On 3/05/21 8:55 pm, Ammad Syed wrote: Hi,
I have upgraded my magnum environment from victoria to wallaby. The upgrade went successfully. When I am trying to deploy a cluster from template, the status of cluster shows UNHEALTHY but create complete.
I have logged into the master nodes and found no error message in heat logs. The nodes status still sees NotReady.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get nodes --all-namespaces NAME STATUS ROLES AGE VERSION k8s-cluster-iomfrpuadezp-master-0 NotReady master 14m v1.18.16 k8s-cluster-iomfrpuadezp-node-0 NotReady <none> 9m51s v1.18.16
Also there is no pods running in kube-system namespace.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get pods --all-namespaces No resources found
I have checked the logs, the flannel was deployed.
- printf 'Starting to run calico-service\n'
- set -e
- set +x
- '[' flannel = calico ']'
- printf 'Finished running calico-service\n'
- set -e
- set +x
Finished running calico-service
- '[' flannel = flannel ']'
- _prefix=quay.io/coreos/
- FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml
- '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ']'
- echo 'Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml'
Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ++ dirname /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
- mkdir -p /srv/magnum/kubernetes/manifests
- set +x
- '[' '' = 0 ']'
- /usr/bin/kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system
podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created
I tried to deploy the flannel again, but it showing unchanged.
[root@k8s-cluster-iomfrpuadezp-master-0 heat-config-script]# kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged configured clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged serviceaccount/flannel unchanged configmap/kube-flannel-cfg unchanged daemonset.apps/kube-flannel-ds unchanged
The other thing I have noticed that cluster deployment still uses old parameters of victoria like heat_agent_tag and others. Its not using latest default tags of wallaby release.
I am using magnum on ubuntu 20.04. The other components in stack are already upgraded to wallaby release.
-- Regards, Ammad Ali
-- Cheers & Best regards, Feilong Wang (王飞龙)
Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington
Hi Bharat / Feilong,
The problem look like in ubuntu wallaby repo's magnum DEB package. The package does not have all the changes that a wallaby release should contain. Like heat_container_agent_tag that should be wallaby-stable-1. In that package it was still pointing to victoria-dev.
I have cloned stable/wallaby from https://opendev.org/openstack/magnum.git and replaced magnum directory in /lib/python3/dist-packages with downloaded files in stable/magnum branch. Now everything is working a expected.
[root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-cluster-2zcsd5n6qnre-master-0 Ready master 5m43s v1.19.10 k8s-cluster-2zcsd5n6qnre-node-0 Ready <none> 39s v1.19.10 [root@k8s-cluster-2zcsd5n6qnre-master-0 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-57999d5467-2l46z 1/1 Running 0 5m43s coredns-57999d5467-rn8d9 1/1 Running 0 5m43s csi-cinder-controllerplugin-0 5/5 Running 0 5m36s csi-cinder-nodeplugin-nmjnb 2/2 Running 0 30s dashboard-metrics-scraper-7b59f7d4df-mzmb6 1/1 Running 0 5m41s k8s-keystone-auth-knhk7 1/1 Running 0 5m39s kube-dns-autoscaler-f57cd985f-7dqj5 1/1 Running 0 5m43s kube-flannel-ds-8d7pc 1/1 Running 0 5m42s kube-flannel-ds-pcncq 1/1 Running 0 60s kubernetes-dashboard-7fb447bf79-x6kvj 1/1 Running 0 5m41s npd-hcw25 1/1 Running 0 30s openstack-cloud-controller-manager-x759f 1/1 Running 0 5m45s
- Ammad
On Mon, May 3, 2021 at 11:13 PM Bharat Kunwar bharat@stackhpc.com wrote:
Can you try the calico plugin? The flannel plug-in has been unmaintained for a while.
Sent from my iPhone
On 3 May 2021, at 18:25, feilong feilong@catalyst.net.nz wrote:
Hi Ammad,
What's the error of your kubelet? If the node is in not ready, then you should be able to see some errors from the kubelet log.
On 3/05/21 8:55 pm, Ammad Syed wrote:
Hi,
I have upgraded my magnum environment from victoria to wallaby. The upgrade went successfully. When I am trying to deploy a cluster from template, the status of cluster shows UNHEALTHY but create complete.
I have logged into the master nodes and found no error message in heat logs. The nodes status still sees NotReady.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get nodes --all-namespaces NAME STATUS ROLES AGE VERSION k8s-cluster-iomfrpuadezp-master-0 NotReady master 14m v1.18.16 k8s-cluster-iomfrpuadezp-node-0 NotReady <none> 9m51s v1.18.16
Also there is no pods running in kube-system namespace.
[root@k8s-cluster-iomfrpuadezp-master-0 kubernetes]# kubectl get pods --all-namespaces No resources found
I have checked the logs, the flannel was deployed.
- printf 'Starting to run calico-service\n'
- set -e
- set +x
- '[' flannel = calico ']'
- printf 'Finished running calico-service\n'
- set -e
- set +x
Finished running calico-service
- '[' flannel = flannel ']'
- _prefix=quay.io/coreos/
- FLANNEL_DEPLOY=/srv/magnum/kubernetes/manifests/flannel-deploy.yaml
- '[' -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ']'
- echo 'Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml'
Writing File: /srv/magnum/kubernetes/manifests/flannel-deploy.yaml ++ dirname /srv/magnum/kubernetes/manifests/flannel-deploy.yaml
- mkdir -p /srv/magnum/kubernetes/manifests
- set +x
- '[' '' = 0 ']'
- /usr/bin/kubectl apply -f
/srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created
I tried to deploy the flannel again, but it showing unchanged.
[root@k8s-cluster-iomfrpuadezp-master-0 heat-config-script]# kubectl apply -f /srv/magnum/kubernetes/manifests/flannel-deploy.yaml --namespace=kube-system podsecuritypolicy.policy/psp.flannel.unprivileged configured clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged serviceaccount/flannel unchanged configmap/kube-flannel-cfg unchanged daemonset.apps/kube-flannel-ds unchanged
The other thing I have noticed that cluster deployment still uses old parameters of victoria like heat_agent_tag and others. Its not using latest default tags of wallaby release.
I am using magnum on ubuntu 20.04. The other components in stack are already upgraded to wallaby release.
-- Regards, Ammad Ali
-- Cheers & Best regards, Feilong Wang (王飞龙)
Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington
participants (3)
-
Ammad Syed
-
Bharat Kunwar
-
feilong