Open Stack

Hello folks,

As of last week (14 Nov), our Magnum (Rocky) environment has stopped
spinning up working Kubernetes clusters. To be precise, Magnum does report
the cluster status as CREATE_COMPLETE, but once it is up all its Kubernetes
pods are stuck in the Pending state.

We use the following commands to create the Kubernetes clusters:
http://paste.openstack.org/show/786348/.

All pods show pending status which shows cluster could not select a minion
node for them.

The deployment fails with the following output " 0/2 nodes are available: 2
node(s) had taints that the pod didn't tolerate"
For reference:  http://paste.openstack.org/show/786717/.
For reference output of  kubectl get all -A
http://paste.openstack.org/show/786729/

If we manually remove  NoSchedule taint from the minion nodes we get all
the pods running
For reference: http://paste.openstack.org/show/786718/

But after the manual fix too openstack-cloud-controller-manager pods are
missing so any interaction from the Kubernetes control plane to OpenStack
services is non-functional.

We are assuming that the missing openstack cloud controller manager pod is
also the reason for the taint issue which we are encountering.

For the node taint issue,
https://ask.openstack.org/en/question/120442/magnum-kubernetes-noschedule-taint/
suggests to add
[trust]
cluster_user_trust = true

to magnum.conf. But there is OSA variable named magnum_cluster_user_trust
that can be set to true for this purpose. However, the default for this
variable has been True and we have confirmed we have the parameter in our
environment cluster_user_trust=True as well.


Note: The kubernetes cluster deployed here uses kube_tag 1.14.8 but we are
getting same result with other kube_tag versions v1.13.12, v1.14.8 also.

Ideally, we should not remove taints manually so can you please confirm our
findings and help us find a way forward
We can provide more logs if needed

Thanks
Namrata Sitlani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20191126/14bf3cba/attachment.html>