Testing out Kubernetes with Magnum project, deployed via kayobe on Victoria we have deployed an auto scaling cluster and have run into a problem and I'm not sure how to proceed. I understand that the cluster tried to scale up but the openstack project did not have enough CPU resources to accommodate it (error= Quota exceeded for cores: Requested 4, but already used 20 of 20 cores).

So the situation is that the cluster shows "healthy" and "UPDATE_FAILED" but also kubectl commands are failing [1].

What is required to return the cluster back to a working status at this point? I have tried:
- cluster resize to reduce number of workers
- cluster resize to increase number of workers after increasing project quota
- cluster resize and maintaining the same number of workers

When trying any of the above, horizon shows an immediate error "Unable to resize given cluster" but magnum logs and heat logs do not show any log update at all at that time.

- using "check stack" and resume stack in the stack horizon menu gives this error [2]

Investigating the kubectl issue, it was noted that some services had failed on the master node [3]. Manual start as well as reboot the node did not bring up the services. Unfortunately I dont have ssh access to the master and no further information has been forthcoming with regards to logs for those service failures so I am unable to provide anything around that here.

I found this link [4] so I decided to delete the master node then run "check" cluster again but the check cluster just fails in the same way except this time it fails saying that it cannot find the master [5] while the previous error was that it could not find a node.

Ideally I would prefer to recover the cluster - whether this is still possible I am unsure. I can probably recreate this scenario again. What steps should be performed in this case to restore the cluster?



[1]
kubectl get no
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)

[2]
Resource CHECK failed: ["['NotFound: resources[4].resources.kube-minion: Instance None could not be found. (HTTP 404) (Request-ID: req-6069ff6a-9eb6-4bce-bb25-4ef001ebc428)']. 'CHECK' not fully supported (see resources)"]

[3]

[systemd]
Failed Units: 3
  etcd.service
  heat-container-agent.service
  logrotate.service

[4] https://bugzilla.redhat.com/show_bug.cgi?id=1459854

[5]

["['NotFound: resources.kube_masters.resources[0].resources.kube-master: Instance c6185e8e-1a98-4925-959b-0a56210b8c9e could not be found. (HTTP 404) (Request-ID: req-bdfcc853-7dbb-4022-9208-68b1ab31008a)']. 'CHECK' not fully supported (see resources)"].

Kind regards,

Tony Pearce