Hi Tony, we don't run victoria release but maybe I can give you some pointers where to look: as far as I understand Magnum and the kubernetes autoscaler, magnum uses heat to create stacks for the initial kubernetes deployment. the problem is, that the kubernetes autoscaler directly talks to the openstack api, e.g. nova for creating and destroying instances. This can result in some weird situations, e.g. the autoscaler deletes volumes but heat never is involved, so heat still thinks a volume is there which isn't. so you might want to check all resources in your magnum heat stacks, if they are really there, or if autoscaler did things to them. if you e.g. find resources deleted by autoscaler, mark the appropriate heat stack as unhealthy and trigger a stack update, so heat can do it's thing. the stack should then return in a healthy status. if someone has a solution to this general problem, I would be very interested (beside the obvious "solution" to just disable the autoscaler)! HTH Sven On Mi, 2021-08-11 at 15:04 +0800, Tony Pearce wrote:
I sent this mail last week looking for some insight with regards to a magnum issue we had. I hadnt seen any reply and searched for my sent mail - I found I did not complete the subject line. Sorry about that.
Resending again here with a subject. If anyone has any insight to this I'd be grateful to hear from you.
Kind regards,
Tony Pearce
---------- Forwarded message --------- From: Tony Pearce <tonyppe@gmail.com> Date: Thu, 5 Aug 2021 at 14:22 Subject: [magnum] [kolla-ansible] [kayobe] [Victoria] To: OpenStack Discuss <openstack-discuss@lists.openstack.org>
Testing out Kubernetes with Magnum project, deployed via kayobe on Victoria we have deployed an auto scaling cluster and have run into a problem and I'm not sure how to proceed. I understand that the cluster tried to scale up but the openstack project did not have enough CPU resources to accommodate it (error= Quota exceeded for cores: Requested 4, but already used 20 of 20 cores).
So the situation is that the cluster shows "healthy" and "UPDATE_FAILED" but also kubectl commands are failing [1].
What is required to return the cluster back to a working status at this point? I have tried: - cluster resize to reduce number of workers - cluster resize to increase number of workers after increasing project quota - cluster resize and maintaining the same number of workers
When trying any of the above, horizon shows an immediate error "Unable to resize given cluster" but magnum logs and heat logs do not show any log update at all at that time.
- using "check stack" and resume stack in the stack horizon menu gives this error [2]
Investigating the kubectl issue, it was noted that some services had failed on the master node [3]. Manual start as well as reboot the node did not bring up the services. Unfortunately I dont have ssh access to the master and no further information has been forthcoming with regards to logs for those service failures so I am unable to provide anything around that here.
I found this link [4] so I decided to delete the master node then run "check" cluster again but the check cluster just fails in the same way except this time it fails saying that it cannot find the master [5] while the previous error was that it could not find a node.
Ideally I would prefer to recover the cluster - whether this is still possible I am unsure. I can probably recreate this scenario again. What steps should be performed in this case to restore the cluster?
[1] kubectl get no Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)
[2] Resource CHECK failed: ["['NotFound: resources[4].resources.kube-minion: Instance None could not be found. (HTTP 404) (Request-ID: req-6069ff6a-9eb6-4bce-bb25-4ef001ebc428)']. 'CHECK' not fully supported (see resources)"]
[3]
[systemd] Failed Units: 3 etcd.service heat-container-agent.service logrotate.service
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1459854
[5]
["['NotFound: resources.kube_masters.resources[0].resources.kube-master: Instance c6185e8e-1a98-4925-959b-0a56210b8c9e could not be found. (HTTP 404) (Request-ID: req-bdfcc853-7dbb-4022-9208-68b1ab31008a)']. 'CHECK' not fully supported (see resources)"].
Kind regards,
Tony Pearce
-- Mit freundlichen Grüßen / Regards Sven Kieske Systementwickler Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp Tel.: 05772 / 293-900 Fax: 05772 / 293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer, Florian Jürgens St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.