Pawel,
I'd identify the individual resources created by magnum and delete them using their native process (e.g. openstack server delete). Once I was certain all of the resources were actually gone, I'd drop into the magnum and heat databases and remove the references to them (assuming that manually cleaning up the aforementioned provisioned resources doesn't let you delete the cluster/stack in the normal manner).
Chris
Sent from ProtonMail mobile
-------- Original Message -------- On Dec 2, 2019, 10:53, Pawel Konczalski wrote:
Hello,
i try to delete a broken magnum (v7.1.1.dev21) cluster but neither the cluster or the stack entries can be deleted. Any idea how to force / fix this?
BR
Pawel
openstack coe cluster list | grep DELETE_FAILED
openstack coe cluster delete ae5c347c-3ba8-4d[39-9125-0](tel:3991250)bdadef253b4
openstack coe cluster show ae5c347c-3ba8-4d[39-9125-0](tel:3991250)bdadef253b4 | status | DELETE_FAILED ... |--- | status_reason | Resource DELETE failed: JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0) ... | faults | {'1': 'JSONDecodeError: resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)', 'kube_masters': 'JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)', '0': 'JSONDecodeError: resources[0].resources.api_pool_member: Expecting value: line 1 column 1 (char 0)', 'api_pool_member': 'JSONDecodeError: resources.api_pool_member: Expecting value: line 1 column 1 (char 0)', 'etcd_pool_member': 'JSONDecodeError: resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)'} |
openstack stack list | grep DELETE_FAILED
openstack stack delete 2a907fc4-7b40-43f6-9254-d4084bd056b8
t Payload DELETE: ResourceGroup "kube_masters" [85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec] Stack "slu-k8s-cluster2-6xiqrpfe7jd6" [2a907fc4-7b40-43f6-9254-d4084bd056b8] 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource Traceback (most recent call last): 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 924, in _action_recorder 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource yield 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 2034, in delete 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource *action_args) 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource step = next(subtask) 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 986, in action_handler_task 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource done = check(handler_data) 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 596, in check_delete_complete 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource return self._check_status_complete(self.DELETE) 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 463, in _check_status_complete 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource action=action) 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource ResourceFailure: JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0) 2019-12-02 12:43:[08.887 22](tel:0888722) ERROR heat.engine.resource
# openstack stack list --nested | grep 7390a9b1d4be4d75b4bd08ab8107e4ff | 77848f9e-ff0c-4f7a-9fd7-8ed9e979b998 | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-1-tpqatosey33u | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:04Z | 2019-12-02T15:30:53Z | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:03Z | 2019-12-02T15:30:53Z | 2a907fc4-7b40-43f6-9254-d4084bd056b8 | | 4c91c53b-42f4-41ca-a97c-49d[9261069](tel:9261069)d6 | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-0-hm2ucynq6hkc | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:03Z | 2019-12-02T15:30:54Z | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | | 2a907fc4-7b40-43f6-9254-d4084bd056b8 | test-k8s-cluster2-6xiqrpfe7jd6 | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:24:45Z | 2019-12-02T15:30:53Z | None |