Pawel,

I'd identify the individual resources created by magnum and delete them using their native process (e.g. openstack server delete). Once I was certain all of the resources were actually gone, I'd drop into the magnum and heat databases and remove the references to them (assuming that manually cleaning up the aforementioned provisioned resources doesn't let you delete the cluster/stack in the normal manner).

Chris

Sent from ProtonMail mobile



-------- Original Message --------
On Dec 2, 2019, 10:53, Pawel Konczalski < pawel.konczalski@everyware.ch> wrote:

Hello,

i try to delete a broken magnum (v7.1.1.dev21) cluster but neither the
cluster or the stack entries can be deleted. Any idea how to force / fix
this?

BR

Pawel

openstack coe cluster list | grep DELETE_FAILED

openstack coe cluster delete ae5c347c-3ba8-4d39-9125-0bdadef253b4

openstack coe cluster show ae5c347c-3ba8-4d39-9125-0bdadef253b4
| status               | DELETE_FAILED
... |---
| status_reason        | Resource DELETE failed: JSONDecodeError:
resources.kube_masters.resources[1].resources.etcd_pool_member:
Expecting value: line 1 column 1 (char 0)
...
| faults               | {'1': 'JSONDecodeError:
resources[1].resources.etcd_pool_member: Expecting value: line 1 column
1 (char 0)', 'kube_masters': 'JSONDecodeError:
resources.kube_masters.resources[1].resources.etcd_pool_member:
Expecting value: line 1 column 1 (char 0)', '0': 'JSONDecodeError:
resources[0].resources.api_pool_member: Expecting value: line 1 column 1
(char 0)', 'api_pool_member': 'JSONDecodeError:
resources.api_pool_member: Expecting value: line 1 column 1 (char 0)',
'etcd_pool_member': 'JSONDecodeError: resources.etcd_pool_member:
Expecting value: line 1 column 1 (char 0)'} |

openstack stack list | grep DELETE_FAILED

openstack stack delete 2a907fc4-7b40-43f6-9254-d4084bd056b8

t Payload           DELETE: ResourceGroup "kube_masters"
[85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec] Stack
"slu-k8s-cluster2-6xiqrpfe7jd6" [2a907fc4-7b40-43f6-9254-d4084bd056b8]
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource Traceback (most
recent call last):
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource   File
"/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py",
line 924, in _action_recorder
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource     yield
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource   File
"/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py",
line 2034, in delete
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource *action_args)
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource   File
"/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/scheduler.py",
line 346, in wrapper
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource     step =
next(subtask)
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource   File
"/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py",
line 986, in action_handler_task
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource     done =
check(handler_data)
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource   File
"/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py",
line 596, in check_delete_complete
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource     return
self._check_status_complete(self.DELETE)
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource   File
"/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py",
line 463, in _check_status_complete
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource action=action)
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource ResourceFailure:
JSONDecodeError:
resources.kube_masters.resources[1].resources.etcd_pool_member:
Expecting value: line 1 column 1 (char 0)
2019-12-02 12:43:08.887 22 ERROR heat.engine.resource

# openstack stack list --nested  | grep 7390a9b1d4be4d75b4bd08ab8107e4ff
| 77848f9e-ff0c-4f7a-9fd7-8ed9e979b998 |
test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-1-tpqatosey33u
| 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED   |
2019-05-13T15:28:04Z | 2019-12-02T15:30:53Z |
85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec |
| 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec |
test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd |
7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED   |
2019-05-13T15:28:03Z | 2019-12-02T15:30:53Z |
2a907fc4-7b40-43f6-9254-d4084bd056b8 |
| 4c91c53b-42f4-41ca-a97c-49d9261069d6 |
test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-0-hm2ucynq6hkc
| 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED   |
2019-05-13T15:28:03Z | 2019-12-02T15:30:54Z |
85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec |
| 2a907fc4-7b40-43f6-9254-d4084bd056b8 | test-k8s-cluster2-6xiqrpfe7jd6
| 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED   |
2019-05-13T15:24:45Z | 2019-12-02T15:30:53Z |
None                                 |