[magnum] [kolla-ansible] [kayobe] [Victoria] Magnum Kubernetes cluster failure recovery

feilong feilong at catalystcloud.nz
Wed Aug 11 18:41:24 UTC 2021


Let me try to explain it from a design perspective:

1. Auto scaler: Now cluster auto scaler talks to Magnum resize API
directly to scale, see
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/magnum/magnum_manager_impl.go#L399

2. Auto healer: As you know auto scaler only cares about the worker
node, it won't scale the master nodes. However, auto healer can repair
both master nodes and worker nodes. With worker nodes repairing, Magnum
auto healer uses magnum resize API. But because the magnum resize api
doesn't support master nodes resizing, so the master nodes repairing is
done by Heat stack update. magnum auto healer will mark some resources
of the master node as unhealthy, then call Heat stack update to rebuild
those resources.


On 11/08/21 10:25 pm, Sven Kieske wrote:
> On Mi, 2021-08-11 at 10:16 +0000, Sven Kieske wrote:
>> the problem is, that the kubernetes autoscaler directly talks to the openstack api, e.g.
>> nova for creating and destroying instances.
> Nevermind I got that wrong.
>
> The autoscaler talks to heat, so there should no problem (but heat trips itself up on some error conditions).
> I was in fact talking about the magnum auto healer (https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/magnum-auto-healer/using-magnum-auto-healer.md )
> which seems to circumvent heat and talks directly with nova.
>
> Are you using the magnum auto healing feature by chance?
>
> HTH
>
-- 
Cheers & Best regards,
------------------------------------------------------------------------------
Feilong Wang (王飞龙) (he/him)
Head of Research & Development

Catalyst Cloud
Aotearoa's own

Mob: +64 21 0832 6348 | www.catalystcloud.nz
Level 6, 150 Willis Street, Wellington 6011, New Zealand

CONFIDENTIALITY NOTICE: This email is intended for the named recipients only.
It may contain privileged, confidential or copyright information. If you are 
not the named recipient, any use, reliance upon, disclosure or copying of this 
email or its attachments is unauthorised. If you have received this email in 
error, please reply via email or call +64 21 0832 6348.
------------------------------------------------------------------------------





More information about the openstack-discuss mailing list