Open Stack

Hi Krzysztof,

Thanks for raising this topic because I'm planning to do improvements
for this area. I would like to help as the original author of this
feature. Now let me explain the current situation:

1. The first method is designed to work for both Fedora Atomic and
Fedora CoreOS. Though I agree after upgrade, the node image will be
remain the old name and ID which will bring troubles for auto healing
later. That's the problem I'm trying to fix but it's not easy. As for
the new node, I think it's a bug and I think I know how to fix it. Your
concern about upgrade from a very old node's OS to a quite new OS
version is valid :(

2. It works under conditions. The node should be image based instead of
volume based, because AFAIK, Nova still doesn't support volume based
instance rebuild. Did you try this with image based nodes? As for the
drain part, it's because we would like to achieve a zero-downtime
upgrade (at least it's my goal for this), so each node will be drained
before upgrading. However, I didn't see a way to manage the
orchestration to call a k8s drain before doing the rebuild of the node,
because it's out of the control of Magnum. Heat is a like a black box at
this stage. Also, even if we can have a chance to call k8s drain to
drain the node, it's impossible to do that if the cluster is a private
cluster. Private cluster means Magnum control plane cannot reach the k8s
API.

Again, thank you raising this and I'm happy to help to address it.


On 22/01/21 10:03 pm, Krzysztof Klimonda wrote:
> Hi,
>
> While testing magnum, a problem of upgrades came up - while work has been done to make kubernetes upgrades without interruption, operating system upgrades seem to be handled only partially.
>
> According to the documentation, two ways of upgrading system are available:
> - via specifying ostree_commit or ostree_remote labels in the cluster template used for upgrade
> - via specifying a new image in the cluster template used for upgrade
>
> The first one is specific to Fedora Atomic (and, while probably untested, seems to be mostly working with Fedora CoreOS) but it has some drawbacks. Firstly, due to base image staying the same we require this image for the life of the cluster, even if OS has already been upgraded. Secondly, using this method only upgrades existing instances and new instances (spawned via scaling cluster up) will not be upgraded. Thirdly, even if that is fixed I'm worried that at some point upgrading from old base image to some future ostree snapshot will fail (there is also cost associated with diff growing with each release).
>
> The second method, of specifying a new image in the cluster template used for upgrade, comes with an ugly warning about nodes not being drained properly before server rebuild (and it actually doesn't seem to be working anyway as the new image parameter is not being passed to the heat template on upgrade). This does however seem like a more valid approach in general.
>
> I'm not that familar with Heat, and the documentation of various OS::Heat::Software* resources seems inconclusive, but is there no way of executing some code before instance is rebuilt? If not, how are other projects and users handling this in general?
>
-- 
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: flwang at catalyst.net.nz
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
------------------------------------------------------