On Thu, 2023-07-13 at 10:16 +0000, Ayumu Ueha (Fujitsu) wrote:

Hi Michał,

Thanks for your reply,

I trying it in my local machine and check logs, but I am not familiar with troubleshooting with kubelet/kubeadm, so I do not know why it errored as for now. We will continue to investigate, but it would be helpful if you could share any small findings.

For your reference, I will attach a log of the kubelet.service when it failed in my local environment.

Looks like kube-apiserver and kube-scheduler are failing to start. Checking logs of them would be my next idea. You might want to use `crictl` to find them.

Thank you.

Best Regards, Ueha

-----Original Message----- From: Michał Dulko <mdulko@redhat.com> Sent: Thursday, July 13, 2023 3:48 PM To: openstack-discuss@lists.openstack.org Subject: Re: [kuryr][tacker] fails to bump the k8s version to v1.26

On Tue, 2023-07-04 at 07:53 +0000, Ayumu Ueha (Fujitsu) wrote:

...

Hi kuryr-kubetenetes team,   Tacker uses the kuryr-kubernetes as the setup for the kubernetes Environment. In the Bobcat version, we will bump the supported version of kubenetes to 1.26.6, and when I tried to bump the version in devstack's local.conf by the patch [1], the following error occurred. (please refer the details to Zuul log [2]) ============================== [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed.   Unfortunately, an error has occurred:               timed out waiting for the condition   This error is likely caused by:               - The kubelet is not running               - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)   If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:               - 'systemctl status kubelet'               - 'journalctl -xeu kubelet'   Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all running Kubernetes containers by using crictl:               - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause'               Once you have found the failing container, you can inspect its logs with:               - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster ==============================   I know it is not yet supporting version 1.26 of the K8S at the kuryr- kubernetes as for now, but do you know any way to avoid the above error? I suspect this is due to a change from version 1.25, but I'm not sure which one is affecting...   Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? Will the Bobcat release support it? Please kindly let me know if you know anything. Thank you.

Kuryr-kubernetes itself supports K8s 1.26 and 1.27 as we test it with these versions as part of OpenShift. What doesn't seem to work is the DevStack plugin, I guess due to some breaking changes in kubelet or kubeadm.

I can advise to try to deploy DevStack with these settings on your local machine and then you should have access to all the logs required to figure out what is causing the problem with the version bump.

...

[1] https://review.opendev.org/c/openstack/tacker/+/886935   Change the parameters:      KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6      CRIO_VERSION: 1.25 to 1.26 [2] https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219 b54d9/log/controller-k8s/logs/devstacklog.txt#9761-9816   Best Regards, Ueha