<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"><meta http-equiv="content-type" content="text/html; charset=utf-8"><div dir="ltr"><meta http-equiv="content-type" content="text/html; charset=utf-8"><div dir="ltr"><meta http-equiv="content-type" content="text/html; charset=utf-8">Hi,<div><br></div><div>I recently deployed Openstack wallaby using kolla-ansible and also deployed magnum. I know it was working fine a while ago and I was able to spin up K8s clusters without a problem. But I think This was on Ussuri back then. I went through the magnum troubleshooting guide but couldn’t solve my problem. Magnum spins up the master node and I can log in via SSH using its floating IP. I checked the logs and saw this after waiting for a few minutes:</div><div><br></div><div><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span dir="ltr">role.kubernetes.io/master=</span>""</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ echo 'Trying to label master node with <span dir="ltr">node-role.kubernetes.io/master=</span>""'</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ sleep 5s</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">++ kubectl get --raw=/healthz</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Error from server (InternalError): an error on the server ("[+]ping ok\n[+]log ok\n[+]etcd ok\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[-]poststarthook/crd-informer-synced failed: reason withheld\n[-]poststarthook/bootstrap-controller failed: reason withheld\n[-]poststarthook/rbac/bootstrap-roles failed: reason withheld\n[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld\n[+]poststarthook/apiserver/bootstrap-system-flowcontrol-configuration ok\n[+]poststarthook/start-cluster-authentication-info-controller ok\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[-]autoregister-completion failed: reason withheld\n[+]poststarthook/apiservice-openapi-controller ok\nhealthz check failed") has prevented the request from succeeding</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ '[' ok = '' ']'</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ echo 'Trying to label master node with <span dir="ltr">node-role.kubernetes.io/master=</span>""'</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ sleep 5s</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Trying to label master node with <span dir="ltr">node-role.kubernetes.io/master=</span>""</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">++ kubectl get --raw=/healthz</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ '[' ok = ok ']'</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ kubectl patch node k8s-test-small-cal-zwe5xmigugwj-master-0 --patch '{"metadata": {"labels": {"<span dir="ltr">node-role.kubernetes.io/master</span>": ""}}}'</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Error from server (NotFound): nodes "k8s-test-small-cal-zwe5xmigugwj-master-0" not found</span><br style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">+ echo 'Trying to label master node with <span dir="ltr">node-role.kubernetes.io/master=</span>""'</span><br><br>Running kubectl get nodes is just empty even when appending all-namespaces. I pretty much used the documentation that I created when I was using Ussuri. I wonder what has changed since then that would make this fail.</div><div><br></div><div>I googled for hours but was not able to find similar issues and if then it was about having different version of k8s server and client. Which is definitely not the case. I also tried this on Xena but it also fails.</div><div><span style="-webkit-text-size-adjust: auto; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span dir="ltr"><br></span></span></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;">I do have the feeling that the issue is network related but I do not see any issues at all spinning up instances and also the communication between instances works fine.</span></font></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;"><br></span></font></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;">Here are my current configs:</span></font></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;"><br></span></font></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;">Globals.yml</span></font></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;"><br></span></font></div><div><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">[</span><span class="hljs-symbol" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">vagrant@</span><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">seed ~]$ grep ^[^#] /etc/kolla/globals.yml</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">---</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">kolla_base_distro: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"centos"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">kolla_install_type: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"source"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">openstack_release: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"wallaby"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">kolla_internal_vip_address: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"192.168.45.222"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">kolla_external_vip_address: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"192.168.2.222"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">network_interface: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"eth2"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">neutron_external_interface: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"eth1"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">keepalived_virtual_router_id: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"222"</span><br style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">enable_haproxy: </span><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">"yes"</span></div><div><font color="#e9e17d" face="monospace, monospace"><span style="caret-color: rgb(233, 225, 125); font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">enable_magnum: “yes”</span></font></div><div><font color="#e9e17d" face="monospace, monospace"><span style="caret-color: rgb(233, 225, 125); font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><br></span></font></div><div><font color="#e9e17d" face="monospace, monospace"><span style="caret-color: rgb(233, 225, 125); font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><br></span></font></div><div><span class="hljs-string" style="color: rgb(233, 225, 125); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><br></span></div><div><font color="#e9e17d" face="monospace, monospace"><span style="caret-color: rgb(233, 225, 125); font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">multinode hosts file</span></font></div><div><font color="#e9e17d" face="monospace, monospace"><span style="caret-color: rgb(233, 225, 125); font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);"><br></span></font></div><div><span style="caret-color: rgb(127, 107, 127); color: rgb(127, 107, 127); font-family: monospace, monospace; font-size: 14px; white-space: pre; -webkit-text-size-adjust: 100%; background-color: rgb(255, 255, 255);">control[01:03] ansible_user=vagrant ansible_password=vagrant ansible_become=true api_interface=eth3
compute[01:02] ansible_user=vagrant ansible_password=vagrant ansible_become=true api_interface=eth3
[control]
# These hostname must be resolvable from your deployment host
control[01:03]
# The above can also be specified as follows:
#control[01:03] ansible_user=vagrant ansible_password=vagrant ansible_become=true
#compute[01:02] ansible_user=vagrant ansible_password=vagrant ansible_become=true
# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
control[01:03]
#network01
#network02
[compute]
compute[01:02]
[monitoring]
control[01:03]
#monitoring01
# When compute nodes and control nodes use different interfaces,
# you need to comment out "api_interface" and other interfaces from the globals.yml
# and specify like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1
[storage]
control[01:03]
#storage01
[deployment]
localhost ansible_connection=local</span></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;"><br></span></font></div><div><pre class="bash geshifilter-bash" style="font-size: 1em; margin-top: 0px; margin-bottom: 0px; padding: 0px; line-height: 1.3; caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34); -webkit-text-size-adjust: 100%; background-color: rgb(248, 248, 248);"><span style="color: rgb(102, 102, 102); font-style: italic;">cat /etc/kolla/config/magnum.conf</span></pre><pre class="bash geshifilter-bash" style="font-size: 1em; margin-top: 0px; margin-bottom: 0px; padding: 0px; line-height: 1.3; caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34); -webkit-text-size-adjust: 100%; background-color: rgb(248, 248, 248);"><pre class="bash geshifilter-bash" style="font-size: 1em; margin-top: 0px; margin-bottom: 0px; padding: 0px; line-height: 1.3;"><span style="color: rgb(122, 8, 116); font-weight: bold;"><br></span></pre><pre class="bash geshifilter-bash" style="font-size: 1em; margin-top: 0px; margin-bottom: 0px; padding: 0px; line-height: 1.3;"><span style="color: rgb(122, 8, 116); font-weight: bold;">[</span>trust<span style="color: rgb(122, 8, 116); font-weight: bold;">]</span>
cluster_user_trust = True</pre><pre class="bash geshifilter-bash" style="font-size: 1em; margin-top: 0px; margin-bottom: 0px; padding: 0px; line-height: 1.3;"><br></pre><pre class="bash geshifilter-bash" style="font-size: 1em; margin-top: 0px; margin-bottom: 0px; padding: 0px; line-height: 1.3;">Sorry for the formatting. Sending this on a smartphone with plenty of copy and paste.</pre></pre></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;"><br></span></font></div><div><div dir="ltr">Best Regards,</div></div><div id="AppleMailSignature" dir="ltr">Oliver</div></div></div></div></body></html>