<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Tony<div class=""><br class=""></div><div class="">I have not used designate myself so not sure about the exact details but if you are using Kayobe/Kolla-Ansible, we recently proposed these backports to train, <a href="https://review.opendev.org/#/c/738882/1/ansible/roles/magnum/templates/magnum.conf.j2" class="">https://review.opendev.org/#/c/738882/1/ansible/roles/magnum/templates/magnum.conf.j2</a>. Magnum queries Keystone catalog for the url instances can use to talk back with Keystone and Magnum itself. Usually this is the public URL but essentially you need to specify an endpoint name which fits the bill. Please check /etc/kolla/magnum-conductor/magnum.conf in your control plane where Magnum is deployed and ensure it it configured to the correct interface.<br class=""><div><br class=""></div><div><br class=""></div><div>Cheers</div><div><br class=""></div><div>Bharat</div><div><br class=""><blockquote type="cite" class=""><div class="">On 13 Jul 2020, at 08:43, Tony Pearce <<a href="mailto:tonyppe@gmail.com" class="">tonyppe@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Hi Bharat, many thanks for your super quick response to me last week. I really appreciate that, especially since I had been trying for so long on this issue here. I wanted to try out your suggestion before coming back and creating a reply. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I tried your suggestion and at first, I got the same experience (failure) when creating a cluster. It appeared to stop in the same place as I described in the mail previous. I noticed some weird things with DNS integration (Designate) during the investigation [1] and [2]. I decided to remove Designate from Openstack and retest and now I am successfully able to deploy a kubernetes cluster! :) </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Regarding those 2 points: </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">[1] - the configured designate zone was <a href="http://project.cloud.company.com/" class="">project.cloud.company.com</a> and instance1 would be <a href="http://instance1.project.cloud.company.com/" class="">instance1.project.cloud.company.com</a> however, the kube master instance hostname was getting <a href="http://master.cloud.company.com/" class="">master.cloud.company.com</a></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">[2] - when doing a dns lookup on <a href="http://master.project.cloud.company.com/" class="">master.project.cloud.company.com</a> the private IP was being returned instead of the floating IP. This meant that from outside the project, the instance couldnt be pinged by hostname. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I've removed both magnum and Designate and then redeployed both by first deploying Magnum and testing successful kubernetes cluster deployment using your fix Bharat. Then I deployed Designate again. Issue [1] is still present while issue [2] is resolved and no longer present. Kubernetes cluster deployment is still successful :) </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Thank you once again and have a great week ahead! <br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Kind regards,<br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class=""><div dir="ltr" class=""><div dir="ltr" class="">Tony Pearce<br class=""><br class=""></div></div></div><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 10 Jul 2020 at 16:24, Bharat Kunwar <<a href="mailto:bharat@stackhpc.com" target="_blank" class="">bharat@stackhpc.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Tony<div class=""><br class=""></div><div class="">That is a known issue and is due to the default version of heat container agent baked into Train release. Please use label heat_container_agent_tag=train-stable-3 and you should be good to go.</div><div class=""><br class=""></div><div class="">Cheers</div><div class=""><br class=""></div><div class="">Bharat</div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 10 Jul 2020, at 09:18, Tony Pearce <<a href="mailto:tonyppe@gmail.com" target="_blank" class="">tonyppe@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Hi team, I hope you are all keeping safe and well at the moment. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I am trying to use magnum to launch a kubernetes cluster. I have tried different images but currently using Fedora-Atomic 27. The cluster deployment from the cluster template is failing and I am here to ask if you could please point me in the right direction? I have become stuck and I am uncertain how to further troubleshoot this. The cluster seems to fail a few minutes after booting up the master node because after I see the logs ([1],[2]), I do not see any progress in terms of new (different) logs or load on the master. Then the 60-minute timeout is reached and fails the cluster. <br class=""><br class="">I deployed this openstack stack using kayobe (kolla-ansible) and this is version Train. This is deployed on CentOS 7 within docker containers. Kayobe manages this deployment through the ansible playbooks.</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">This was previously working some months back although I think I may have used coreos image at that time, and that is also not working today. The deployment would have been back around February 2020. I then deleted that deployment and re-deployed. The only change being the hostname for controller node as updated in the inventory file for the kayobe.</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Since then which was a month or so back I've been unable to successfully deploy a kubernetes cluster. I've tried other fedora-atomic images as well as coreos without success. When using the coreos image and when tagging the image with the coreos tag as per the magnum docs, the instance fails to boot and goes to the rescue shell. However if I manually launch the coreos image then it does successfully boot and get configured via cloud-init. All of the deployment attempts stop at the same place when using fedora image and I have a different experience if I disable TLS: </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">TLS enabled: master launched, no nodes. Fails when running /usr/lib/python2.7/site-packages/magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">TLS disabled: master and nodes launched but later fails. I didnt investigate this very much. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">When looking for help around the web, I found this which looks to be the same issue that I have at the moment (although he's deployed slightly differently, using centos8 and mentions magnum 10): </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><a href="https://ask.openstack.org/en/question/128391/magnum-ussuri-container-not-booting-up/" target="_blank" class="">https://ask.openstack.org/en/question/128391/magnum-ussuri-container-not-booting-up/</a> <br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I have the same log messages on the master node within heat. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">When going through the troubleshooting guide I see that etcd is running and no errors however I dont see any flannel service at all. But I also don't know if this has simply failed before getting to deploy flannel or whether flannel is the reason. I did try to deploy using a cluster template that is using calico as a test but the same result from the logs.</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">When looking at the stack via cli to see the failed stacks this is what I see there: <a href="http://paste.openstack.org/show/795736/" style="font-family:Arial,Helvetica,sans-serif" target="_blank" class="">http://paste.openstack.org/show/795736/</a></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I'm using master node flavour with 4cpu and 4GB memory. Node with 2cpu and 2GB memory. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Storage is only via cinder as I am using iscsi storage with a cinder driver. I dont have any other storage. </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">On the master, after the failure the heat log repeats these logs: <br class=""><br class=""><div class="gmail_default">++ curl --silent <a href="http://127.0.0.1:8080/healthz" target="_blank" class="">http://127.0.0.1:8080/healthz</a></div><div class="gmail_default">+ '[' ok = ok ']'</div><div class="gmail_default">+ kubectl patch node k8s-cluster-onvaoh2zxotf-master-0 --patch '{"metadata": {"labels": {"<a href="http://node-role.kubernetes.io/master" target="_blank" class="">node-role.kubernetes.io/master</a>": ""}}}'</div><div class="gmail_default">error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable</div><div class="gmail_default">Trying to label master node with <a href="http://node-role.kubernetes.io/master=" target="_blank" class="">node-role.kubernetes.io/master=</a>""</div><div class="gmail_default">+ echo 'Trying to label master node with <a href="http://node-role.kubernetes.io/master=" target="_blank" class="">node-role.kubernetes.io/master=</a>""'</div><div class="gmail_default">+ sleep 5s</div><div class="gmail_default"><br class=""></div><div class="gmail_default">[1]Here's the cloud-init.log: <a href="http://paste.openstack.org/show/795737/" target="_blank" class="">http://paste.openstack.org/show/795737/</a> </div><div class="gmail_default">[2]and cloud-init-output.log: <a href="http://paste.openstack.org/show/795738/" target="_blank" class="">http://paste.openstack.org/show/795738/</a></div></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">May I ask if anyone has a recent deployment of Magnum and a working deployment of kubernetes that could share with me the relevant details like the image you have used so that I can try and replicate? </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">To create the cluster template I have been using: <br class=""><div class="gmail_default">openstack coe cluster template create k8s-cluster-template \</div><div class="gmail_default"> --image Fedora-Atomic-27 \</div><div class="gmail_default"> --keypair testpair \</div><div class="gmail_default"> --external-network physnet2vlan20 \</div><div class="gmail_default"> --dns-nameserver 192.168.7.233 \</div><div class="gmail_default"> --flavor 2GB-2vCPU \</div><div class="gmail_default"> --docker-volume-size 15 \</div><div class="gmail_default"> --network-driver flannel \</div><div class="gmail_default"> --coe kubernetes</div><div class="gmail_default"><br class=""></div><div class="gmail_default"><br class=""></div><div class="gmail_default">If I have missed anything, I am happy to provide it. </div><div class="gmail_default"><br class=""></div><div class="gmail_default">Many thanks in advance for any help or pointers on this. </div><div class="gmail_default"><br class=""></div><div class="gmail_default">Regards,</div></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br class=""></div><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class="">Tony Pearce<br class=""><br class=""></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></body></html>