<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Your cluster failed because of <a
href="https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh#L24">https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh#L24</a></p>
<p>So please just remove your label "tls_enabled=true" and try
again.</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 1/09/21 2:27 pm, Karera Tony wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CA+69TL2TmJDMKERj=8vgMtKLmx68aZ2YezPpnopmN6RJWeXiTQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_default" style="font-size:small;color:#444444">Attached
are the logs<br clear="all">
</div>
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font face="arial, sans-serif">Regards</font></div>
<div><font face="arial, sans-serif"><br>
</font></div>
<div dir="ltr"><font face="arial, sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Sep 1, 2021 at 4:11 AM
Karera Tony <<a href="mailto:tonykarera@gmail.com"
moz-do-not-send="true">tonykarera@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Hello Feilong,</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I would be
happy to join.</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Can you please
assist with the link .</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Thanks a lot <br
clear="all">
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font face="arial, sans-serif">Regards</font></div>
<div><font face="arial, sans-serif"><br>
</font></div>
<div dir="ltr"><font face="arial, sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Sep 1, 2021 at
4:04 AM feilong <<a
href="mailto:feilong@catalyst.net.nz" target="_blank"
moz-do-not-send="true">feilong@catalyst.net.nz</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Karera,</p>
<p>Can you please share all the log under <span
style="color:rgb(34,34,34)">/var/log/heat-config/heat-</span><span
style="color:rgb(34,34,34)">config-script ? Or you
can jump in #openstack-containers channel on OFTC,
I'm online now.</span></p>
<p><span style="color:rgb(34,34,34)"><br>
</span></p>
<p><br>
</p>
<div>On 1/09/21 1:51 pm, Karera Tony wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Hey
Feilong,</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Thanks
a lot.</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">The
services are fine and indeed the log files are
there in the directory [<span
style="color:rgb(34,34,34)">/var/log/heat-config/heat-</span><span
style="color:rgb(34,34,34)">config-script</span>]</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">After
checking, the master log is fine but the
cluster log has this error below as I had
mentioned earlier</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">
<div class="gmail_default">Starting to run
kube-apiserver-to-kubelet-role<br>
+ echo 'Waiting for Kubernetes API...'<br>
Waiting for Kubernetes API...<br>
++ kubectl get --raw=/healthz<br>
The connection to the server localhost:8080
was refused - did you specify the right host
or port?<br>
+ '[' ok = '' ']'<br>
</div>
<br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font face="arial, sans-serif">Regards</font></div>
<div><font face="arial, sans-serif"><br>
</font></div>
<div dir="ltr"><font face="arial,
sans-serif">Tony Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Aug 31,
2021 at 9:52 PM feilong <<a
href="mailto:feilong@catalyst.net.nz"
target="_blank" moz-do-not-send="true">feilong@catalyst.net.nz</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Karea,</p>
<p>Given you can see heat-container-agent
container from podman which means you should
be able to see logs from below path:</p>
<p>[root@k8s-100-eh6s5l6d73ie-master-0
heat-config-script]# ls<br>
c64e6ac2-db7e-4786-a387-1d45359812b8-k8s-100-eh6s5l6d73ie-kube_cluster_config-uxpsylgnayjy.log<br>
fa1f6247-51a8-4e70-befa-cbc61ee99e59-k8s-100-eh6s5l6d73ie-kube_masters-kmi423lgbjw3-0-oii7uzemq7aj-master_config-dhfam54i456j.log<br>
[root@k8s-100-eh6s5l6d73ie-master-0
heat-config-script]# pwd<br>
/var/log/heat-config/heat-config-script<br>
</p>
<p>If you can not see the path and the log, then
it means the heat-container-agent didn't work
well. You need to check the service status by
systemctl command and check the log by
journalctl. From there, you should be able to
see why the cluster failed.</p>
<p><br>
</p>
<p><br>
</p>
<div>On 1/09/21 1:41 am, Karera Tony wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Dear
Ammad,</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Sorry to
bother you again but I have failed to get
the right command to use to check.</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Every
Kubectl command I run on either the master
or worker.</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">The
connection to the server localhost:8080
was refused - did you specify the right
host or port?<br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I
get the error below.</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font face="arial,
sans-serif">Regards</font></div>
<div><font face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
face="arial, sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri,
Aug 27, 2021 at 9:15 AM Ammad Syed <<a
href="mailto:syedammad83@gmail.com"
target="_blank" moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="auto">Your hyperkube services
are not started.</div>
<div dir="auto"><br>
</div>
<div dir="auto">You need to check
hyperkube services.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Ammad</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Fri, Aug 27, 2021 at 10:35 AM Karera
Tony <<a
href="mailto:tonykarera@gmail.com"
target="_blank"
moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Dear
Ammad,</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Below
is the output of podman ps</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">CONTAINER
ID IMAGE
COMMAND
CREATED STATUS
PORTS NAMES<br>
319fbebc2f50 <a
href="http://docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1"
target="_blank"
moz-do-not-send="true">docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1</a>
/usr/bin/start-he... 23 hours
ago Up 23 hours ago
heat-container-agent<br>
[root@k8s-cluster-2-4faiphvzsmzu-master-0 core]#<br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Thu, Aug 26, 2021 at 9:54 AM
Ammad Syed <<a
href="mailto:syedammad83@gmail.com"
target="_blank"
moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">The output in
logfile 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd
is incomplete.
<div><br>
</div>
<div>There should be the
installation and
configuration of many other
things that are missing.
Also it looks that hyperkube
is not installed.</div>
<div><br>
</div>
<div>Can you check the
response of "podman ps"
command on master nodes. </div>
<div><br>
</div>
<div>Ammad </div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Thu,
Aug 26, 2021 at 11:30 AM
Karera Tony <<a
href="mailto:tonykarera@gmail.com"
target="_blank"
moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Here is the beginning of the
Log</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Starting to run
kube-apiserver-to-kubelet-role<br>
+ echo 'Waiting for
Kubernetes API...'<br>
Waiting for Kubernetes
API...<br>
++ kubectl get
--raw=/healthz<br>
The connection to the
server localhost:8080
was refused - did you
specify the right host
or port?<br>
+ '[' ok = '' ']'<br>
<br>
<br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On
Thu, Aug 26, 2021 at
7:53 AM Bharat Kunwar
<<a
href="mailto:bharat@stackhpc.com"
target="_blank"
moz-do-not-send="true">bharat@stackhpc.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="auto">I assume
these are from the
master nodes? Can you
share the logs shortly
after creation rather
than when it times
out? I think there is
some missing logs from
the top.<br>
<br>
<div dir="ltr">Sent
from my iPhone</div>
<div dir="ltr"><br>
<blockquote
type="cite">On 26
Aug 2021, at
06:14, Karera Tony
<<a
href="mailto:tonykarera@gmail.com"
target="_blank"
moz-do-not-send="true">tonykarera@gmail.com</a>> wrote:<br>
<br>
</blockquote>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Hello Guys,</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Attached are the two logs
from the
/var/log/heat-config/heat-config-script directory<br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On
Thu, Aug 26,
2021 at 5:59
AM Karera Tony
<<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Dear Sir,</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">You are right.</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I am getting this error <br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">kubectl get --raw=/healthz<br>
The connection
to the server
localhost:8080
was refused -
did you
specify the
right host or
port?</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 10:55 PM Bharat Kunwar <<a
href="mailto:bharat@stackhpc.com" target="_blank" moz-do-not-send="true">bharat@stackhpc.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
dir="auto">I’d
check the logs
under
/var/log/heat-config.<br>
<br>
<div dir="ltr">Sent
from my iPhone</div>
<div dir="ltr"><br>
<blockquote
type="cite">On
25 Aug 2021,
at 19:39,
Karera Tony
<<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
<br>
</blockquote>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">DeaR Ammad,</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I was able to make the
communication
work and the
Worker nodes
were created
as well but
the cluster
failed.</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I logged in to the master
node and there
was no error
but below are
the error when
I run
systemctl
status
heat-container-agent
on the worker
noed.</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Aug 25 17:52:24
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:52:55
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:53:26
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:53:57
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:54:28
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:54:59
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:55:29
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:56:00
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:56:31
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
Aug 25
17:57:02
cluster1-fmkpva3nozf7-node-0
podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 10:38 AM Ammad Syed <<a
href="mailto:syedammad83@gmail.com"
target="_blank" moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Yes,
keystone,
Heat,
Barbicane and
magnum public
endpoints must
be reachable
from master
and worker
nodes.
<div><br>
</div>
<div>You can
use below
guide for the
reference as
well.</div>
<div><br>
</div>
<div><a
href="https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11"
target="_blank" moz-do-not-send="true">https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11</a><br>
<div><br>
</div>
<div>Ammad</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 12:08 PM Karera Tony <<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Hello Ammad,</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I have deployed using the
given image
but I think
there is an
issue with
keystone as
per the screen
shot below
when I checked
the master
node's
heat-container-agent
status</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">
<div><image.png></div>
<br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 8:28 AM Karera Tony <<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Hello Ammad,</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I actually first used that
one and it was
also getting
stuck.</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">I will try this one again
and update you
with the Logs
though.</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 8:25 AM Ammad Syed <<a
href="mailto:syedammad83@gmail.com"
target="_blank" moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">It
seems from the
logs that you
are using
fedora atomic.
Can you try
with FCOS 32
image.
<div><br>
</div>
<div><a
href="https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz"
target="_blank" moz-do-not-send="true">https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz</a><br>
</div>
<div><br>
</div>
<div>Ammad</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 11:20 AM Karera Tony <<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Hello Sir,</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Attached is the Log file</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 7:31 AM Ammad Syed <<a
href="mailto:syedammad83@gmail.com"
target="_blank" moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Karera,
<div><br>
</div>
<div>Can you
share us the
full log file.</div>
<div><br>
</div>
<div>Ammad</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Aug 25, 2021 at 9:42 AM Karera Tony <<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Hello Guys,</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">Thanks a lot for the help
but
unfortunately
I dont see
much
information in
the log file
indicating a
failure apart
from the log
that keeps
appearing.</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)"><br>
</div>
<div
class="gmail_default"
style="font-size:small;color:rgb(68,68,68)">
<div><image.png></div>
<br>
<br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif">Regards</font></div>
<div><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)"
face="arial,
sans-serif"><br>
</font></div>
<div dir="ltr"><font
style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
sans-serif">Tony
Karera</font></div>
<div dir="ltr"><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Tue, Aug 24, 2021 at 8:12 PM Mohammed Naser <<a
href="mailto:mnaser@vexxhost.com" target="_blank" moz-do-not-send="true">mnaser@vexxhost.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Also
check out
/var/log/cloud-init.log
:)<br>
<br>
On Tue, Aug
24, 2021 at
1:39 PM Ammad
Syed <<a
href="mailto:syedammad83@gmail.com"
target="_blank" moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
><br>
> Then
check
journalctl -xe
or status of
heat agent
service
status.<br>
><br>
><br>
> Ammad<br>
> On Tue,
Aug 24, 2021
at 10:36 PM
Karera Tony
<<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
>><br>
>> Hello
Ammad,<br>
>><br>
>> There
is no
directory or
log relevant
to heat in the
/var/log
directory<br>
>><br>
>>
Regards<br>
>><br>
>> Tony
Karera<br>
>><br>
>><br>
>><br>
>><br>
>> On
Tue, Aug 24,
2021 at 12:43
PM Ammad Syed
<<a
href="mailto:syedammad83@gmail.com"
target="_blank" moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
>>><br>
>>>
Hi Karera,<br>
>>><br>
>>>
Login to
master node
and check the
logs of heat
agent in var
log. There
must be
something the
cluster is
stucking
somewhere in
creating.<br>
>>><br>
>>>
Ammad<br>
>>><br>
>>>
On Tue, Aug
24, 2021 at
3:41 PM Karera
Tony <<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
>>>><br>
>>>> Hello Ammad,<br>
>>>><br>
>>>> I had done as explained and it worked upto a certain
point. The
master node
was created
but the
cluster
remained in
Creation in
progress for
over an hour
and failed
with error
below<br>
>>>><br>
>>>> Stack Faults<br>
>>>> as follows:<br>
>>>> default-master<br>
>>>> Timed out<br>
>>>> default-worker<br>
>>>> Timed out<br>
>>>><br>
>>>><br>
>>>> Regards<br>
>>>><br>
>>>> Tony Karera<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Tue, Aug 24, 2021 at 9:25 AM Ammad Syed <<a
href="mailto:syedammad83@gmail.com"
target="_blank" moz-do-not-send="true">syedammad83@gmail.com</a>>
wrote:<br>
>>>>><br>
>>>>> Hi Tony,<br>
>>>>><br>
>>>>> You can try by creating your private vxlan network
prior to
deployment of
cluster and
explicitly
create your
cluster in
vxlan network.<br>
>>>>><br>
>>>>> --fixed-network private --fixed-subnet
private-subnet<br>
>>>>><br>
>>>>> You can specify above while creating a cluster.<br>
>>>>><br>
>>>>> Ammad<br>
>>>>><br>
>>>>> On Tue, Aug 24, 2021 at 11:59 AM Karera Tony <<a
href="mailto:tonykarera@gmail.com" target="_blank"
moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
>>>>>><br>
>>>>>> Hello MOhamed,<br>
>>>>>><br>
>>>>>> I think the Kubernetes cluster is ok but it
when I deploy
it, It creates
a fixed
network using
vlan which I
am not using
for internal
networks.<br>
>>>>>><br>
>>>>>> When I create a a vxlan Network and use it in
the cluster
creation, It
fails. Is
there a trick
around this ?<br>
>>>>>> Regards<br>
>>>>>><br>
>>>>>> Tony Karera<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On Fri, Aug 20, 2021 at 9:00 AM feilong <<a
href="mailto:feilong@catalyst.net.nz" target="_blank"
moz-do-not-send="true">feilong@catalyst.net.nz</a>>
wrote:<br>
>>>>>>><br>
>>>>>>> Oooh, are you using Swarm? I don't think
that driver is
well
maintained. I
didn't see any
interest in
the last 4
years since I
involved in
the Magnum
project. If
there is no
specific
reason, I
would suggest
go for k8s.<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On 20/08/21 5:08 pm, Mohammed Naser wrote:<br>
>>>>>>><br>
>>>>>>> Please keep replies on list so others can
help too.<br>
>>>>>>><br>
>>>>>>> I don’t know how well tested the Swarm
driver is at
this point. I
believe most
Magnum users
are using it
for Kubernetes
only.<br>
>>>>>>><br>
>>>>>>> On Fri, Aug 20, 2021 at 1:05 AM Karera Tony
<<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
>>>>>>>><br>
>>>>>>>> Hello Naser,<br>
>>>>>>>><br>
>>>>>>>> Please check below.<br>
>>>>>>>><br>
>>>>>>>> openstack coe cluster template create
swarm-cluster-template1
\<br>
>>>>>>>> --image
fedora-atomic-latest
\<br>
>>>>>>>> --external-network
External_1700\<br>
>>>>>>>> --dns-nameserver
8.8.8.8 \<br>
>>>>>>>> --master-flavor
m1.small \<br>
>>>>>>>> --flavor m1.small
\<br>
>>>>>>>> --coe swarm<br>
>>>>>>>> openstack coe cluster create
swarm-cluster
\<br>
>>>>>>>>
--cluster-template
swarm-cluster-template \<br>
>>>>>>>> --master-count
1 \<br>
>>>>>>>> --node-count 2
\<br>
>>>>>>>> --keypair
Newkey<br>
>>>>>>>><br>
>>>>>>>> Regards<br>
>>>>>>>><br>
>>>>>>>> Tony Karera<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> On Fri, Aug 20, 2021 at 7:03 AM
Mohammed Naser
<<a
href="mailto:mnaser@vexxhost.com"
target="_blank" moz-do-not-send="true">mnaser@vexxhost.com</a>>
wrote:<br>
>>>>>>>>><br>
>>>>>>>>> What does your cluster template and
cluster create
command look
like?<br>
>>>>>>>>><br>
>>>>>>>>> On Fri, Aug 20, 2021 at 12:59 AM
Karera Tony
<<a
href="mailto:tonykarera@gmail.com"
target="_blank" moz-do-not-send="true">tonykarera@gmail.com</a>>
wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>> Hello Wang,<br>
>>>>>>>>>><br>
>>>>>>>>>> Thanks for the feedback.<br>
>>>>>>>>>><br>
>>>>>>>>>> Unfortunately Octavia is not
deployed in my
environment
(at least not
yet) and LB is
not enabled on
either the
cluster
template or
the cluster
itself.<br>
>>>>>>>>>><br>
>>>>>>>>>> Regards<br>
>>>>>>>>>><br>
>>>>>>>>>> Tony Karera<br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> On Fri, Aug 20, 2021 at 6:52 AM
feilong <<a
href="mailto:feilong@catalyst.net.nz" target="_blank"
moz-do-not-send="true">feilong@catalyst.net.nz</a>>
wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hi Karera,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> It's probably a bug. If you
do have
Octavia
deployed, can
you try to not
disable the LB
and see how it
goes?<br>
>>>>>>>>>>><br>
>>>>>>>>>>><br>
>>>>>>>>>>> On 20/08/21 4:18 pm, Karera
Tony wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hello Team,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> I deployed Openstack Wallby
on Ubuntu20
and enabled
Magum, however
when I create
a cluster I
get the error
below.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Status Reason<br>
>>>>>>>>>>> ERROR: Property error: :
resources.api_lb.properties:
: Property
allowed_cidrs
not assigned<br>
>>>>>>>>>>> Can someone advise on where
I could be
wrong. Btw, I
disabled load
balancer while
creating the
cluster.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Regards<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Tony Karera<br>
>>>>>>>>>>><br>
>>>>>>>>>>><br>
>>>>>>>>>>> --<br>
>>>>>>>>>>> Cheers & Best regards,<br>
>>>>>>>>>>>
------------------------------------------------------------------------------<br>
>>>>>>>>>>> Feilong Wang (王飞龙) (he/him)<br>
>>>>>>>>>>> Head of Research &
Development<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Catalyst Cloud<br>
>>>>>>>>>>> Aotearoa's own<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Mob: +64 21 0832 6348 | <a
href="http://www.catalystcloud.nz" rel="noreferrer" target="_blank"
moz-do-not-send="true">www.catalystcloud.nz</a><br>
>>>>>>>>>>> Level 6, <a
href="https://www.google.com/maps/search/150+Willis+Street,+Wellington+6011,+New+Zealand?entry=gmail&source=g"
target="_blank" moz-do-not-send="true">150 Willis Street, Wellington
6011, New
Zealand</a><br>
>>>>>>>>>>><br>
>>>>>>>>>>> CONFIDENTIALITY NOTICE:
This email is
intended for
the named
recipients
only.<br>
>>>>>>>>>>> It may contain privileged,
confidential
or copyright
information.
If you are<br>
>>>>>>>>>>> not the named recipient,
any use,
reliance upon,
disclosure or
copying of
this<br>
>>>>>>>>>>> email or its attachments is
unauthorised.
If you have
received this
email in<br>
>>>>>>>>>>> error, please reply via
email or call
+64 21 0832
6348.<br>
>>>>>>>>>>>
------------------------------------------------------------------------------<br>
>>>>>>>>><br>
>>>>>>>>> --<br>
>>>>>>>>> Mohammed Naser<br>
>>>>>>>>> VEXXHOST, Inc.<br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> Mohammed Naser<br>
>>>>>>> VEXXHOST, Inc.<br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> Cheers & Best regards,<br>
>>>>>>>
------------------------------------------------------------------------------<br>
>>>>>>> Feilong Wang (王飞龙) (he/him)<br>
>>>>>>> Head of Research & Development<br>
>>>>>>><br>
>>>>>>> Catalyst Cloud<br>
>>>>>>> Aotearoa's own<br>
>>>>>>><br>
>>>>>>> Mob: +64 21 0832 6348 | <a
href="http://www.catalystcloud.nz"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.catalystcloud.nz</a><br>
>>>>>>> Level 6, <a
href="https://www.google.com/maps/search/150+Willis+Street,+Wellington+6011,+New+Zealand?entry=gmail&source=g"
target="_blank" moz-do-not-send="true">150 Willis Street, Wellington
6011, New
Zealand</a><br>
>>>>>>><br>
>>>>>>> CONFIDENTIALITY NOTICE: This email is
intended for
the named
recipients
only.<br>
>>>>>>> It may contain privileged, confidential or
copyright
information.
If you are<br>
>>>>>>> not the named recipient, any use, reliance
upon,
disclosure or
copying of
this<br>
>>>>>>> email or its attachments is unauthorised.
If you have
received this
email in<br>
>>>>>>> error, please reply via email or call +64
21 0832 6348.<br>
>>>>>>>
------------------------------------------------------------------------------<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --<br>
>>>>> Regards,<br>
>>>>><br>
>>>>><br>
>>>>> Syed Ammad Ali<br>
>>><br>
>>><br>
>>><br>
>>>
--<br>
>>>
Regards,<br>
>>><br>
>>><br>
>>>
Syed Ammad Ali<br>
><br>
> --<br>
> Regards,<br>
><br>
><br>
> Syed
Ammad Ali<br>
<br>
<br>
<br>
-- <br>
Mohammed Naser<br>
VEXXHOST, Inc.<br>
</blockquote>
</div>
</blockquote>
</div>
<br
clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Regards,
<div><br>
</div>
<div><br>
</div>
<div>Syed
Ammad Ali</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br
clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Regards,
<div><br>
</div>
<div><br>
</div>
<div>Syed
Ammad Ali</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br
clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Regards,
<div><br>
</div>
<div><br>
</div>
<div>Syed
Ammad Ali</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
<div><29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log></div>
<div><6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Regards,
<div><br>
</div>
<div><br>
</div>
<div>Syed Ammad Ali</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
-- <br>
<div dir="ltr">Regards,
<div><br>
</div>
<div><br>
</div>
<div>Syed Ammad Ali</div>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Cheers & Best regards,
------------------------------------------------------------------------------
Feilong Wang (王飞龙) (he/him)
Head of Research & Development
Catalyst Cloud
Aotearoa's own
Mob: +64 21 0832 6348 | <a href="http://www.catalystcloud.nz" target="_blank" moz-do-not-send="true">www.catalystcloud.nz</a>
Level 6, 150 Willis Street, Wellington 6011, New Zealand
CONFIDENTIALITY NOTICE: This email is intended for the named recipients only.
It may contain privileged, confidential or copyright information. If you are
not the named recipient, any use, reliance upon, disclosure or copying of this
email or its attachments is unauthorised. If you have received this email in
error, please reply via email or call +64 21 0832 6348.
------------------------------------------------------------------------------</pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Cheers & Best regards,
------------------------------------------------------------------------------
Feilong Wang (王飞龙) (he/him)
Head of Research & Development
Catalyst Cloud
Aotearoa's own
Mob: +64 21 0832 6348 | <a href="http://www.catalystcloud.nz" target="_blank" moz-do-not-send="true">www.catalystcloud.nz</a>
Level 6, 150 Willis Street, Wellington 6011, New Zealand
CONFIDENTIALITY NOTICE: This email is intended for the named recipients only.
It may contain privileged, confidential or copyright information. If you are
not the named recipient, any use, reliance upon, disclosure or copying of this
email or its attachments is unauthorised. If you have received this email in
error, please reply via email or call +64 21 0832 6348.
------------------------------------------------------------------------------</pre>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Cheers & Best regards,
------------------------------------------------------------------------------
Feilong Wang (王飞龙) (he/him)
Head of Research & Development
Catalyst Cloud
Aotearoa's own
Mob: +64 21 0832 6348 | <a class="moz-txt-link-abbreviated" href="http://www.catalystcloud.nz">www.catalystcloud.nz</a>
Level 6, 150 Willis Street, Wellington 6011, New Zealand
CONFIDENTIALITY NOTICE: This email is intended for the named recipients only.
It may contain privileged, confidential or copyright information. If you are
not the named recipient, any use, reliance upon, disclosure or copying of this
email or its attachments is unauthorised. If you have received this email in
error, please reply via email or call +64 21 0832 6348.
------------------------------------------------------------------------------</pre>
</body>
</html>