<div dir="ltr"><div class="gmail_default" style="font-size:small;color:#444444">Hello Feilong,</div><div class="gmail_default" style="font-size:small;color:#444444"><br></div><div class="gmail_default" style="font-size:small;color:#444444">I would be happy to join.</div><div class="gmail_default" style="font-size:small;color:#444444"><br></div><div class="gmail_default" style="font-size:small;color:#444444">Can you please assist with the link .</div><div class="gmail_default" style="font-size:small;color:#444444"><br></div><div class="gmail_default" style="font-size:small;color:#444444">Thanks a lot <br clear="all"></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><font face="arial, sans-serif">Regards</font></div><div><font face="arial, sans-serif"><br></font></div><div dir="ltr"><font face="arial, sans-serif">Tony Karera</font></div><div dir="ltr"><br></div><div><br></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 1, 2021 at 4:04 AM feilong <<a href="mailto:feilong@catalyst.net.nz">feilong@catalyst.net.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div>
    <p>Hi Karera,</p>
    <p>Can you please share all the log under <span style="color:rgb(34,34,34)">/var/log/heat-config/heat-</span><span style="color:rgb(34,34,34)">config-script ? Or you can jump in
        #openstack-containers channel on OFTC, I'm online now.</span></p>
    <p><span style="color:rgb(34,34,34)"><br>
      </span></p>
    <p><br>
    </p>
    <div>On 1/09/21 1:51 pm, Karera Tony wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Hey Feilong,</div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
            </div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Thanks a lot.</div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
            </div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">The services
              are fine and indeed the log files are there in the
              directory [<span style="color:rgb(34,34,34)">/var/log/heat-config/heat-</span><span style="color:rgb(34,34,34)">config-script</span>]</div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
            </div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">After
              checking, the master log is fine but the cluster log has
              this error below as I had mentioned earlier</div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
            </div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">
              <div class="gmail_default">Starting to run
                kube-apiserver-to-kubelet-role<br>
                + echo 'Waiting for Kubernetes API...'<br>
                Waiting for Kubernetes API...<br>
                ++ kubectl get --raw=/healthz<br>
                The connection to the server localhost:8080 was refused
                - did you specify the right host or port?<br>
                + '[' ok = '' ']'<br>
              </div>
              <br>
            </div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
            </div>
            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
            </div>
            <div>
              <div dir="ltr">
                <div dir="ltr">
                  <div>
                    <div dir="ltr">
                      <div>
                        <div><font face="arial, sans-serif">Regards</font></div>
                        <div><font face="arial, sans-serif"><br>
                          </font></div>
                        <div dir="ltr"><font face="arial, sans-serif">Tony
                            Karera</font></div>
                        <div dir="ltr"><br>
                        </div>
                        <div><br>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
            <br>
          </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Tue, Aug 31, 2021 at 9:52
          PM feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div>
            <p>Hi Karea,</p>
            <p>Given you can see heat-container-agent container from
              podman which means you should be able to see logs from
              below path:</p>
            <p>[root@k8s-100-eh6s5l6d73ie-master-0 heat-config-script]#
              ls<br>
c64e6ac2-db7e-4786-a387-1d45359812b8-k8s-100-eh6s5l6d73ie-kube_cluster_config-uxpsylgnayjy.log<br>
fa1f6247-51a8-4e70-befa-cbc61ee99e59-k8s-100-eh6s5l6d73ie-kube_masters-kmi423lgbjw3-0-oii7uzemq7aj-master_config-dhfam54i456j.log<br>
              [root@k8s-100-eh6s5l6d73ie-master-0 heat-config-script]#
              pwd<br>
              /var/log/heat-config/heat-config-script<br>
            </p>
            <p>If you can not see the path and the log, then it means
              the heat-container-agent didn't work well. You need to
              check the service status by systemctl command and check
              the log by journalctl. From there, you should be able to
              see why the cluster failed.</p>
            <p><br>
            </p>
            <p><br>
            </p>
            <div>On 1/09/21 1:41 am, Karera Tony wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Dear
                  Ammad,</div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                </div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Sorry to
                  bother you again but I have failed to get the right
                  command to use to check.</div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                </div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Every
                  Kubectl command I run on either the master or worker.</div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">The
                  connection to the server localhost:8080 was refused -
                  did you specify the right host or port?<br>
                </div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">I get the
                  error below.</div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                </div>
                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                </div>
                <div>
                  <div dir="ltr">
                    <div dir="ltr">
                      <div>
                        <div dir="ltr">
                          <div>
                            <div><font face="arial, sans-serif">Regards</font></div>
                            <div><font face="arial, sans-serif"><br>
                              </font></div>
                            <div dir="ltr"><font face="arial,
                                sans-serif">Tony Karera</font></div>
                            <div dir="ltr"><br>
                            </div>
                            <div><br>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
                <br>
              </div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr" class="gmail_attr">On Fri, Aug 27, 2021
                  at 9:15 AM Ammad Syed <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                  <div dir="auto">Your hyperkube services are not
                    started.</div>
                  <div dir="auto"><br>
                  </div>
                  <div dir="auto">You need to check hyperkube services.</div>
                  <div dir="auto"><br>
                  </div>
                  <div dir="auto">Ammad</div>
                  <div><br>
                    <div class="gmail_quote">
                      <div dir="ltr" class="gmail_attr">On Fri, Aug 27,
                        2021 at 10:35 AM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                        wrote:<br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                        <div dir="ltr">
                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Dear
                            Ammad,</div>
                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                          </div>
                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Below
                            is the output of podman ps</div>
                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                          </div>
                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">CONTAINER
                            ID  IMAGE                                  
                                                     COMMAND            
                              CREATED       STATUS           PORTS      
                            NAMES<br>
                            319fbebc2f50  <a href="http://docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1" target="_blank">docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1</a>
                             /usr/bin/start-he...  23 hours ago  Up 23
                            hours ago              heat-container-agent<br>
                            [root@k8s-cluster-2-4faiphvzsmzu-master-0
                            core]#<br>
                          </div>
                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                          </div>
                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                          </div>
                          <div>
                            <div dir="ltr">
                              <div dir="ltr">
                                <div>
                                  <div dir="ltr">
                                    <div>
                                      <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial, sans-serif">Regards</font></div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                        <div dir="ltr">
                          <div>
                            <div dir="ltr">
                              <div dir="ltr">
                                <div>
                                  <div dir="ltr">
                                    <div>
                                      <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial, sans-serif"><br>
                                        </font></div>
                                      <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial, sans-serif">Tony
                                          Karera</font></div>
                                      <div dir="ltr"><br>
                                      </div>
                                      <div><br>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                          <br>
                        </div>
                        <br>
                        <div class="gmail_quote">
                          <div dir="ltr" class="gmail_attr">On Thu, Aug
                            26, 2021 at 9:54 AM Ammad Syed <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                            wrote:<br>
                          </div>
                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                            <div dir="ltr">The output in
logfile 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd
                              is incomplete.
                              <div><br>
                              </div>
                              <div>There should be the installation and
                                configuration of many other things that
                                are missing. Also it looks that
                                hyperkube is not installed.</div>
                              <div><br>
                              </div>
                              <div>Can you check the response of "podman
                                ps" command on master nodes. </div>
                              <div><br>
                              </div>
                              <div>Ammad </div>
                            </div>
                            <br>
                            <div class="gmail_quote">
                              <div dir="ltr" class="gmail_attr">On Thu,
                                Aug 26, 2021 at 11:30 AM Karera Tony
                                <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                wrote:<br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div dir="ltr">
                                  <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Here
                                    is the beginning of the Log</div>
                                  <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                  </div>
                                  <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Starting
                                    to run
                                    kube-apiserver-to-kubelet-role<br>
                                    + echo 'Waiting for Kubernetes
                                    API...'<br>
                                    Waiting for Kubernetes API...<br>
                                    ++ kubectl get --raw=/healthz<br>
                                    The connection to the server
                                    localhost:8080 was refused - did you
                                    specify the right host or port?<br>
                                    + '[' ok = '' ']'<br>
                                    <br>
                                    <br>
                                  </div>
                                  <div>
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">
                                            <div>
                                              <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                  sans-serif">Regards</font></div>
                                              <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                  sans-serif"><br>
                                                </font></div>
                                              <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                  sans-serif">Tony
                                                  Karera</font></div>
                                              <div dir="ltr"><br>
                                              </div>
                                              <div><br>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                  <br>
                                </div>
                                <br>
                                <div class="gmail_quote">
                                  <div dir="ltr" class="gmail_attr">On
                                    Thu, Aug 26, 2021 at 7:53 AM Bharat
                                    Kunwar <<a href="mailto:bharat@stackhpc.com" target="_blank">bharat@stackhpc.com</a>>
                                    wrote:<br>
                                  </div>
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                    <div dir="auto">I assume these are
                                      from the master nodes? Can you
                                      share the logs shortly after
                                      creation rather than when it times
                                      out? I think there is some missing
                                      logs from the top.<br>
                                      <br>
                                      <div dir="ltr">Sent from my iPhone</div>
                                      <div dir="ltr"><br>
                                        <blockquote type="cite">On 26
                                          Aug 2021, at 06:14, Karera
                                          Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                          wrote:<br>
                                          <br>
                                        </blockquote>
                                      </div>
                                      <blockquote type="cite">
                                        <div dir="ltr">
                                          <div dir="ltr">
                                            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Hello
                                              Guys,</div>
                                            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                            </div>
                                            <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Attached
                                              are the two logs from the
/var/log/heat-config/heat-config-script directory<br>
                                            </div>
                                            <div>
                                              <div dir="ltr">
                                                <div dir="ltr">
                                                  <div>
                                                    <div dir="ltr">
                                                      <div>
                                                        <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Regards</font></div>
                                                        <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif"><br>
                                                          </font></div>
                                                        <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Tony
                                                          Karera</font></div>
                                                        <div dir="ltr"><br>
                                                        </div>
                                                        <div><br>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                            <br>
                                          </div>
                                          <br>
                                          <div class="gmail_quote">
                                            <div dir="ltr" class="gmail_attr">On Thu,
                                              Aug 26, 2021 at 5:59 AM
                                              Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                              wrote:<br>
                                            </div>
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div dir="ltr">
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Dear Sir,</div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                </div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">You are right.</div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                </div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">I am getting this error <br>
                                                </div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                </div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">kubectl get --raw=/healthz<br>
                                                  The connection to the
                                                  server localhost:8080
                                                  was refused - did you
                                                  specify the right host
                                                  or port?</div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                </div>
                                                <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                </div>
                                                <div>
                                                  <div dir="ltr">
                                                    <div dir="ltr">
                                                      <div>
                                                        <div dir="ltr">
                                                          <div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Regards</font></div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif"><br>
                                                          </font></div>
                                                          <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Tony
                                                          Karera</font></div>
                                                          <div dir="ltr"><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                                <br>
                                              </div>
                                              <br>
                                              <div class="gmail_quote">
                                                <div dir="ltr" class="gmail_attr">On
                                                  Wed, Aug 25, 2021 at
                                                  10:55 PM Bharat Kunwar
                                                  <<a href="mailto:bharat@stackhpc.com" target="_blank">bharat@stackhpc.com</a>>
                                                  wrote:<br>
                                                </div>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div dir="auto">I’d
                                                    check the logs under
/var/log/heat-config.<br>
                                                    <br>
                                                    <div dir="ltr">Sent
                                                      from my iPhone</div>
                                                    <div dir="ltr"><br>
                                                      <blockquote type="cite">On
                                                        25 Aug 2021, at
                                                        19:39, Karera
                                                        Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                        wrote:<br>
                                                        <br>
                                                      </blockquote>
                                                    </div>
                                                    <blockquote type="cite">
                                                      <div dir="ltr">
                                                        <div dir="ltr">
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">DeaR Ammad,</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">I was able to make the
                                                          communication
                                                          work and the
                                                          Worker nodes
                                                          were created
                                                          as well but
                                                          the cluster
                                                          failed.</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">I logged in to the master
                                                          node and there
                                                          was no error
                                                          but below are
                                                          the error when
                                                          I run
                                                          systemctl
                                                          status
                                                          heat-container-agent
                                                          on the worker
                                                          noed.</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Aug 25 17:52:24
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:52:55
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:53:26
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:53:57
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:54:28
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:54:59
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:55:29
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:56:00
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:56:31
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          Aug 25
                                                          17:57:02
                                                          cluster1-fmkpva3nozf7-node-0
                                                          podman[2268]:
/var/lib/os-collect-config/local-data not found. Skipping<br>
                                                          </div>
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Regards</font></div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif"><br>
                                                          </font></div>
                                                          <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Tony
                                                          Karera</font></div>
                                                          <div dir="ltr"><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                        </div>
                                                        <br>
                                                        <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Wed, Aug 25, 2021 at 10:38 AM Ammad Syed <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">Yes,
                                                          keystone,
                                                          Heat,
                                                          Barbicane and
                                                          magnum public
                                                          endpoints must
                                                          be reachable
                                                          from master
                                                          and worker
                                                          nodes.
                                                          <div><br>
                                                          </div>
                                                          <div>You can
                                                          use below
                                                          guide for the
                                                          reference as
                                                          well.</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11" target="_blank">https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11</a><br>
                                                          <div><br>
                                                          </div>
                                                          <div>Ammad</div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Wed, Aug 25, 2021 at 12:08 PM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Hello Ammad,</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">I have deployed using the
                                                          given image
                                                          but I think
                                                          there is an
                                                          issue with
                                                          keystone as
                                                          per the screen
                                                          shot below
                                                          when I checked
                                                          the master
                                                          node's
                                                          heat-container-agent
                                                          status</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">
                                                          <div><image.png></div>
                                                          <br>
                                                          </div>
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Regards</font></div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif"><br>
                                                          </font></div>
                                                          <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Tony
                                                          Karera</font></div>
                                                          <div dir="ltr"><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Wed, Aug 25, 2021 at 8:28 AM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Hello Ammad,</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">I actually first used that
                                                          one and it was
                                                          also getting
                                                          stuck.</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">I will try this one again
                                                          and update you
                                                          with the Logs
                                                          though.</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Regards</font></div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif"><br>
                                                          </font></div>
                                                          <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Tony
                                                          Karera</font></div>
                                                          <div dir="ltr"><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Wed, Aug 25, 2021 at 8:25 AM Ammad Syed <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">It
                                                          seems from the
                                                          logs that you
                                                          are using
                                                          fedora atomic.
                                                          Can you try
                                                          with FCOS 32
                                                          image.
                                                          <div><br>
                                                          </div>
                                                          <div><a href="https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz" target="_blank">https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz</a><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Ammad</div>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Wed, Aug 25, 2021 at 11:20 AM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Hello Sir,</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Attached is the Log file</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Regards</font></div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif"><br>
                                                          </font></div>
                                                          <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Tony
                                                          Karera</font></div>
                                                          <div dir="ltr"><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Wed, Aug 25, 2021 at 7:31 AM Ammad Syed <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">Hi Karera,
                                                          <div><br>
                                                          </div>
                                                          <div>Can you
                                                          share us the
                                                          full log file.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Ammad</div>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Wed, Aug 25, 2021 at 9:42 AM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Hello Guys,</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Thanks a lot for the help
                                                          but
                                                          unfortunately
                                                          I dont see
                                                          much
                                                          information in
                                                          the log file
                                                          indicating a
                                                          failure apart
                                                          from the log
                                                          that keeps
                                                          appearing.</div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br>
                                                          </div>
                                                          <div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">
                                                          <div><image.png></div>
                                                          <br>
                                                          <br>
                                                          </div>
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Regards</font></div>
                                                          <div><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif"><br>
                                                          </font></div>
                                                          <div dir="ltr"><font style="font-family:arial,sans-serif;color:rgb(0,0,0)" face="arial,
                                                          sans-serif">Tony
                                                          Karera</font></div>
                                                          <div dir="ltr"><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr" class="gmail_attr">On Tue, Aug 24, 2021 at 8:12 PM Mohammed Naser <<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Also
                                                          check out
                                                          /var/log/cloud-init.log
                                                          :)<br>
                                                          <br>
                                                          On Tue, Aug
                                                          24, 2021 at
                                                          1:39 PM Ammad
                                                          Syed <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                                                          wrote:<br>
                                                          ><br>
                                                          > Then
                                                          check
                                                          journalctl -xe
                                                          or status of
                                                          heat agent
                                                          service
                                                          status.<br>
                                                          ><br>
                                                          ><br>
                                                          > Ammad<br>
                                                          > On Tue,
                                                          Aug 24, 2021
                                                          at 10:36 PM
                                                          Karera Tony
                                                          <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
                                                          >><br>
                                                          >> Hello
                                                          Ammad,<br>
                                                          >><br>
                                                          >> There
                                                          is no
                                                          directory or
                                                          log relevant
                                                          to heat in the
                                                          /var/log
                                                          directory<br>
                                                          >><br>
                                                          >>
                                                          Regards<br>
                                                          >><br>
                                                          >> Tony
                                                          Karera<br>
                                                          >><br>
                                                          >><br>
                                                          >><br>
                                                          >><br>
                                                          >> On
                                                          Tue, Aug 24,
                                                          2021 at 12:43
                                                          PM Ammad Syed
                                                          <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                                                          wrote:<br>
                                                          >>><br>
                                                          >>>
                                                          Hi Karera,<br>
                                                          >>><br>
                                                          >>>
                                                          Login to
                                                          master node
                                                          and check the
                                                          logs of heat
                                                          agent in var
                                                          log. There
                                                          must be
                                                          something the
                                                          cluster is
                                                          stucking
                                                          somewhere in
                                                          creating.<br>
                                                          >>><br>
                                                          >>>
                                                          Ammad<br>
                                                          >>><br>
                                                          >>>
                                                          On Tue, Aug
                                                          24, 2021 at
                                                          3:41 PM Karera
                                                          Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
>>>><br>
>>>> Hello Ammad,<br>
>>>><br>
>>>> I had done as explained and it worked upto a certain
                                                          point. The
                                                          master node
                                                          was created
                                                          but the
                                                          cluster
                                                          remained in
                                                          Creation in
                                                          progress for
                                                          over an hour
                                                          and failed
                                                          with error
                                                          below<br>
>>>><br>
>>>> Stack Faults<br>
>>>> as follows:<br>
>>>> default-master<br>
>>>> Timed out<br>
>>>> default-worker<br>
>>>> Timed out<br>
>>>><br>
>>>><br>
>>>> Regards<br>
>>>><br>
>>>> Tony Karera<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Tue, Aug 24, 2021 at 9:25 AM Ammad Syed <<a href="mailto:syedammad83@gmail.com" target="_blank">syedammad83@gmail.com</a>>
                                                          wrote:<br>
>>>>><br>
>>>>> Hi Tony,<br>
>>>>><br>
>>>>> You can try by creating your private vxlan network
                                                          prior to
                                                          deployment of
                                                          cluster and
                                                          explicitly
                                                          create your
                                                          cluster in
                                                          vxlan network.<br>
>>>>><br>
>>>>> --fixed-network private --fixed-subnet
                                                          private-subnet<br>
>>>>><br>
>>>>> You can specify above while creating a cluster.<br>
>>>>><br>
>>>>> Ammad<br>
>>>>><br>
>>>>> On Tue, Aug 24, 2021 at 11:59 AM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
>>>>>><br>
>>>>>> Hello MOhamed,<br>
>>>>>><br>
>>>>>> I think the Kubernetes cluster is ok but it
                                                          when I deploy
                                                          it, It creates
                                                          a fixed
                                                          network using
                                                          vlan which I
                                                          am not using
                                                          for internal
                                                          networks.<br>
>>>>>><br>
>>>>>> When I create a a vxlan Network and use it in
                                                          the cluster
                                                          creation, It
                                                          fails. Is
                                                          there a trick
                                                          around this ?<br>
>>>>>> Regards<br>
>>>>>><br>
>>>>>> Tony Karera<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On Fri, Aug 20, 2021 at 9:00 AM feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>
                                                          wrote:<br>
>>>>>>><br>
>>>>>>> Oooh, are you using Swarm? I don't think
                                                          that driver is
                                                          well
                                                          maintained. I
                                                          didn't see any
                                                          interest in
                                                          the last 4
                                                          years since I
                                                          involved in
                                                          the Magnum
                                                          project. If
                                                          there is no
                                                          specific
                                                          reason, I
                                                          would suggest
                                                          go for k8s.<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On 20/08/21 5:08 pm, Mohammed Naser wrote:<br>
>>>>>>><br>
>>>>>>> Please keep replies on list so others can
                                                          help too.<br>
>>>>>>><br>
>>>>>>> I don’t know how well tested the Swarm
                                                          driver is at
                                                          this point. I
                                                          believe most
                                                          Magnum users
                                                          are using it
                                                          for Kubernetes
                                                          only.<br>
>>>>>>><br>
>>>>>>> On Fri, Aug 20, 2021 at 1:05 AM Karera Tony
                                                          <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
>>>>>>>><br>
>>>>>>>> Hello Naser,<br>
>>>>>>>><br>
>>>>>>>> Please check below.<br>
>>>>>>>><br>
>>>>>>>> openstack coe cluster template create
                                                          swarm-cluster-template1
                                                          \<br>
>>>>>>>>                      --image
                                                          fedora-atomic-latest
                                                          \<br>
>>>>>>>>                      --external-network
                                                          External_1700\<br>
>>>>>>>>                      --dns-nameserver
                                                          8.8.8.8 \<br>
>>>>>>>>                      --master-flavor
                                                          m1.small \<br>
>>>>>>>>                      --flavor m1.small
                                                          \<br>
>>>>>>>>                      --coe swarm<br>
>>>>>>>> openstack coe cluster create
                                                          swarm-cluster
                                                          \<br>
>>>>>>>>                       
                                                           --cluster-template
swarm-cluster-template \<br>
>>>>>>>>                         --master-count
                                                          1 \<br>
>>>>>>>>                         --node-count 2
                                                          \<br>
>>>>>>>>                         --keypair
                                                          Newkey<br>
>>>>>>>><br>
>>>>>>>> Regards<br>
>>>>>>>><br>
>>>>>>>> Tony Karera<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> On Fri, Aug 20, 2021 at 7:03 AM
                                                          Mohammed Naser
                                                          <<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>>
                                                          wrote:<br>
>>>>>>>>><br>
>>>>>>>>> What does your cluster template and
                                                          cluster create
                                                          command look
                                                          like?<br>
>>>>>>>>><br>
>>>>>>>>> On Fri, Aug 20, 2021 at 12:59 AM
                                                          Karera Tony
                                                          <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>>
                                                          wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>> Hello Wang,<br>
>>>>>>>>>><br>
>>>>>>>>>> Thanks for the feedback.<br>
>>>>>>>>>><br>
>>>>>>>>>> Unfortunately Octavia is not
                                                          deployed in my
                                                          environment
                                                          (at least not
                                                          yet) and LB is
                                                          not enabled on
                                                          either the
                                                          cluster
                                                          template or
                                                          the cluster
                                                          itself.<br>
>>>>>>>>>><br>
>>>>>>>>>> Regards<br>
>>>>>>>>>><br>
>>>>>>>>>> Tony Karera<br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> On Fri, Aug 20, 2021 at 6:52 AM
                                                          feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>
                                                          wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hi Karera,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> It's probably a bug. If you
                                                          do have
                                                          Octavia
                                                          deployed, can
                                                          you try to not
                                                          disable the LB
                                                          and see how it
                                                          goes?<br>
>>>>>>>>>>><br>
>>>>>>>>>>><br>
>>>>>>>>>>> On 20/08/21 4:18 pm, Karera
                                                          Tony wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hello Team,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> I deployed Openstack Wallby
                                                          on Ubuntu20
                                                          and enabled
                                                          Magum, however
                                                          when I create
                                                          a cluster I
                                                          get the error
                                                          below.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Status Reason<br>
>>>>>>>>>>> ERROR: Property error: :
                                                          resources.api_lb.properties:
                                                          : Property
                                                          allowed_cidrs
                                                          not assigned<br>
>>>>>>>>>>> Can someone advise on where
                                                          I could be
                                                          wrong. Btw, I
                                                          disabled load
                                                          balancer while
                                                          creating the
                                                          cluster.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Regards<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Tony Karera<br>
>>>>>>>>>>><br>
>>>>>>>>>>><br>
>>>>>>>>>>> --<br>
>>>>>>>>>>> Cheers & Best regards,<br>
>>>>>>>>>>>
------------------------------------------------------------------------------<br>
>>>>>>>>>>> Feilong Wang (王飞龙) (he/him)<br>
>>>>>>>>>>> Head of Research &
                                                          Development<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Catalyst Cloud<br>
>>>>>>>>>>> Aotearoa's own<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Mob: +64 21 0832 6348 | <a href="http://www.catalystcloud.nz" rel="noreferrer" target="_blank">www.catalystcloud.nz</a><br>
>>>>>>>>>>> Level 6, <a href="https://www.google.com/maps/search/150+Willis+Street,+Wellington+6011,+New+Zealand?entry=gmail&source=g" target="_blank">150 Willis Street, Wellington
                                                          6011, New
                                                          Zealand</a><br>
>>>>>>>>>>><br>
>>>>>>>>>>> CONFIDENTIALITY NOTICE:
                                                          This email is
                                                          intended for
                                                          the named
                                                          recipients
                                                          only.<br>
>>>>>>>>>>> It may contain privileged,
                                                          confidential
                                                          or copyright
                                                          information.
                                                          If you are<br>
>>>>>>>>>>> not the named recipient,
                                                          any use,
                                                          reliance upon,
                                                          disclosure or
                                                          copying of
                                                          this<br>
>>>>>>>>>>> email or its attachments is
                                                          unauthorised.
                                                          If you have
                                                          received this
                                                          email in<br>
>>>>>>>>>>> error, please reply via
                                                          email or call
                                                          +64 21 0832
                                                          6348.<br>
>>>>>>>>>>>
------------------------------------------------------------------------------<br>
>>>>>>>>><br>
>>>>>>>>> --<br>
>>>>>>>>> Mohammed Naser<br>
>>>>>>>>> VEXXHOST, Inc.<br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> Mohammed Naser<br>
>>>>>>> VEXXHOST, Inc.<br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> Cheers & Best regards,<br>
>>>>>>>
------------------------------------------------------------------------------<br>
>>>>>>> Feilong Wang (王飞龙) (he/him)<br>
>>>>>>> Head of Research & Development<br>
>>>>>>><br>
>>>>>>> Catalyst Cloud<br>
>>>>>>> Aotearoa's own<br>
>>>>>>><br>
>>>>>>> Mob: +64 21 0832 6348 | <a href="http://www.catalystcloud.nz" rel="noreferrer" target="_blank">www.catalystcloud.nz</a><br>
>>>>>>> Level 6, <a href="https://www.google.com/maps/search/150+Willis+Street,+Wellington+6011,+New+Zealand?entry=gmail&source=g" target="_blank">150 Willis Street, Wellington
                                                          6011, New
                                                          Zealand</a><br>
>>>>>>><br>
>>>>>>> CONFIDENTIALITY NOTICE: This email is
                                                          intended for
                                                          the named
                                                          recipients
                                                          only.<br>
>>>>>>> It may contain privileged, confidential or
                                                          copyright
                                                          information.
                                                          If you are<br>
>>>>>>> not the named recipient, any use, reliance
                                                          upon,
                                                          disclosure or
                                                          copying of
                                                          this<br>
>>>>>>> email or its attachments is unauthorised.
                                                          If you have
                                                          received this
                                                          email in<br>
>>>>>>> error, please reply via email or call +64
                                                          21 0832 6348.<br>
>>>>>>>
------------------------------------------------------------------------------<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --<br>
>>>>> Regards,<br>
>>>>><br>
>>>>><br>
>>>>> Syed Ammad Ali<br>
                                                          >>><br>
                                                          >>><br>
                                                          >>><br>
                                                          >>>
                                                          --<br>
                                                          >>>
                                                          Regards,<br>
                                                          >>><br>
                                                          >>><br>
                                                          >>>
                                                          Syed Ammad Ali<br>
                                                          ><br>
                                                          > --<br>
                                                          > Regards,<br>
                                                          ><br>
                                                          ><br>
                                                          > Syed
                                                          Ammad Ali<br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          -- <br>
                                                          Mohammed Naser<br>
                                                          VEXXHOST, Inc.<br>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr">Regards,
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Syed
                                                          Ammad Ali</div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr">Regards,
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Syed
                                                          Ammad Ali</div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr">Regards,
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Syed
                                                          Ammad Ali</div>
                                                          </div>
                                                          </blockquote>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                  </div>
                                                </blockquote>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <div><29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log></div>
                                          <div><6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log></div>
                                        </div>
                                      </blockquote>
                                    </div>
                                  </blockquote>
                                </div>
                              </blockquote>
                            </div>
                            <br clear="all">
                            <div><br>
                            </div>
                            -- <br>
                            <div dir="ltr">Regards,
                              <div><br>
                              </div>
                              <div><br>
                              </div>
                              <div>Syed Ammad Ali</div>
                            </div>
                          </blockquote>
                        </div>
                      </blockquote>
                    </div>
                  </div>
                  -- <br>
                  <div dir="ltr">Regards,
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                    <div>Syed Ammad Ali</div>
                  </div>
                </blockquote>
              </div>
            </blockquote>
            <pre cols="72">-- 
Cheers & Best regards,
------------------------------------------------------------------------------
Feilong Wang (王飞龙) (he/him)
Head of Research & Development

Catalyst Cloud
Aotearoa's own

Mob: +64 21 0832 6348 | <a href="http://www.catalystcloud.nz" target="_blank">www.catalystcloud.nz</a>
Level 6, 150 Willis Street, Wellington 6011, New Zealand

CONFIDENTIALITY NOTICE: This email is intended for the named recipients only.
It may contain privileged, confidential or copyright information. If you are 
not the named recipient, any use, reliance upon, disclosure or copying of this 
email or its attachments is unauthorised. If you have received this email in 
error, please reply via email or call +64 21 0832 6348.
------------------------------------------------------------------------------</pre>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <pre cols="72">-- 
Cheers & Best regards,
------------------------------------------------------------------------------
Feilong Wang (王飞龙) (he/him)
Head of Research & Development

Catalyst Cloud
Aotearoa's own

Mob: +64 21 0832 6348 | <a href="http://www.catalystcloud.nz" target="_blank">www.catalystcloud.nz</a>
Level 6, 150 Willis Street, Wellington 6011, New Zealand

CONFIDENTIALITY NOTICE: This email is intended for the named recipients only.
It may contain privileged, confidential or copyright information. If you are 
not the named recipient, any use, reliance upon, disclosure or copying of this 
email or its attachments is unauthorised. If you have received this email in 
error, please reply via email or call +64 21 0832 6348.
------------------------------------------------------------------------------</pre>
  </div>

</blockquote></div>