<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Hi Bharat, in fact I had noticed that property when creating the
      image in OS and made some more research about this.</p>
    <p>I now have 2 images (atomic and coreos) and have set the
      different flags in the image creation process.</p>
    <p>The documentation from Victoria to latest has also changed to
      this:</p>
    <p>Victoria (Kubernetes cluster creation) - Create a cluster
      template for a Kubernetes cluster using the
      <code class="docutils literal notranslate"><span class="pre">fedora-atomic-latest</span></code>
      image</p>
    <p>latest - Create a cluster template for a Kubernetes cluster using
      the
      <code class="docutils literal notranslate"><span class="pre">fedora-coreos-latest</span></code>
      image</p>
    <p>So in the end it seems that the CoreOS image is now being
      suggested for the Kubernetes cluster creation. The bootstrapping
      process seems to be handled by ignition which handles the ssh keys
      (I need to find out in more detail how the ignition mechanism
      works to better understand this process)</p>
    <p>Thanks<br>
    </p>
    <div class="moz-cite-prefix">On 08/04/2021 08:19, Bharat Kunwar
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:BD33EB1C-683E-4815-AD4C-DD6C03F4381D@stackhpc.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      As in, do you have that label set in the image property?<br>
      <br>
      <div dir="ltr">Sent from my iPhone</div>
      <div dir="ltr"><br>
        <blockquote type="cite">On 8 Apr 2021, at 07:05, Bharat Kunwar
          <a class="moz-txt-link-rfc2396E" href="mailto:bharat@stackhpc.com"><bharat@stackhpc.com></a> wrote:<br>
          <br>
        </blockquote>
      </div>
      <blockquote type="cite">
        <div dir="ltr">
          <meta http-equiv="content-type" content="text/html;
            charset=UTF-8">
          Is your os_distro=fedora-coreos or fedora-atomic?<br>
          <br>
          <div dir="ltr">Sent from my iPhone</div>
          <div dir="ltr"><br>
            <blockquote type="cite">On 7 Apr 2021, at 22:12, Luke
              Camilleri <a class="moz-txt-link-rfc2396E" href="mailto:luke.camilleri@zylacomputing.com"><luke.camilleri@zylacomputing.com></a> wrote:<br>
              <br>
            </blockquote>
          </div>
          <blockquote type="cite">
            <div dir="ltr">
              <meta http-equiv="Content-Type" content="text/html;
                charset=UTF-8">
              <p>Hi Bharat, I am on Victoria so that should satisfy the
                requirement:</p>
              <p># rpm -qa | grep -i heat<br>
                openstack-heat-api-cfn-15.0.0-1.el8.noarch<br>
                openstack-heat-api-15.0.0-1.el8.noarch<br>
                python3-heatclient-2.2.1-2.el8.noarch<br>
                openstack-heat-common-15.0.0-1.el8.noarch<br>
                openstack-heat-engine-15.0.0-1.el8.noarch<br>
                openstack-heat-ui-4.0.0-1.el8.noarch<br>
              </p>
              <p>So from what I can see during the stack's step at
                OS::Heat::SoftwareConfig is the step that gets the data
                right?</p>
              <p>agent_config:<br>
                    type: OS::Heat::SoftwareConfig<br>
                    properties:<br>
                      group: ungrouped<br>
                      config:<br>
                        list_join:<br>
                          - "\n"<br>
                          -<br>
                            - str_replace:<br>
                                template: {get_file: user_data.json}<br>
                                params:<br>
                                  __HOSTNAME__: {get_param: name}<br>
                                  __SSH_KEY_VALUE__: {get_param:
                ssh_public_key}<br>
                                  __OPENSTACK_CA__: {get_param:
                openstack_ca}<br>
                                  __CONTAINER_INFRA_PREFIX__:<br>
              </p>
              <p><br>
              </p>
              <p>In the stack I can see that the step below which
                corresponds to the agent_config above and has just been
                initialized:<br>
              </p>
              <table id="resources" class="table table-striped datatable
                tablesorter tablesorter-default">
                <tbody>
                  <tr class="ajax-update status_up odd"
                    data-object-id="kube_cluster_config"
                    data-update-interval="2500"
data-update-url="/dashboard/project/stacks/stack/84330fda-efe6-4b94-96da-b836b60e2586/?action=row_update&table=resources&obj_id=kube_cluster_config"
                    id="resources__row__kube_cluster_config">
                    <td class="sortable anchor normal_column"><a
href="https://portal.zylacloud.com/dashboard/project/stacks/stack/84330fda-efe6-4b94-96da-b836b60e2586/kube_cluster_config/"
                        moz-do-not-send="true">kube_cluster_config</a></td>
                    <td class="sortable normal_column"><br>
                    </td>
                    <td class="sortable normal_column">
                      OS::Heat::SoftwareConfig </td>
                    <td class="sortable normal_column"> 46 minutes </td>
                    <td class="sortable normal_column"> Init Complete </td>
                    <td class="sortable normal_column"><br>
                    </td>
                  </tr>
                </tbody>
              </table>
              <p>My question here would be:</p>
              <p>1- is the file the user_data</p>
              <p>2- at which step is this data aplied to the instance as
                from the fedora docs (
                <a class="moz-txt-link-freetext"
href="https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/#_ignition_overview"
                  moz-do-not-send="true">https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/#_ignition_overview</a>
                ) this step seems to be at the initial stages of the
                boot process</p>
              <p>Thanks in advance for any assistance<br>
              </p>
              <div class="moz-cite-prefix">On 07/04/2021 22:54, Bharat
                Kunwar wrote:<br>
              </div>
              <blockquote type="cite"
                cite="mid:4A94086F-F79A-4EC4-8E3F-A6AE8EDF4C16@stackhpc.com">
                <meta http-equiv="content-type" content="text/html;
                  charset=UTF-8">
                The ssh key gets injected via ignition which is why it’s
                not present in the HOT template. You need minimum train
                release of Heat for this to work however.<br>
                <br>
                <div dir="ltr">Sent from my iPhone</div>
                <div dir="ltr"><br>
                  <blockquote type="cite">On 7 Apr 2021, at 21:45, Luke
                    Camilleri <a class="moz-txt-link-rfc2396E"
                      href="mailto:luke.camilleri@zylacomputing.com"
                      moz-do-not-send="true"><luke.camilleri@zylacomputing.com></a>
                    wrote:<br>
                    <br>
                  </blockquote>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">
                    <meta http-equiv="Content-Type" content="text/html;
                      charset=UTF-8">
                    <p>Hello Ammad and thanks for your assistance. I
                      followed the guide and it has all the details and
                      steps except for one thing, the ssh key is not
                      being passed over to the instance, if I deploy an
                      instance from that image and pass the ssh key it
                      works fine but if I use the image as part of the
                      HOT it lists the key as "-"</p>
                    <p>Did you have this issue by any chance? Never
                      thought I would be asking this question as it is a
                      basic thing but I find it very strange that this
                      is not working. I tried to pass the ssh key in
                      either the template or in the cluster creation
                      command but for both options the Key Name metadata
                      option for the instance remains "None" when the
                      instance is deployed.</p>
                    <p>I then went on and checked the yaml file the
                      resource uses that loads/gets the parameters
/usr/lib/python3.6/site-packages/magnum/drivers/k8s_fedora_coreos_v1/templates/kubemaster.yaml
                      has the below yaml configurations:</p>
                    <p>kube-master:<br>
                          type: OS::Nova::Server<br>
                          condition: image_based<br>
                          properties:<br>
                            name: {get_param: name}<br>
                            image: {get_param: server_image}<br>
                            flavor: {get_param: master_flavor}<br>
                                                                     
                      MISSING ----->   key_name: {get_param:
                      ssh_key_name}<br>
                            user_data_format: SOFTWARE_CONFIG<br>
                            software_config_transport: POLL_SERVER_HEAT<br>
                            user_data: {get_resource: agent_config}<br>
                            networks:<br>
                              - port: {get_resource: kube_master_eth0}<br>
                            scheduler_hints: { group: { get_param:
                      nodes_server_group_id }}<br>
                            availability_zone: {get_param:
                      availability_zone}<br>
                    </p>
                    <p>kube-master-bfv:<br>
                          type: OS::Nova::Server<br>
                          condition: volume_based<br>
                          properties:<br>
                            name: {get_param: name}<br>
                            flavor: {get_param: master_flavor}<br>
                                                                     
                      MISSING ----->   key_name: {get_param:
                      ssh_key_name}<br>
                            user_data_format: SOFTWARE_CONFIG<br>
                            software_config_transport: POLL_SERVER_HEAT<br>
                            user_data: {get_resource: agent_config}<br>
                            networks:<br>
                              - port: {get_resource: kube_master_eth0}<br>
                            scheduler_hints: { group: { get_param:
                      nodes_server_group_id }}<br>
                            availability_zone: {get_param:
                      availability_zone}<br>
                            block_device_mapping_v2:<br>
                              - boot_index: 0<br>
                                volume_id: {get_resource:
                      kube_node_volume}<br>
                    </p>
                    <p>If i add the lines which show as missing, then
                      everything works well and the key is actually
                      injected in the kubemaster. Did anyone had this
                      issue?<br>
                    </p>
                    <div class="moz-cite-prefix">On 07/04/2021 10:24,
                      Ammad Syed wrote:<br>
                    </div>
                    <blockquote type="cite"
cite="mid:CAKOoz51UbO07fjGsGOiWfnH+JEc++UHkZN=4AS18PKbryrYB1Q@mail.gmail.com">
                      <meta http-equiv="content-type"
                        content="text/html; charset=UTF-8">
                      <div dir="auto">Hi Luke,</div>
                      <div dir="auto"><br>
                      </div>
                      <div dir="auto">You may refer to below guide for
                        magnum installation and its template </div>
                      <div dir="auto"><br>
                      </div>
                      <div><a
href="https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=10"
                          moz-do-not-send="true">https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=10</a></div>
                      <div dir="auto"><br>
                      </div>
                      <div dir="auto">It worked pretty well for me.</div>
                      <div dir="auto"><br>
                      </div>
                      <div dir="auto">- Ammad<br>
                        <div class="gmail_quote" dir="auto">
                          <div dir="ltr" class="gmail_attr">On Wed, Apr
                            7, 2021 at 5:02 AM Luke Camilleri <<a
                              href="mailto:luke.camilleri@zylacomputing.com"
                              moz-do-not-send="true">luke.camilleri@zylacomputing.com</a>>
                            wrote:<br>
                          </div>
                          <blockquote class="gmail_quote"
                            style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
                            <div>
                              <p>Thanks for your quick reply. Do you
                                have a download link for that image as I
                                cannot find an archive for the 32
                                release?</p>
                              <p>As for the image upload into openstack
                                you still use the fedora-atomic property
                                right to be available for coe
                                deployments?<br>
                              </p>
                            </div>
                            <div>
                              <div>On 07/04/2021 00:03, feilong wrote:<br>
                              </div>
                              <blockquote type="cite">
                                <p>Hi Luke,</p>
                                <p>The Fedora Atomic driver has been
                                  deprecated a while since the Fedora
                                  Atomic has been deprecated by
                                  upstream. For now, I would suggest
                                  using Fedora CoreOS <span>32.20201104.3.0</span></p>
                                <p><span
                                    style="font-family:BlinkMacSystemFont,-apple-system,"Segoe
UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira
                                    Sans","Droid
                                    Sans","Helvetica
Neue",Helvetica,Arial,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important;background-color:rgb(255,255,255);color:rgb(54,54,54)">The
                                    latest version of Fedora CoreOS is
                                    33.xxx, but there are something when
                                    booting based my testing, see </span><span
style="font-family:BlinkMacSystemFont,-apple-system,"Segoe
UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira
                                    Sans","Droid
                                    Sans","Helvetica
Neue",Helvetica,Arial,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important;background-color:rgb(255,255,255);color:rgb(54,54,54)"><a
href="https://github.com/coreos/fedora-coreos-tracker/issues/735"
                                      target="_blank"
                                      style="font-family:BlinkMacSystemFont,-apple-system,"Segoe
UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira
                                      Sans","Droid
                                      Sans","Helvetica
                                      Neue",Helvetica,Arial,sans-serif"
                                      moz-do-not-send="true">https://github.com/coreos/fedora-coreos-tracker/issues/735</a></span></p>
                                <p>Please feel free to let me know if
                                  you have any question about using
                                  Magnum. We're using stable/victoria on
                                  our public cloud and it works very
                                  well. I can share our public templates
                                  if you want. Cheers.</p>
                                <p><br>
                                </p>
                                <p><br>
                                </p>
                                <div>On 7/04/21 9:51 am, Luke Camilleri
                                  wrote:<br>
                                </div>
                                <blockquote type="cite">
                                  <div>
                                    <p>We have insatlled magnum
                                      following the installation guide
                                      here <a
href="https://docs.openstack.org/magnum/victoria/install/install-rdo.html"
                                        target="_blank"
                                        moz-do-not-send="true">https://docs.openstack.org/magnum/victoria/install/install-rdo.html</a>
                                      and the process was quite smooth
                                      but we have been having some
                                      issues with the deployment of the
                                      clusters.</p>
                                    <p>The image being used as per the
                                      documentation is <a
href="https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180419.0/CloudImages/x86_64/images/Fedora-Atomic-27-20180419.0.x86_64"
                                        target="_blank"
                                        moz-do-not-send="true">https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180419.0/CloudImages/x86_64/images/Fedora-Atomic-27-20180419.0.x86_64</a></p>
                                    <p>Our first issue was that podman
                                      was being used even if we
                                      specified the use_podman=false
                                      (since the image above did not
                                      include podman) but this was
                                      resulting in a timeout and the
                                      cluster would fail to deploy. We
                                      have then installed podman in the
                                      image and the cluster progressed a
                                      bit further <br>
                                    </p>
                                    <p><i>+ echo 'WARNING Attempt 60:
                                        Trying to install kubectl.
                                        Sleeping 5s'</i><i><br>
                                      </i><i>+ sleep 5s</i><i><br>
                                      </i><i>+ ssh -F
                                        /srv/magnum/.ssh/config
                                        root@localhost '/usr/bin/podman
                                        run     --entrypoint
                                        /bin/bash     --name
                                        install-kubectl     --net
                                        host     --privileged    
                                        --rm     --user root    
                                        --volume
                                        /srv/magnum/bin:/host/srv/magnum/bin    
                                        <a
                                          href="http://k8s.gcr.io/hyperkube:v1.15.7"
                                          target="_blank"
                                          moz-do-not-send="true">k8s.gcr.io/hyperkube:v1.15.7</a>    
                                        -c '\''cp /usr/local/bin/kubectl
/host/srv/magnum/bin/kubectl'\'''</i><i><br>
                                      </i><i>bash: /usr/bin/podman: No
                                        such file or directory</i><i><br>
                                      </i><i>ERROR Unable to install
                                        kubectl. Abort.</i><i><br>
                                      </i><i>+ i=61</i><i><br>
                                      </i><i>+ '[' 61 -gt 60 ']'</i><i><br>
                                      </i><i>+ echo 'ERROR Unable to
                                        install kubectl. Abort.'</i><i><br>
                                      </i><i>+ exit 1</i><br>
                                    </p>
                                    <p>The cluster is now failing here
                                      at "<span>kube_cluster_deploy" and
                                        when checking the logs on the
                                        master node we noticed the
                                        following in the log files:</span></p>
                                    <p><span><i>Starting to run
                                          kube-apiserver-to-kubelet-role</i><i><br>
                                        </i><i>Waiting for Kubernetes
                                          API...</i><i><br>
                                        </i><i>+ echo 'Waiting for
                                          Kubernetes API...'</i><i><br>
                                        </i><i>++ curl --silent <a
                                            href="http://127.0.0.1:8080/healthz"
                                            target="_blank"
                                            moz-do-not-send="true">http://127.0.0.1:8080/healthz</a></i><i><br>
                                        </i><i>+ '[' ok = '' ']'</i><i><br>
                                        </i><i>+ sleep 5</i><br>
                                      </span></p>
                                    <p>This is because the kubernetes
                                      API server is not installed
                                      either. I have noticed some
                                      scripts that should handle the
                                      installation but I would like to
                                      know if anyone here has had
                                      similar issues with a clean
                                      Victoria installation.</p>
                                    Also should we have to install any
                                    packages in the fedora atomic image
                                    file or should the installation
                                    requirements be part of the stack?</div>
                                  <div><br>
                                  </div>
                                  <div>Thanks in advance for any
                                    asistance</div>
                                  <br>
                                </blockquote>
                                <pre cols="72" style="font-family:monospace">-- 
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a href="mailto:flwang@catalyst.net.nz" target="_blank" style="font-family:monospace" moz-do-not-send="true">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, <a href="https://www.google.com/maps/search/150+Willis+Street,+Wellington?entry=gmail&source=g" style="font-family:monospace" moz-do-not-send="true">150 Willis Street, Wellington</a>
------------------------------------------------------ </pre>
                              </blockquote>
                            </div>
                          </blockquote>
                        </div>
                      </div>
                      -- <br>
                      <div dir="ltr" class="gmail_signature"
                        data-smartmail="gmail_signature">Regards,
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div>Syed Ammad Ali</div>
                      </div>
                    </blockquote>
                  </div>
                </blockquote>
              </blockquote>
            </div>
          </blockquote>
        </div>
      </blockquote>
    </blockquote>
  </body>
</html>