<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Hi Feilong, </div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">"<span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">Let me answer #2 first. Did you try to use CLI? Please make sure using the latest python-magnumclient version. It should work. As for the dashboard issue, please try to use the latest version of magnum-ui. I encourage using resize because the node update is not recommended to use." </span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">I did not attempt the resize via CLI but I can try it. Thank you for your guidance on this :) </span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif">"As for #1, I probably missed something. If the root disk=0MB, where the will operating system be installed? </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">t would be nice if you can share your original requirement to help me understand the issue. e.g why do you have concern the node disk being used?"</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">Sure, although I'd like to understand why you have no concern that the node disk is being used :) I may be missing something here...<br><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">In this environment I have this setup:<br>controller node<br>compute node<br>network storage appliance, integrated with Cinder iscsi.</span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">All VM/Instance data needs to be on the network storage appliance for the reasons;</span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">- it's faster than node storage (Flash storage backed array of disks, provides write-cache and read-cache)<br>- resilience built into the array</span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">- has much higher storage capacity</span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">- is designed for multi-access (ie many connections from hosts)</span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">There are other reasons as well, such as deploying compute nodes as disposable services. Example, a compute node dies resulting in a new node being deployed. Instances are not locked to any node and can be started again on other nodes. </span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">Going back to 2016 when I deployed Openstack Pike, when running post-deployment tests I noticed that the node storage was being consumed even though I have this network storage array. I done some research online and came to the understanding that the reason was the flavors having "root disk" (and swap) having some positive value other than 0MB. So since 2016 I have been using all flavors with disk = 0MB to force the network storage to be used for instance disks and storage. This is working since 2016 Pike, Queens and Train for launching instances (but not Magnum).</span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">The requirement is to utilise network storage (not node storage) - is there some other way that this is achieved today? </span></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br></span></div><div class="gmail_default" style="">I dont understand the point of shared storage options in Openstack if node storage is being consumed for instances. Could you help me understand if this specific environment is just not considered by the Openstack devs? Or some other reason unknown to me? </div><div class="gmail_default" style="">For example, in my (limited) experience with other Virtualisation systems (vmware, ovirt for example) they avoid consuming compute storage for a number of similar reasons to mine. </div><div class="gmail_default" style=""><br></div><div class="gmail_default" style="">So to summarise on this one, I'm not stating that "I am right" here but I am politely asking for more info on the same so I can better understand what I am possibly doing wrong with this deployment or other reasons. </div><div class="gmail_default" style=""><br></div><div class="gmail_default" style="">Lastly thank you again for your time to reply to me, I really appreciate this. </div><div class="gmail_default" style=""><br></div><div class="gmail_default" style="">Regards,</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Tony Pearce<br><br></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 2 Sep 2020 at 16:54, feilong <<a href="mailto:feilong@catalyst.net.nz">feilong@catalyst.net.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Tony,</p>
<p>Let me answer #2 first. Did you try to use CLI? Please make sure
using the latest python-magnumclient version. It should work. As
for the dashboard issue, please try to use the latest version of
magnum-ui. I encourage using resize because the node update is not
recommended to use.</p>
<p>As for #1, I probably missed something. If the root disk=0MB,
where the will operating system be installed? It would be nice if
you can share your original requirement to help me understand the
issue. e.g why do you have concern the node disk being used?</p>
<p><br>
</p>
<div>On 2/09/20 8:12 pm, Tony Pearce wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Feilong, thank you for replying to my
message. <br>
<br>
"1. I'm not sure it's a Magnum issue. Did you try to
draft a simple Heat template to use that flavor and same
image to create instance? Does it work?"<br>
<br>
No I didn't try it and dont think that I know enough
about it to try. I am using Magnum which in turn signals
Heat but I've never used Heat directly. When I use the
new flavour with root disk = 15GB then I dont have any
issue with launching the cluster. But I have a future
issue of consuming all available disk space on the
compute node.<br>
<br>
"2. When you say "resize cluster" failed, what's the
error you got from magnum conductor log?"<br>
<br>
I did not see any error in conductor log. Only the
Magnum API and Horizon log as mentioned. It looks like
horizon was calling bad URLs so maybe this is the reason
why there was no conductor log? Just to mention again
though, that "update cluster" option is working fine to
increase the size of the cluster. <br>
<br>
However my main issue here is with regards to the
flavour being used. Can you or anyone confirm about the
root disk = 0MB? </div>
<div dir="ltr">OR can you or anyone share any information
about how to utilise Magnum/Kubernetes without consuming
Compute node HDD storage? I've been unable to achieve
this and the docs do not give any information about this
specifically<span class="gmail_default"><font face="verdana, sans-serif" color="#666666"> </font>(unless
of course I have missed it?)</span>. The documentation
says I can use any flavour [1].
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br>
</span></div>
<div class="gmail_default">[1]<span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif"> </span><a href="https://docs.openstack.org/magnum/latest/user/" style="color:rgb(102,102,102);font-family:Arial,Helvetica,sans-serif" target="_blank">https://docs.openstack.org/magnum/latest/user/</a><font face="verdana, sans-serif" color="#666666"> </font><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif"> </span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"><br>
</span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)">Regards, </span></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>Tony Pearce<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 2 Sep 2020 at 15:44,
feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Tony,</p>
<p>My comments about your two issues:</p>
<p>1. I'm not sure it's a Magnum issue. Did you try to draft
a simple Heat template to use that flavor and same image
to create instance? Does it work?</p>
<p>2. When you say "resize cluster" failed, what's the error
you got from magnum conductor log?<br>
</p>
<p><br>
</p>
<div>On 1/09/20 9:22 pm, Tony Pearce wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Hi
guys, I hope you are all keeping safe and
well at the moment. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I
am trying to launch Kubernetes clusters into
Openstack Train which has been deployed via
Kayobe (Kayobe as I understand is a wrapper
for kolla-ansible). There have been a few
strange issues here and I've struggled to
isolate them. These issues started recently
after a fresh Openstack deployment some
months ago (around February 2020) to give
some context. This Openstack is not "live"
as I've been trying to get to the bottom of
the issues:</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Issue
1. When trying to launch a cluster we get
error "Resource Create Failed: Forbidden:
Resources.Kube
Masters.Resources[0].Resources.Kube-Master:
Only Volume-Backed Servers Are Allowed For
Flavors With Zero Disk. "</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Issue
2. After successfully creating a cluster of
a smaller node size, the "resize cluster" is
failing (however update the cluster is
working). </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Some
background on this specific environment: </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Deployed
via Kayobe, with these components: <br>
Cinder, Designate, iscsid, Magnum,
Multipathd, neutron provider networks</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">The
Cinder component integrates with iSCSI SAN
storage using the Nimble driver. This is the
only storage. In order to prevent Openstack
from allocating Compute node local HDD as
instance storage, I have all flavours
configured with root disk / ephemeral disk /
swap disk = "0MB". This then results in all
instance data being stored on the backend
Cinder storage appliance. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I
was able to get a cluster deployed by first
creating the template as needed, then when
launching the cluster Horizon prompts you
for items already there in the template such
as number of nodes, node flavour and labels
etc. I re-supplied all of the info (as to
duplicate it) and then tried creating the
cluster. After many many times trying over
the course of a few weeks to a few months it
was successful. I was then able to work
around the issue #2 above to get it
increased in size. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">When
looking at the logs for issue #2, it looks
like some content is missing in the API but
I am not certain. I will include a link to
the pastebin below [1]. <br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">When
trying to resize the cluster, Horizon gives
error: "Error: Unable to resize given
cluster id:
99693dbf-160a-40e0-9ed4-93f3370367ee". I
then searched the controller node /var/log
directory for this ID and found
"horizon.log [:error] [pid 25] Not Found:
/api/container_infra/clusters/99693dbf-160a-40e0-9ed4-93f3370367ee/resize". <br>
Going to the Horizon menu "update cluster"
allows you to increase the number of nodes
and then save/apply the config which does
indeed resize the cluster. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Regarding
issue #1, we've been unable to deploy a
cluster in a new project and the error is
hinting it relates to the flavours having
0MB disk specified, though this error is new
and we've been successful previously with
deploying clusters (albeit with the
hit-and-miss experiences) using the flavour
with 0MB disk as described above. Again I
searched for the (stack) ID after the
failure, in the logs on the controller and I
obtained not much more than the error
already seen with Horizon [2]. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">I
was able to create new flavours with root
disk = 15GB and then successfully deploy a
cluster on the next immediate try. Update
cluster from 3 nodes to 6 nodes was also
immediately successful. However I see the
compute nodes "used" disk space increasing
after increasing the cluster size which is
an issue as the compute node has very
limited HDD capacity (32GB SD card). </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">At
this point I also checked 1) previously
installed cluster using the 0MB disk flavour
and 2) new instances using the 0MB disk
flavour. I notice that the previous cluster
is having host storage allocated but while
the new instance is not having host storage
allocated. So the cluster create success is
using flavour with disk = 0MB while the
result is compute HDD storage being
consumed. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">So
with the above, please may I clarify on the
following? </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">1.
It seems that 0MB disk flavours may not be
supported with magnum now? Could the experts
confirm? :) Is there another way that I
should be configuring this so that compute
node disk is not being consumed (because it
is slow and has limited capacity). </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">2.
The issue #1 looks like a bug to me, is it
known? If not, is this mail enough to get it
realised? </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Pastebin
links as mentioned </div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">[1] <a href="http://paste.openstack.org/show/797316/" style="font-family:Arial,Helvetica,sans-serif" target="_blank">http://paste.openstack.org/show/797316/</a></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">[2] <a href="http://paste.openstack.org/show/797318/" style="font-family:Arial,Helvetica,sans-serif" target="_blank">http://paste.openstack.org/show/797318/</a></div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Many
thanks,</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)">Regards,</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(102,102,102)"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">Tony Pearce<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<pre cols="72">--
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a href="mailto:flwang@catalyst.net.nz" target="_blank">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
------------------------------------------------------ </pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a href="mailto:flwang@catalyst.net.nz" target="_blank">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
------------------------------------------------------ </pre>
</div>
</blockquote></div>