<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.m4975858222606055292msolistparagraph, li.m4975858222606055292msolistparagraph, div.m4975858222606055292msolistparagraph
{mso-style-name:m_4975858222606055292msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">>>> Mike Smith <mismith@overstock.com></p>
<p class="MsoNormal">>>On the plus side, Cinder does allow you to do QOS to limit I/O, whereas I do not believe that’s an option with Nova ephemeral. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You can specify the IOPS limits in the flavor.<br>
Drawbacks:<o:p></o:p></p>
<p class="MsoNormal">* You might end up with a lot of different flavors because of IOPS requirements<o:p></o:p></p>
<p class="MsoNormal">* Modifying an existing flavor won’t retroactively apply it to existing instances<o:p></o:p></p>
<p class="MsoNormal"> You can hack it directly in the database but the instances will still either need to be rebooted or you need to run a lot of virsh command.
<o:p></o:p></p>
<p class="MsoNormal"> (not sure if this is any better for cinder)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">>>> Mike Smith <mismith@overstock.com><o:p></o:p></p>
<p class="MsoNormal">>> And, again depending on the Cinder solution employed, the disk I/O for this kind of setup can be significantly better t<o:p></o:p></p>
<p class="MsoNormal">>>han some other options including Nova ephemeral with a Ceph backend. <o:p></o:p></p>
<p class="MsoNormal">IMHO specifically ceph performance scales out very well (e.g. lots of 100 IOPS instances) but scaling up might be an issue (e.g. running a significant database with lots of sync writes doing 10K IOPS)<o:p></o:p></p>
<div>
<p class="MsoNormal">Even with an optimally tuned SSD/nvme clusters it still might not be as fast as you would like it to be.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">>>>Kimball, Conrad <conrad.kimball@boeing.com><o:p></o:p></p>
<p class="MsoNormal">>> and while it is possible to boot an image onto a new volume this is clumsy <o:p></o:p></p>
<p class="MsoNormal">As mentioned you can make RBD the default backend for ephemeral so you no longer need to specify boot from volume.<o:p></o:p></p>
<p class="MsoNormal">Another option would be to use some other automation tools to bring up our instances.<br>
I recommend looking at e.g. terraform or some other way to automate deployments.<o:p></o:p></p>
<p class="MsoNormal">Running a single command to install a whole environment, and boot from volume if necessary, is really great and makes sure things are reproducible.<o:p></o:p></p>
<p class="MsoNormal">Our tech savvy users like it but if you have people who can just understand the web interface it might be a challenge ;)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Some more points regarding ephemeral local storage:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Pros ephemeral local storage: <o:p></o:p></p>
<p class="MsoNormal">* No SPOF for your cloud (e.g. if a ceph software upgrade goes wrong the whole cloud will hang)<o:p></o:p></p>
<p class="MsoNormal">* Assuming SSDs: great performance<o:p></o:p></p>
<p class="MsoNormal">* Discourages pets, people will get used to instances being down for maintenance or unrecoverable due to hardware failure and will build and automate accordingly<o:p></o:p></p>
<p class="MsoNormal">* No volume storage to manage, assuming you will not offer it anyway<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cons ephemeral local storage:<o:p></o:p></p>
<p class="MsoNormal">* IMHO live migration with block migrations is not really useable
<br>
(the instance will behave a bit slow for some time and e.g. the whole Cassandra or Elasticsearch cluster performance will tank)<o:p></o:p></p>
<p class="MsoNormal">* No independent scaling of compute and space. E.g. with ephemeral you might have lots of disk left but no mem/cpu on the compute node or the other way around.<o:p></o:p></p>
<p class="MsoNormal">* Hardware failure will mean loss of that local data for at least a period of time assuming recoverable at all. With enough compute nodes this will become weekly/daily events.<o:p></o:p></p>
<p class="MsoNormal">* Some pets (e.g. Jenkins boxes) are hard to get rid of even if you control the application landscape to a great degree.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I think that if you have a lot of “pets” or other reasons e.g. a server/rack/availability zone cannot go down for maintenance you probably want to run from volume storage.<o:p></o:p></p>
<p class="MsoNormal">You get your data highly available and can do live-migrations for maintenance.
<o:p></o:p></p>
<p class="MsoNormal">Note that you still have to do some manual work to boot instances somewhere else if a hypervisor goes down but that’s being worked on IIRC.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">>>>Kimball, Conrad <conrad.kimball@boeing.com><o:p></o:p></p>
<p class="MsoNormal">>> Bottom line: it depends what you need, as both options work well and there are people doing both out there in the wild.<o:p></o:p></p>
<p class="MsoNormal">Totally agree.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal">Robert van Leeuwen<o:p></o:p></p>
</div>
</div>
</body>
</html>