<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Tim,</div>
<div><br>
</div>
<div>For the storage backing Elasticsearch data nodes, we have been using one 500GB Cinder volume (backed by a Ceph cluster built from DL380s filled with 1.2TB 10k SAS drives) per data node. However, we’ve found that a VM with 8 vCPU and 64GB RAM can make use
of more than 500GB disk capacity without bottlenecking on CPU or memory, so we are experimenting with 1TB or 1.5TB options per data node. We are also moving to a different tier of storage that uses an array of SSDs instead of spinning rust, though this change
has very little to do with performance and very much to do with the automatic deduplication, compression, and encryption offered by the hardware backend (Solidfire) for that storage tier. # <-- Not a vendor promo; just letting you know what we’re using for
that tier.</div>
<div><br>
</div>
<div>We get a lot of duplicated messages in Elasticsearch since we’re using if for log monitoring, and JSON documents compress very well… so it actually costs us significantly less to leverage a storage hardware platform that provides native deduplication and
compression. Having SSDs in the mix probably helps reduce latency a bit (due to lower seek times), but honestly we didn’t have enough of a latency problem to justify moving away from volumes backed by Ceph.</div>
<div><br>
</div>
<div>Guidance from Elastic is always going to advocate using local SSDs when possible, but I’m pretty sure that’s not what Elastic uses for their own cloud offering…</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Nathan Randall</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Tim Bell <<a href="mailto:Tim.Bell@cern.ch">Tim.Bell@cern.ch</a>><br>
<span style="font-weight:bold">Date: </span>Saturday, September 3, 2016 at 1:12 AM<br>
<span style="font-weight:bold">To: </span>David Medberry <<a href="mailto:openstack@medberry.net">openstack@medberry.net</a>><br>
<span style="font-weight:bold">Cc: </span>openstack-operators <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] ElasticSearch on OpenStack<br>
</div>
<div><br>
</div>
<div xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks. How’s the storage handled ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We’re seeing slow I/O on local storage (which is also limited on space) and latencies with Ceph for block storage.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><<a href="mailto:medberry@gmail.com">medberry@gmail.com</a>> on behalf of David Medberry <<a href="mailto:openstack@medberry.net">openstack@medberry.net</a>><br>
<b>Date: </b>Friday 2 September 2016 at 22:18<br>
<b>To: </b>Tim Bell <<a href="mailto:Tim.Bell@cern.ch">Tim.Bell@cern.ch</a>><br>
<b>Cc: </b>openstack-operators <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [Openstack-operators] ElasticSearch on OpenStack<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-right:0cm" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<div>
<div>
<p class="MsoNormal">Nathan: The page at <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html">
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html</a> gives you good advice on a maximum size for the elasticsearch VM's memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Nathan: suggest you pick a flavor with 64GB RAM or less, then base other sizing things off of that (i.e. choose a flavor with 64GB of RAM and as many CPUs as possible for that RAM allocation, then base disk size on testing of your use case)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Nathan: give java heap 30GB, and leave the rest of the memory to the OS filesystem cache so that Lucene can make best use of it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Nathan: that's mostly it for tuning. elasticsearch publishes many other docs for tuning recommendations, but there isn't anything specific to openstack besides the flavor choice. i personally chose CPU size (8CPU) such that all vCPUs for
each VM would fit on a single NUMA node, which is a best practice for ESXi but not sure if it applies to KVM.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">(resending for clarity)<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Sep 2, 2016 at 6:46 AM, David Medberry <<a href="mailto:openstack@medberry.net" target="_blank">openstack@medberry.net</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Hey Tim, <o:p></o:p></p>
<div>
<p class="MsoNormal">We've just started this effort. I'll see if the guy running the service can comment today.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell <<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Has anyone had experience running ElasticSearch on top of OpenStack VMs ?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Are there any tuning recommendations ?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Thanks</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;color:#888888">Tim</span><span style="color:#888888"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</span>
</body>
</html>