<div dir="ltr"><span style="font-size:12.8px">As VW mentioned, we're running our ELK clusters on a nova undercloud.</span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">My elasticsearch cluster looks like this:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">* 3 master nodes</div><div style="font-size:12.8px">* 6 client nodes (these nodes are running elasticsearch as clients only - no data storage. they handle all api requests. they're also running kibana)</div><div style="font-size:12.8px">* 40-ish data nodes</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">The data nodes have 30gb memory with 15gb allocated to java heap. We're using local storage on ssd.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I don't have the elasticsearch configs available to me at the moment, but we're using fairly standard master/client/data node config settings. We've tuned some of the threadpool settings. Some of our indices are also using longer refresh times. We've increased the cluster and shard recovery settings as the defaults were fairly low for the size of our data and number of indices. I've also increased max file descriptors.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Most of our data in elasticsearch is from from logs (mostly openstack service logs via logstash), and most of the default config settings have worked well for us so far.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I think the biggest performance improvements we saw came from using ssd drives, using the latest versions of elasticsearch, and splitting all client/api request to dedicated elasticsearch client nodes.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Depending on your data and how you're using it, you may need or want to visit other elasticsearch settings, increase number of replicas, increase (or disable) index refresh time, increase number of shards, etc.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I use kopf [1] for checking the cluster status, and rally [2] for performance testing.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">-Nick</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">[1] <a href="https://github.com/lmenezes/elasticsearch-kopf" target="_blank">https://github.com/<wbr>lmenezes/elasticsearch-kopf</a></div><div style="font-size:12.8px">[2] <a href="https://github.com/elastic/rally" target="_blank">https://github.com/<wbr>elastic/rally</a></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 2, 2016 at 7:59 AM, Matt Van Winkle <span dir="ltr"><<a href="mailto:mvanwink@rackspace.com" target="_blank">mvanwink@rackspace.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
<div>We are running entire ELK clusters on our undercloud. Let me see if I can point our resident metrics expert at this as well.</div>
<div><br>
</div>
<div>VW<br>
<br>
Sent from my iPhone</div><div><div class="gmail-h5">
<div><br>
On Sep 2, 2016, at 7:50 AM, David Medberry <<a href="mailto:openstack@medberry.net" target="_blank">openstack@medberry.net</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Hey Tim,
<div>We've just started this effort. I'll see if the guy running the service can comment today.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell <span dir="ltr">
<<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="white" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Has anyone had experience running ElasticSearch on top of OpenStack VMs ?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Are there any tuning recommendations ?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Thanks<span><font color="#888888"><u></u><u></u></font></span></span></p>
<span><font color="#888888">
<p class="MsoNormal"><span style="font-size:11pt">Tim<u></u><u></u></span></p>
</font></span></div>
</div>
<br>
______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.open<wbr>stack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-operators</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>OpenStack-operators mailing list</span><br>
<span><a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.<wbr>openstack.org</a></span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a></span><br>
</div>
</blockquote>
</div></div></div>
<br>______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
<br></blockquote></div><br>
</div></div>