<div dir="ltr"><br><div id="gmail-magicdomid3166" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><span class="gmail-" style="margin:0px;padding:1px 0px">Hi,</span></div><div id="gmail-magicdomid2941" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><br style="margin:0px;padding:0px"></div><div id="gmail-magicdomid3750" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">tl;dr Besides sending CI logs, does someone use Logstash on Opendev to push information to the new Elasticsearch (Opensearch) service?</span></div><div id="gmail-magicdomid3238" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><br style="margin:0px;padding:0px"></div><div id="gmail-magicdomid3751" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">Long version:</span></div><div id="gmail-magicdomid1874" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><br style="margin:0px;padding:0px"></div><div id="gmail-magicdomid3752" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px"> I am currently looking into a new log workflow for OpenDev's OpenSearch that is using the logscraper tool [1], as an improvement over the current workflow [2].</span></div><div id="gmail-magicdomid3753" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">I spotted a few bottlenecks that are complicating pushing logs to the new Elasticsearch service.</span></div><div id="gmail-magicdomid3754" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">The most important bottlenecks are:</span></div><div id="gmail-magicdomid3756" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><ul class="gmail-list-indent1" style="margin:0px 0px 0px 1.5em;padding:0px;list-style-type:none"><li>- gearman worker - this service needs at least 10 instances to keep up with the current influx of build logs.</li><li style="margin:0px;padding:0px"><span class="gmail-" style="margin:0px;padding:1px 0px">                               As a reminder, this service consumes information prepared by gearman client, downloads the logs, adds information and</span></li></ul></div><div id="gmail-magicdomid3757" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><ul class="gmail-list-indent1" style="margin:0px 0px 0px 1.5em;padding:0px;list-style-type:none"><li style="margin:0px;padding:0px"><span class="gmail-" style="margin:0px;padding:1px 0px">                               pushes data to the logstash service,</span></li><li style="margin:0px;padding:0px">- logstash service - this service is "freezing" from time to time and it requires many resources such as CPU and RAM.</li></ul></div><div id="gmail-magicdomid3759" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><ul class="gmail-list-indent1" style="margin:0px 0px 0px 1.5em;padding:0px;list-style-type:none"><li style="margin:0px;padding:0px"><span class="gmail-" style="margin:0px;padding:1px 0px">                               As a reminder, the service receives data and by using grok filter [3] prepares information that later will be </span>pushed </li><li style="margin:0px;padding:0px">                               to Elasticsearch.</li></ul></div><div id="gmail-magicdomid2670" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><ul class="gmail-list-indent1" style="margin:0px 0px 0px 1.5em;padding:0px;list-style-type:none"><li style="margin:0px;padding:0px"><br style="margin:0px;padding:0px"></li></ul></div><div id="gmail-magicdomid3761" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">The new workflow would eliminate these bottlenecks by pushing data directly into Elasticsearch.</span></div><div id="gmail-magicdomid3762" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">This would therefore drop the following services:</span></div><div id="gmail-magicdomid3763" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">    - gearman client</span></div><div id="gmail-magicdomid3764" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">    - gearman worker</span></div><div id="gmail-magicdomid3765" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">    - logstash service</span></div><div id="gmail-magicdomid2936" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><br style="margin:0px;padding:0px"></div><div id="gmail-magicdomid3766" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">Before proceeding further, we need to know if these services, especially logstash, are (apart from sending logs from CI) not used anywhere, </span></div><div id="gmail-magicdomid3767" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">so that we don't accidentally pull the plug on any other workflows. If you are aware of these, please let me know.</span></div><div id="gmail-magicdomid3162" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><br style="margin:0px;padding:0px"></div><div id="gmail-magicdomid3768" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">Dan</span></div><div id="gmail-magicdomid2778" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><br style="margin:0px;padding:0px"></div><div id="gmail-magicdomid2778" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-size:12px;font-family:monospace"><span class="gmail-" style="font-family:Arial,sans-serif;margin:0px;padding:1px 0px">[1] </span><span class="gmail-url" style="font-family:Arial,sans-serif;margin:0px;padding:1px 0px"><a href="https://opendev.org/openstack/ci-log-processing" style="margin:0px;padding:0px;white-space:pre-wrap">https://opendev.org/openstack/ci-log-processing</a></span><br></div><div id="gmail-magicdomid3769" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">[2] </span><span class="gmail-url" style="margin:0px;padding:1px 0px"><a href="https://docs.opendev.org/opendev/system-config/latest/logstash.html" style="margin:0px;padding:0px;white-space:pre-wrap">https://docs.opendev.org/opendev/system-config/latest/logstash.html</a></span></div><div id="gmail-magicdomid3771" class="gmail-ace-line" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px"><span class="gmail-" style="margin:0px;padding:1px 0px">[3] </span><span class="gmail-url" style="margin:0px;padding:1px 0px"><a href="https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf" style="margin:0px;padding:0px;white-space:pre-wrap">https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf</a></span></div><table border="0"><tbody></tbody></table></div>