<html><body><div dir="ltr">I just came back from vacation and reading this does worry me a lot. The ES server is a crucial piece used to identify zuul job failures on openstack. LOTS of them have causes external to the job itself, either infra, packaging (os or pip) or unavailability of some services used during build.<div><br></div><div dir="ltr">Without being able to query specific error messages across multiple jobs we will be in a vary bad spot as we loose the ability to look outside a single project.</div><div dir="ltr"><br></div><div>TripleO health check project relies on being able to query ER from both opendev and rdo in order to easy identification of problems.<br></div><div dir="ltr"><br></div><div dir="ltr">Maybe instead of dropping we should rethink what it is supposed to index and not, set some hard limits per job and scale down the deployment. IMHO, one of the major issues with it is that it does try to index maybe too much w/o filtering noisy output before indexing.</div><div dir="ltr"><br></div><div dir="ltr">If we can delay making a decision a little bit so we can investigate all available options it would really be great.</div><div><br></div><div dir="ltr">I worth noting that I personally do not have a special love for ES but I do value a lot what it does. I am also pragmatic and I would not be very upset to make use of a SaaS service as an alternative, especially as I recognize how costly is to run and maintain an instance.</div><div dir="ltr"><br></div><div dir="ltr">Maybe we can find a SaaS log processing vendor willing to sponsor OpenStack? In the past I used DataDog for monitoring but they also offer log processing and they have a <a href="https://www.datadoghq.com/partner/open-source/" style="">program for open-source</a> but I am not sure they would be willing to process that amount of data for us.</div><div><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Cheers</div><div dir="ltr">Sorin Sbarnea</div><div dir="ltr">Red Hat</div></div></div><div dir="ltr">
<br><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr" style="">On 10 May 2021 at 18:34:40, Clark Boylan <<a href="mailto:cboylan@sapwetik.org">cboylan@sapwetik.org</a>> wrote:<br></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<div>
Hello everyone,<br><br>Xenial has recently reached the end of its life. Our logstash+kibana+elasticsearch and subunit2sql+health data crunching services all run on Xenial. Even without the distro platform EOL concerns these services are growing old and haven't received the care they need to keep running reliably.<br><br>Additionally these services represent a large portion of our resource consumption:<br><br>* 6 x 16 vcpu + 60GB RAM + 1TB disk Elasticsearch servers<br>* 20 x 4 vcpu + 4GB RAM logstash-worker servers<br>* 1 x 2 vcpu + 2GB RAM logstash/kibana central server<br>* 2 x 8 vcpu + 8GB RAM subunit-worker servers<br>* 64GB RAM + 500GB disk subunit2sql trove db server<br>* 1 x 4 vcpu + 4GB RAM health server<br><br>To put things in perspective, they account for more than a quarter of our control plane servers, occupying over a third of our block storage and in excess of half the total memory footprint.<br><br>The OpenDev/OpenStack Infra team(s) don't seem to have the time available currently to do the major lifting required to bring these services up to date. I would like to propose that we simply turn them off. All of these services operate off of public data that will not be going away (specifically job log content). If others are interested in taking this on they can hook into this data and run their own processing pipelines.<br><br>I am sure not everyone will be happy with this proposal. I get it. I came up with the idea for the elasticsearch job log processing way back at the San Diego summit. I spent many many many hours since working to get it up and running and to keep it running. But pragmatism means that my efforts and the team's efforts are better spent elsewhere.<br><br>I am happy to hear feedback on this. Thank you for your time.<br><br>Clark<br><br><br>
</div>
</div>
</blockquote>
</div>
</div>
</div></div></body></html>