Open Stack

On Mon, May 10, 2021, at 10:34 AM, Clark Boylan wrote:
> Hello everyone,
> 
> Xenial has recently reached the end of its life. Our 
> logstash+kibana+elasticsearch and subunit2sql+health data crunching 
> services all run on Xenial. Even without the distro platform EOL 
> concerns these services are growing old and haven't received the care 
> they need to keep running reliably.
> 
> Additionally these services represent a large portion of our resource 
> consumption:
> 
> * 6 x 16 vcpu + 60GB RAM + 1TB disk Elasticsearch servers
> * 20 x 4 vcpu + 4GB RAM logstash-worker servers
> * 1 x 2 vcpu + 2GB RAM logstash/kibana central server
> * 2 x 8 vcpu + 8GB RAM subunit-worker servers
> * 64GB RAM + 500GB disk subunit2sql trove db server
> * 1 x 4 vcpu + 4GB RAM health server
> 
> To put things in perspective, they account for more than a quarter of 
> our control plane servers, occupying over a third of our block storage 
> and in excess of half the total memory footprint.
> 
> The OpenDev/OpenStack Infra team(s) don't seem to have the time 
> available currently to do the major lifting required to bring these 
> services up to date. I would like to propose that we simply turn them 
> off. All of these services operate off of public data that will not be 
> going away (specifically job log content). If others are interested in 
> taking this on they can hook into this data and run their own 
> processing pipelines.
> 
> I am sure not everyone will be happy with this proposal. I get it. I 
> came up with the idea for the elasticsearch job log processing way back 
> at the San Diego summit. I spent many many many hours since working to 
> get it up and running and to keep it running. But pragmatism means that 
> my efforts and the team's efforts are better spent elsewhere.
> 
> I am happy to hear feedback on this. Thank you for your time.

Since this thread was started we have heard feedback and the OpenStack TC has brought this up with the Board to try and find volunteers to help address the hosting, upgrades, and maintenance of these services. We have said we are not in a rush to shut them off (still no rush), but feel that setting a deadline for finding help is important.

At the TC meeting yesterday we decided that we would try to limp the server along through the Yoga cycle. Rough math says that is going to end April 2022. Getting this addressed sooner is definitely better, as there is always the risk that external events will force us to shut these services down prior to that date. Hopefully, having a concrete date can create some urgency and help us find the aid we need.

If you would like to help definitely read through this thread as it provides details on what sorts of things need doing. Also feel free to reach out to myself or others on the OpenDev team and we'll do our best to provide direction as necessary.

Clark