Open Stack

On Wed, Oct 3, 2018, at 3:16 AM, Whaley, Graham wrote:
> Hi Infra team.
> 
> First, a brief overview for folks who have not been in this loop.
> Kata Containers has a CI that runs some metrics, and spits out JSON 
> results. We'd like to store those results in the OSF logstash ELK infra 
> (http://logstash.openstack.org/#/dashboard/file/logstash.json), and set 
> up some Kibana views and dashboards so we can monitor historical trends 
> and project health (and longer term maybe some advanced trend regression 
> triggers).
> 
> There is a relevant github Issue/thread showing a working PoC here:
> https://github.com/kata-containers/ci/issues/60#issuecomment-426579084
> 
> I believe we are at the point where we should set up a trial index and 
> keys (for the filebeat/logstash/elastic injection) etc. so we can start 
> to tie the parts together.
> What do I need to make that next step happen? Do I need to open a formal 
> request ticket/Issue somewhere, or can we just thrash it out here?

Here and in code review is fine.

> 
> This email is part aimed at ClarkB :-), but obviously may involve others 
> etc. Input most welcome.

Currently all of our jobs submit jobs to the logstash processing system via the ansible role here [0]. That is then fed through our logstash configuration [1]. The first step in this is probably to update the logstash configuration and update the existing Kata jobs in zuul to submit jobs to index that information as appropriate?

As for Kibana views I'm not sure we ever sorted that out with the in browser version of kibana we are running. I think we can embed dashboard information in the kibana "source" or inject it into elasticsearch? This piece would take some investigation as I don't know what needs to be done off the top of my head. Note that our elasticsearch can't be written to via the public interface to it as there are no authentication and authorization controls for elasticsearch.

[0] https://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml
[1] https://git.openstack.org/cgit/openstack-infra/logstash-filters/tree/filters/openstack-filters.conf

Hope this helps,
Clark