Open Stack

On Tue, Nov 5, 2019, at 8:56 AM, Matt Riedemann wrote:
> We used to be able to query for things like this:
> 
> message:"in reserve_block_device_name" AND message:"MessagingTimeout" 
> AND tags:"screen-n-api.txt"
> 
> to fingerprint a traceback in logstash like this [1] but that no longer 
> works. The multiline logstash filter is at [2] but doesn't seem to be 
> getting applied anymore.
> 
> I asked about this in -infra today and fungi said:
> 
> "(4:44:00 PM) fungi: mriedem: i suspect that coincided with switching 
> away from osla, we may need some means of parsing tracebacks out of logs 
> in the indexer"
> 
> I don't know what that means (what's osla? is [2] no longer used?) but 
> if someone could point me at some things to look at I could see if I can 
> generate a fix.

os-loganalyze, https://opendev.org/openstack/os-loganalyze, was in use on the old log server to do filtering of severity and related manipulation. One thing it would do is collapse lines that didn't have a timestamps or severity prefix. However I think that may have only been for the html rendering which logstash didn't use. I'm not sure this is the issue.

As for debugging this you can grab a log file and send it through logstash locally and fiddle with the rules until you get what you want. I'd help but currently at the summit and not in a good spot to do so.

> 
> [1] 
> https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4#31403
> [2] 
> https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
> 
> -- 
> 
> Thanks,
> 
> Matt
> 
>