On Tue, Nov 5, 2019, at 8:56 AM, Matt Riedemann wrote:

We used to be able to query for things like this:

message:"in reserve_block_device_name" AND message:"MessagingTimeout" AND tags:"screen-n-api.txt"

to fingerprint a traceback in logstash like this [1] but that no longer works. The multiline logstash filter is at [2] but doesn't seem to be getting applied anymore.

I asked about this in -infra today and fungi said:

"(4:44:00 PM) fungi: mriedem: i suspect that coincided with switching away from osla, we may need some means of parsing tracebacks out of logs in the indexer"

I don't know what that means (what's osla? is [2] no longer used?) but if someone could point me at some things to look at I could see if I can generate a fix.

os-loganalyze, https://opendev.org/openstack/os-loganalyze, was in use on the old log server to do filtering of severity and related manipulation. One thing it would do is collapse lines that didn't have a timestamps or severity prefix. However I think that may have only been for the html rendering which logstash didn't use. I'm not sure this is the issue. As for debugging this you can grab a log file and send it through logstash locally and fiddle with the rules until you get what you want. I'd help but currently at the summit and not in a good spot to do so.

[1] https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/... [2] https://opendev.org/openstack/logstash-filters/src/branch/master/filters/ope...

--

Thanks,

Matt