[openstack-dev] log-classify project update (anomaly detection in CI/CD logs)
tdecacqu at redhat.com
Tue Jul 3 07:39:58 UTC 2018
This is a follow-up to the initial project creation thread.
At the Vancouver Summit, we met to discuss ML for CI and I lead a workshop
on logreduce. The log-classify project bootstrap is still waiting
for review and I am still looking forward to pushing logreduce
source code in openstack-infra/log-classify.
The current implementation is working fine and I am going to enable it
for every job running on Software Factory. However the core
process HashingNeighbors is rather slow (0.3MB per second) and I
would like to improve it and/or implement other algorithms.
To do that effectively, we need to gather more datasets. I would like
to propose some enhancements to the os-loganalyze middleware to enable
users to annotate and report anomalies they find in log files.
To store the anoamlies reference, an additional webservice, or
perhaps direct access to an elasticsearch cluster would be required.
In parallel, we need to collect the users' feedback and create datasets daily
using the baseline available at the time each anomaly was discovered.
Ideally, we would create an ipfs (or any other network filesystem) that
could then be used by anyone willing to work on $subject.
There is a lot to do and it will be challening. To that effect, I would
like to propose an initial meeting with all interested parties.
Please register your irc name and timezone in this etherpad:
Due to OpenStack's exceptional infrastructure and recent Zuul v3 release,
I think we are in a strong position to tackle this challenge.
Others suggestions to bootstrap this effort within our community are welcome.
 git clone https://softwarefactory-project.io/r/logreduce
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 488 bytes
Desc: not available
More information about the OpenStack-dev