Open Stack

On November 21, 2017 5:48 pm, Clark Boylan wrote:
> On Tue, Nov 21, 2017, at 09:17 AM, Tristan Cacqueray wrote:
>>
> 
> snip
> 
>> Actually the rfc is this thread :-)
>> 
>> Though I forgot to mention the first steps that could use comments before
>> we move on:
>> * create the openstack-infra/log-classify project,
>> * import the log-classify.crm script,
>> * wrap the script with a more user friendly interface, and
>> * modify the puppet-log_processor to use that new project instead
> 
> This sounds like a great place to start. Considering the interest
> already forming around this I would say go ahead and create the project
> and start with the import process so that people have a concrete place
> to start working on this. I am sure it will evolve from there, but
> getting started is often the most difficult step.
> 
> Related to the last step we have temporarily disabled CRM classification
> in the log processor pipeline because we treat the whole file path as a
> unique file to classify which ended up filling our workers' disks with
> classification files. I think one of the things we will want to address
> early on is using the basename rather than the whole path to
> significantly reduce the total number of data files on disk. This way we
> can get it running in the log processor pipeline again for proper
> production feedback of changes that are happening.
> 
> Once again let me know if I can help with anything (happy to review new
> project creation changes for example).
> 

Excellent, project creation is proposed here:
  https://review.openstack.org/#/q/topic:log-classify

I'm open to suggestion regarding the name and structure of the project.
Otherwise I'll create a standard openstack python project with:

logclassify.logstash module to interface with the script using the
design of the log-gearman-client.py (e.g. a process(event)).
logclassify.cmd module to use the script standalone.

And then write a first test and implementation of that basename base data
files improvement.

If that works ok, then a follow-up change will modify the
log-gearman-client to import logclassify instead of running the script
directly.

> Thank you for getting this started,
> Clark
> 

Thanks for the quick feedback!
-Tristan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20171122/ce0114ce/attachment.sig>