Open Stack

Celiometer Team,

I am a developer on the Project Meniscus team.  I noticed the conversation on adding ElasticSearch and Kibana to Ceilometer and thought I would share some information regarding our project.  I would love to discuss a way our projects could work together on some of these common goals and possibly collaborate.

Project Meniscus is an open-source Python logging-as-a-service solution.  The multi-tenant service will allow the dispatch of log messages to sinks such as ElasticSearch, Swift, and HDFS.  Our initial implementation is defaulting to ElasticSearch.

The system was designed with the intention to scale and to be resilient to failure.  We have written a tcp server for receiving syslog messages from standard syslog servers/daemons such as RSYSLOG and SYSLOG-NG.  The server receives syslog messages over long-lived tcp connections and parses individual log messages into json documents.  The server uses the tornado tcp server, and the parser itself is written in C and uses Cython bindings.

We have implemented features such as normalization of log data by writing a python library that that binds to liglognorm, a C library for log processing.

In our very early alpha implementation we have been able to process about 30-40 GB of syslog messages per day on a single worker
node with a very small amount of load on the server.  Our current worker nodes are 8GB RAM Virtual Machines running on Nova.


Currently we are working on:
 1. load balancing for syslog messages after parsing(since syslog servers transmit using long lived tcp connections)
 2. Implementing keystone authentication into Kibana 3
 3. Building a proxy in front of ElasticSearch to limit queries by tenant.

Our project page is http://projectmeniscus.org/
Our repo is located at: https://github.com/ProjectMeniscus

The repo contains the main code base and all supporting projects, including our chef repository.

We would love to discuss a way our projects could work together on some of these common goals and possibly collaborate.  Would it be possible to set up a time for us talk briefly?

Steven Gonzales
Software Developer
Rackspace Hosting
steven.gonzales at rackspace.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130924/e3a4f000/attachment.html>