<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>Celiometer Team,</div>
<div><br>
</div>
<div>I am a developer on the Project Meniscus team. I noticed the conversation on adding ElasticSearch and Kibana to Ceilometer and thought I would share some information regarding our project. I would love to discuss a way our projects could work together
on some of these common goals and possibly collaborate.</div>
<div><br>
</div>
<div>Project Meniscus is an open-source Python logging-as-a-service solution. The multi-tenant service will allow the dispatch of log messages to sinks such as ElasticSearch, Swift, and HDFS. Our initial implementation is defaulting to ElasticSearch. </div>
<div><br>
</div>
<div>The system was designed with the intention to scale and to be resilient to failure. We have written a tcp server for receiving syslog messages from standard syslog servers/daemons such as RSYSLOG and SYSLOG-NG. The server receives syslog messages over
long-lived tcp connections and parses individual log messages into json documents. The server uses the tornado tcp server, and the parser itself is written in C and uses Cython bindings.</div>
<div><br>
</div>
<div>We have implemented features such as normalization of log data by writing a python library that that binds to liglognorm, a C library for log processing.</div>
<div><br>
</div>
<div>In our very early alpha implementation we have been able to process about 30-40 GB of syslog messages per day on a single worker</div>
<div>node with a very small amount of load on the server. Our current worker nodes are 8GB RAM Virtual Machines running on Nova. </div>
<div><br>
</div>
<div><br>
</div>
<div>Currently we are working on:</div>
<div> 1. load balancing for syslog messages after parsing(since syslog servers transmit using long lived tcp connections)</div>
<div> 2. Implementing keystone authentication into Kibana 3 </div>
<div> 3. Building a proxy in front of ElasticSearch to limit queries by tenant.</div>
<div><br>
</div>
<div>Our project page is http://projectmeniscus.org/</div>
<div>Our repo is located at: https://github.com/ProjectMeniscus </div>
<div><br>
</div>
<div>The repo contains the main code base and all supporting projects, including our chef repository. </div>
<div><br>
</div>
<div>We would love to discuss a way our projects could work together on some of these common goals and possibly collaborate. Would it be possible to set up a time for us talk briefly?</div>
<div><br>
</div>
<div>Steven Gonzales</div>
<div>Software Developer</div>
<div>Rackspace Hosting</div>
<div>steven.gonzales@rackspace.com</div>
</div>
</body>
</html>