Open Stack

Hey Steven!

Sorry for missing my chance to chat on IRC today. We have our next weekly meeting on October 3rd (next Thursday) at 15:00 UTC  (10:00 CST). That's probably the best chance at getting us all in a room, since we're pretty well distributed around the planet. I definitely agree with you that we're working on very similar goals and ought to communicate about what we're trying to accomplish and how we can help each other. Feel free to ping us on #openstack-metering; also you sit a few desks away from me, feel free to come by and chat. =]

Cheers!

-Thomas

On 9/24/13 2:10 PM, "Steven Gonzales" <steven.gonzales at RACKSPACE.COM<mailto:steven.gonzales at RACKSPACE.COM>> wrote:

Celiometer Team,

I am a developer on the Project Meniscus team.  I noticed the conversation on adding ElasticSearch and Kibana to Ceilometer and thought I would share some information regarding our project.  I would love to discuss a way our projects could work together on some of these common goals and possibly collaborate.

Project Meniscus is an open-source Python logging-as-a-service solution.  The multi-tenant service will allow the dispatch of log messages to sinks such as ElasticSearch, Swift, and HDFS.  Our initial implementation is defaulting to ElasticSearch.

The system was designed with the intention to scale and to be resilient to failure.  We have written a tcp server for receiving syslog messages from standard syslog servers/daemons such as RSYSLOG and SYSLOG-NG.  The server receives syslog messages over long-lived tcp connections and parses individual log messages into json documents.  The server uses the tornado tcp server, and the parser itself is written in C and uses Cython bindings.

We have implemented features such as normalization of log data by writing a python library that that binds to liglognorm, a C library for log processing.

In our very early alpha implementation we have been able to process about 30-40 GB of syslog messages per day on a single worker
node with a very small amount of load on the server.  Our current worker nodes are 8GB RAM Virtual Machines running on Nova.


Currently we are working on:
 1. load balancing for syslog messages after parsing(since syslog servers transmit using long lived tcp connections)
 2. Implementing keystone authentication into Kibana 3
 3. Building a proxy in front of ElasticSearch to limit queries by tenant.

Our project page is http://projectmeniscus.org/
Our repo is located at: https://github.com/ProjectMeniscus

The repo contains the main code base and all supporting projects, including our chef repository.

We would love to discuss a way our projects could work together on some of these common goals and possibly collaborate.  Would it be possible to set up a time for us talk briefly?

Steven Gonzales
Software Developer
Rackspace Hosting
steven.gonzales at rackspace.com<mailto:steven.gonzales at rackspace.com>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130925/afc27a79/attachment.html>