<HTML>
<HEAD>
<TITLE>Re: [Openstack] Monitoring / Billing Architecture proposed</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>I like <a href="http://wiki.openstack.org/SystemUsageData">http://wiki.openstack.org/SystemUsageData</a><BR>
<BR>
I just wonder if there should be offical formats for the data sent across the wire.<BR>
<BR>
Or at least high level formats that define the data, maybe in a simple python “map” style layout.<BR>
<BR>
With examples would be awesome.<BR>
<BR>
On 4/23/12 10:40 AM, "Luis Gervaso" <<a href="luis@woorea.es">luis@woorea.es</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>I have been looking at : <a href="http://wiki.openstack.org/SystemUsageData">http://wiki.openstack.org/SystemUsageData</a><BR>
<BR>
On Mon, Apr 23, 2012 at 7:35 PM, Brian Schott <<a href="brian.schott@nimbisservices.com">brian.schott@nimbisservices.com</a>> wrote:<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Is there a document somewhere on what events the services emit? <BR>
<BR>
</SPAN></FONT><FONT SIZE="1"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:9pt'>-------------------------------------------------<BR>
Brian Schott, CTO<BR>
Nimbis Services, Inc.<BR>
<a href="brian.schott@nimbisservices.com">brian.schott@nimbisservices.com</a><BR>
ph: 443-274-6064 <tel:443-274-6064> fx: 443-274-6060 <tel:443-274-6060> <BR>
</SPAN></FONT></FONT><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:12pt'><BR>
</SPAN></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
On Apr 23, 2012, at 12:39 PM, Monsyne Dragon wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>This already exists in trunk. The Notification system was designed specifically to feed billing and monitoring systems. <BR>
<BR>
Basically, we don't want Nova/Glance/etc to be in the business of trying to determine billing logic, since it is different for pretty much everyone, so we just emit notifications to a queue and the interested pull what they want, and aggregate according to their own rules. <BR>
<BR>
On Apr 22, 2012, at 1:50 PM, Luis Gervaso wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hi, <BR>
<BR>
I want to share the architecture i am developing in order to perform the monitorig / billing OpenStack support:<BR>
<BR>
1. AMQP Client which listen to RabbitMQ / QPid (this should be interchangeable) (Own Stuff or ServiceMix / Camel)<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'> <BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>2. Events should be stored on a NoSQL document oriented database (I think mongodb is perfect, since we can query in a super easy fashion)<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
We have an existing system called Yagi (<a href="https://github.com/Cerberus98/yagi/">https://github.com/Cerberus98/yagi/</a>) that listens to the notification queues and persists events to a Redis database. It then provides feeds as ATOM formatted documents that a billing system can pull to aggregate data, It also can support PubSub notification of clients thru the pubsubhubub protocol, and push events to a long-term archiving store thru the AtomPub protocol. <BR>
<BR>
That said, the notification system outputs its events as JSON, so it should be easy to pipe into a json document-oriented db if that's what you need. (we only use ATOM because we have a atom-based archiving/search/aggregation engine (it's open source: <a href="http://atomhopper.org/">http://atomhopper.org/</a> ) our in-house systems already plug into. )<BR>
<BR>
<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
3a. The monitoring system can pull/push MongoDB<BR>
<BR>
3b. The billing system can pull to create invoices <BR>
<BR>
4. A mediation EIP should be necessary to integrate a billing/monitoring product. (ServiceMix / Camel)<BR>
<BR>
This is to receive your feedback. So please, critics are welcome!<BR>
<BR>
Cheers!<BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE>
</BODY>
</HTML>