<br><br><div class="gmail_quote">On Thu, May 17, 2012 at 5:47 AM, Nick Barcet <span dir="ltr"><<a href="mailto:nick.barcet@canonical.com" target="_blank">nick.barcet@canonical.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">On 05/17/2012 11:13 AM, Loic Dachary wrote:<br>
> On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:<br>
>><br>
>> I'm now of the opinion that we exclude storage and API from the<br>
>> metering project scope. Let's just focus on defining a metering<br>
>> message format, bus, and maybe a client-library to make it easy to<br>
>> write metering consumers.<br></div></div></blockquote><div><br></div><div>The plan, as I understand it, is to ensure that all metering messages appear on a common bus using a documented format. Deployers who do not want the storage system and REST API will not need to use it, and can set up their own clients to listen on that bus. I'm not sure how much of a client library is needed, since the bus is AMQP and the messages are JSON, both of which have standard libraries in most common languages.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
>><br>
>> That way we avoid building a lot of things that we only _think will<br>
>> be useful_ for potential billing integration. Only<br>
>> writing/delivering such an integration component would prove that<br>
>> we built at least something that is useful.<br>
>><br>
>><br>
><br>
> Hi,<br>
><br>
> I'm a little reluctant to question the whole approach because I'm<br>
> engaged in it :-) It's scary to contemplate the idea of throwing away<br>
> some of the work done but I welcome the challenge. Better lose a few<br>
> days work than keep going in the wrong direction.<br>
><br>
> Are you proposing that such a library would then be integrated in<br>
> whatever billing system someone has in mind ? For instance<br>
><br>
> Dough <a href="https://github.com/lzyeval/dough" target="_blank">https://github.com/lzyeval/dough</a> <a href="http://trystack.org" target="_blank">trystack.org</a> billing<br>
> <a href="https://github.com/trystack/dash_billing" target="_blank">https://github.com/trystack/dash_billing</a> nova-billing<br>
> <a href="https://github.com/griddynamics/nova-billing" target="_blank">https://github.com/griddynamics/nova-billing</a><br>
><br>
> Would depend on this library and rely on its API to collect what they<br>
> need. The same would be done by proprietary billing systems ?<br>
><br>
> Regarding the relevance of the metrics exposed by the API, I made<br>
> sure they match the need of the eNovance sales. I'm quite sure<br>
> Nicolas checked for Canonical and Doug did the same regarding the<br>
> needs of Dreamhost. I'm confident that the information is actualy<br>
> useful, at least in these practical cases.<br></div></div></blockquote><div><br></div><div>Yes, all of the feedback I have provided has been directly related to needs we have here at DreamHost. I expect us to use the API service to pull data into our legacy billing system, which does not have an API of its own. </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
><br>
> Getting rid of the storage imposes a constraint on the billing system<br>
> : it must make 100% sure that once a message is consumed it will be<br>
> reliably archived. It also adds a constraint on the chosen bus : it<br>
> must be able to retain all messages for as long as a consumer needs,<br>
> which may be days or weeks. Or it adds a constraint on the billing<br>
> system which must make 100% sure it will consume all relevant<br>
> messages the bus at all times before they expire.<br></div></div></blockquote><div><br></div><div>That's exactly right. It will be easier for me to bridge between our two systems by pulling a day's worth of details from the ceilometer API and storing them in the billing system using a batch job, rather than trying to ensure that the billing database performs well enough to record the information in real time. My goal is to not have to change the billing system at all.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
><br>
> I have no strong feelings about exposing a bus for everyone to use<br>
> instead of a REST API. I tend to prefer the REST API because it is an<br>
> established standard for OpenStack. Could you expand on why a bus<br>
> would be significantly better than a REST API in this specific case<br>
> ?<br>
<br>
</div></div>Thanks for challenging our thought process, Francis, it is a great<br>
sanity check :)<br>
<br>
I do have a few use cases where a REST API is better than a bus:<br>
<br>
* Private clouds: Users are unlikely to want to activate a complete<br>
billing system but still want to be able top provide to their users some<br>
metrics of their uses. The REST API plus some scripts would allow them<br>
to do this without too much pain.<br>
<br>
* Metric of usage on GUI: It might be usefull to provide a quick way to<br>
assess usage in almost real time to users as part of an extension of<br>
Horizon for example. The REST API would allow for such data to be<br>
extracted dynamically without having to run a full billing tool in real<br>
time.<br></blockquote><div><br></div><div>I don't know about real time, but I do know we eventually plan to provide usage information like this to our users. So, change "might be useful" to "will be useful." :-)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
* In house billing tools: about half of the ISPs I surveyed are running<br>
some form of a customized ERP system to handle the billing for their<br>
customers. They need to be able to produce CSVs on a weekly basis to<br>
feed their custom billing solutions. Integrating a bus would be much<br>
more complex than the script that would issue request to the rest API.<br></blockquote><div><br></div><div>Right. A lot of accounting systems run in a batch model like this.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
* Auditability/Non repudiability: if the messages go from the queue to<br>
some unknown DB, how do you solve, in a generic way, coherent audit<br>
check and ensure non-repudiability?<br>
<br>
I do not mean to totally discard Francis' bus idea though, and think<br>
that we should allow for (a not necessarily db-less integration but)<br>
direct queue attachment model for billing systems. I do think that for<br>
all the above reasons, plus the simplification of testing of the overall<br>
system, the REST API must remain part of this project.<br></blockquote><div><br></div><div>As I mentioned above, we will have the bus available. I expect to have several routing keys, much as is done with notifications. For example, we should have an easy way to receive all messages ("metering.all") as well as a way to select only messages for counters the consumer is interested in ("metering.c1", "metering.n2", etc.). That should allow someone to replace the ceilometer collector daemon with their own client and storage backend, or to simply watch the messages for monitoring purposes.</div>
<div><br></div><div>Doug</div><div><br></div></div>