<div dir="ltr"><div><div>so my understanding is that the billing part of ceilometer has to do with financial requirements in how and what is reported in metrics. ie... you cannot bill for rounded values or heuristics. but stacktach has no such specific need to solve for financial needs and really is intended to help an operator isolate or identify issues / potential issues.<br><br></div>that's a pretty big difference in use case. <br><br></div>-mayy<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 12, 2015 at 11:30 AM, Kris G. Lindgren <span dir="ltr"><<a href="mailto:klindgren@godaddy.com" target="_blank">klindgren@godaddy.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Event-based Monitoring & Billing solution for OpenStack<br>
<br>
Unsure what its checking out for billing though.<br>
____________________________________________<br>
<br>
Kris Lindgren<br>
Senior Linux Systems Engineer<br>
GoDaddy, LLC.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On 2/12/15, 9:17 AM, "Matt Joyce" <<a href="mailto:matt@nycresistor.com">matt@nycresistor.com</a>> wrote:<br>
<br>
>I thought stacktach was more in the vein of diagnostic. Not billable<br>
>resources.<br>
><br>
>On Feb 12, 2015 10:47 AM, Tim Bell <<a href="mailto:Tim.Bell@cern.ch">Tim.Bell@cern.ch</a>> wrote:<br>
>><br>
>> Does anyone have any proposals regarding<br>
>><br>
>> > - Possible replacements for Ceilometer that you have used instead<br>
>><br>
>> It seems that many sites have written their own systems. The<br>
>>stacktach/monasca teams are due to demo to the operators meetup in<br>
>>Philadelphia in March.<br>
>><br>
>> Does anyone have experience to share comparing ceilometer with<br>
>>stacktach ?<br>
>><br>
>> Tim<br>
>><br>
>> > -----Original Message-----<br>
>> > From: Daniele Venzano [mailto:<a href="mailto:daniele.venzano@eurecom.fr">daniele.venzano@eurecom.fr</a>]<br>
>> > Sent: 12 February 2015 12:24<br>
>> > To: <a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
>> > Subject: Re: [Openstack-operators] [Ceilometer] Real world experience<br>
>>with<br>
>> > Ceilometer deployments - Feedback requested<br>
>> ><br>
>> > Unfortunately, I can only confirm the sorry state of Ceilometer.<br>
>> > We tried it on a very small setup (6 compute nodes) and run in so<br>
>>many issues,<br>
>> > we dropped it and created our own solution based on a mix of scripts<br>
>>that read<br>
>> > from the nova/neutron DB, iptables and collectd data. No need for<br>
>>more<br>
>> > collection agents than what we are already running for the systems<br>
>>monitoring.<br>
>> ><br>
>> > We tried the version in Havana and, later, in Icehouse. For starters<br>
>>the<br>
>> > documentation was suggesting MySQL as default backend. MySQL will<br>
>>last just a<br>
>> > few days and then break down under the size of the tables. We tried<br>
>>MongoDB,<br>
>> > but were still not satisfied with performance on such a small<br>
>>cluster.<br>
>> > Then there is the metering agent. It is yet another daemon, not<br>
>>integrated in<br>
>> > Neutron and there is no documentation about what it is actually<br>
>>measuring.<br>
>> > What if I have multiple routers? Ingress and Egress? From which point<br>
>>of view?<br>
>> > The same applies to Cinder, it requires and external agent (to be run<br>
>>via cron!).<br>
>> ><br>
>> > Some metrics were not recorded, we couldn't understand why and,<br>
>>again, no<br>
>> > documentation and no tooling to help us understand whether we were<br>
>>just<br>
>> > missing some config options somewhere in nova-compute or there was<br>
>>some<br>
>> > other problem with KVM/libvirt versions.<br>
>> > And even when we had some data and wanted to generate just a<br>
>>proof-of-<br>
>> > concept report with some information about tenant resource usage, we<br>
>>found<br>
>> > problems with the API. The fact that no one had bothered to write a<br>
>>simple<br>
>> > proof of concept script that uses the API to actually do something<br>
>>useful was<br>
>> > really off-putting.<br>
>> ><br>
>> > We had to dig in libvirt to understand what some of the metrics<br>
>>actually mean.<br>
>> > We found that we could read those same metrics from our (more<br>
>>efficient, well-<br>
>> > known) monitoring system.<br>
>> ><br>
>> > For some time we run just the agents and aggregated the data in an<br>
>> > elasticsearch instance through the UDP msgpack pipeline (more bugs,<br>
>>message<br>
>> > format is inconsistent, different agents generate different fields,<br>
>>in slightly<br>
>> > different formats).<br>
>> > It works. But for our needs it was just too much work. Most of the<br>
>>data is<br>
>> > already available from other sources with well-known APIs.<br>
>> ><br>
>> > Ah, also there is a long standing bug open: Sahara and Ceilometer<br>
>>cannot be<br>
>> > used together. And we use Sahara.<br>
>> ><br>
>> > I opened bugs for some of these issues, but since then I lost<br>
>>interest.<br>
>> ><br>
>> > In the end, I think it really depends on what kind of data you need<br>
>>and what<br>
>> > (developer) resources you can throw at the problem.<br>
>> > Unless in Juno things changed dramatically, Ceilometer will not work<br>
>>out of the<br>
>> > box. You will have to lose time because of the non-existent<br>
>>documentation, you<br>
>> > will have to develop code and scripts anyway and finally you will<br>
>>have to create<br>
>> > something between your billing system and the ceilometer API, because<br>
>>to the<br>
>> > best of my knowledge there is nothing that uses it.<br>
>> ><br>
>> > eBay has the resources to do all that. We don't.<br>
>> ><br>
>> ><br>
>> ><br>
>> > -----Original Message-----<br>
>> > From: George Shuklin [mailto:<a href="mailto:george.shuklin@gmail.com">george.shuklin@gmail.com</a>]<br>
>> > Sent: Thursday 12 February 2015 02:59<br>
>> > To: <a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
>> > Subject: Re: [Openstack-operators] [Ceilometer] Real world experience<br>
>>with<br>
>> > Ceilometer deployments - Feedback requested<br>
>> ><br>
>> > Ceilometer is in sad state.<br>
>> ><br>
>> > 1. Collector leaks memory. We ran it on same host with mongo, and it<br>
>>grab<br>
>> > 29Gb out of 32, leaving mongo with less than gig memory available.<br>
>> > 2. Metering agent cause huge load on neutron-server. o(n) of metering<br>
>>rules and<br>
>> > tenants. Few bugs reported, one bugfix in review.<br>
>> > 3. Metering agent simply do no work on multi-network-nodes<br>
>>installation.<br>
>> > It exepects all routers be on same host. Fixed or not - I don't know,<br>
>>we have our<br>
>> > own crude fix.<br>
>> > 4. Many rough edges. Ceilometer much less tested than nova. Sometimes<br>
>>it<br>
>> > traces and skip counting. Fresh example: if metadata has '.' in the<br>
>>name,<br>
>> > ceilometer trace on it and did not count in glance usage.<br>
>> > 5. Very slow on reports (using mongo's mapreduce).<br>
>> ><br>
>> > Overall feeling: barely usable, but with my experience with cloud<br>
>>billings, not the<br>
>> > worst thing I saw in my life.<br>
>> ><br>
>> > About load: except reporting and memory leaks, it use rather small<br>
>>amount of<br>
>> > resources.<br>
>> ><br>
>> > On 02/11/2015 09:37 PM, Maish Saidel-Keesing wrote:<br>
>> > > Is Ceilometer ready for prime time?<br>
>> > ><br>
>> > > I would be interested in hearing from people who have deployed<br>
>> > > OpenStack clouds with Ceilometer, and their experience. Some of the<br>
>> > > topics I am looking for feedback on are:<br>
>> > ><br>
>> > > - Database Size<br>
>> > > - MongoDB management, Sharding, replica sets etc.<br>
>> > > - Replication strategies<br>
>> > > - Database backup/restore<br>
>> > > - Overall useability<br>
>> > > - Gripes, pains and problems (things to look out for)<br>
>> > > - Possible replacements for Ceilometer that you have used instead<br>
>> > ><br>
>> > ><br>
>> > > If you are willing to share - I am sure it will be beneficial to<br>
>>the<br>
>> > > whole community.<br>
>> > ><br>
>> > > Thanks in Advance<br>
>> > ><br>
>> > ><br>
>> > > With best regards,<br>
>> > ><br>
>> > ><br>
>> > > Maish Saidel-Keesing<br>
>> > > Platform Architect<br>
>> > > Cisco<br>
>> > ><br>
>> > ><br>
>> > ><br>
>> > ><br>
>> > > _______________________________________________<br>
>> > > OpenStack-operators mailing list<br>
>> > > <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
>> > ><br>
>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator</a><br>
>> > > s<br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > OpenStack-operators mailing list<br>
>> > <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
>> ><br>
>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > OpenStack-operators mailing list<br>
>> > <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
>> ><br>
>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-operators mailing list<br>
>> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>_______________________________________________<br>
>OpenStack-operators mailing list<br>
><a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
</div></div></blockquote></div><br></div>