<div>
If John Dickinson can steal me a 30 minute block at the conference I'll probably be giving a talk about it, but we (Rackspace) started switching to Graphite back in December. We're basically just following the etsy cookbook to "graph all the things!".
</div><div><br></div><div>We're using https://github.com/pandemicsyn/swift-informant to fire events to statsd. It takes care of answering questions like:</div><div><br></div><div>How many Object GET 200's are we currently getting per second.</div><div>How many container ops are we doing per second.</div><div>What was the average request time of container HEAD's between 4-5PM last tuesday (which always seems to lead to the question of why are they so much slower today…oh look that node is having a weird hw issue)?</div><div><br></div><div>Swift's also really good about dumping info to the error log. We convert the majority of those log lines to events thats get fired to statsd using https://github.com/pandemicsyn/statsdlog.</div><div><br></div><div>That lets us track everything from container-replicator timeouts, auth service retries, to OSError's on the object servers (think we're tracking about 25-30 log line patterns at the moment).</div><div><br></div><div>The last piece is just a hacked version of the swift-recon cli. It's what reports async-pending's, replication times, etc to graphite.</div><div><br></div><div>Right now it gets tied together by tiny hackish Flask app that generates some tv dashboard's and will probably start doing the monitoring/alerting for the traffic prediction/confidence bands (experimenting with just doing it with an irc bot).</div><div><br></div><div>-- <br>Florian Hines | @pandemicsyn<div>http://about.me/pandemicsyn<br><br></div></div>
<p style="color: #A0A0A8;">On Wednesday, February 22, 2012 at 2:50 AM, Jasper Capel wrote:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>I've uploaded the checks we use in production here at Spil Games to <a href="https://github.com/spilgames/swift">https://github.com/spilgames/swift</a>. Besides check_swift (which is a functional test) everything's meant to gather statistics from the cluster and we're looking to replace that with a Graphite-based solution to avoid having to parse access logs and having more real-time metrics available. Nothing fancy, but it may be of use to someone.</div><div><br></div><div>Jasper</div><div><br></div><div><br></div><div><br></div><div>On Feb 21, 2012, at 11:54 PM, Tim Bell wrote:</div><div><br></div><blockquote type="cite"><div><div><br></div><div>This does bring up a more generic problem of sharing the</div><div>availability/performance code for all of the OpenStack components.</div><div><br></div><div>At the design summit, this was proposed as one of the example use cases of</div><div>the OpenStack community forge (I forget the exact name) but it was intended</div><div>as a place for sharing code/procedures which were not intended to be part of</div><div>the core but may be of interest to others.</div><div><br></div><div>Was anything set up along these lines ?</div><div><br></div><div>A set of production quality Nagios/Ganglia sensors would be very interesting</div><div>if someone has these....</div><div><br></div><div>Tim</div><div><br></div><blockquote type="cite"><div><div>-----Original Message-----</div><div>From: openstack-bounces+tim.bell=<a href="mailto:cern.ch@lists.launchpad.net">cern.ch@lists.launchpad.net</a></div><div>[<a href="mailto:openstack-bounces">mailto:openstack-bounces</a>+tim.bell=<a href="mailto:cern.ch@lists.launchpad.net">cern.ch@lists.launchpad.net</a>] On Behalf</div><div>Of Jasper Capel</div><div>Sent: 21 February 2012 18:29</div><div>To: John Dickinson</div><div>Cc: <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a></div><div>Subject: Re: [Openstack] swprobe: swift middleware for sending metrics to</div><div>graphite using statsd</div><div><br></div><div>Hi John,</div><div><br></div><div>Apparently my google-fu is not up to snuff, as I wasn't aware of that</div></div></blockquote><div>project.</div><blockquote type="cite"><div><div>Had I been, I probably would've just extemded that one. :)</div><div><br></div><div>Cheers,</div><div>Jasper</div><div><br></div><div>________________________________________</div><div>From: John Dickinson [<a href="mailto:me@not.mn">me@not.mn</a>]</div><div>Sent: Tuesday, February 21, 2012 5:44 PM</div><div>To: Jasper Capel</div><div>Cc: <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a></div><div>Subject: Re: [Openstack] swprobe: swift middleware for sending metrics to</div><div>graphite using statsd</div><div><br></div><div>That's great. Have you by any chance seen</div><div><a href="https://github.com/pandemicsyn/swift-informant?">https://github.com/pandemicsyn/swift-informant?</a> It's something similar</div><div>that we've been playing with at Rackspace.</div><div><br></div><div>--John</div><div><br></div><div><br></div><div>On Feb 21, 2012, at 10:36 AM, Jasper Capel wrote:</div><div><br></div><blockquote type="cite"><div><div>Hi all,</div><div><br></div><div>I'm announcing a piece of Swift middleware, swprobe [1], designed to</div></div></blockquote><div>gather run-time metrics and ship them off to Graphite [2] for near</div></div></blockquote><div>real-time</div><blockquote type="cite"><div><div>monitoring. Currently it sends out bytes up- and downloaded per account,</div><div>http methods and response codes and timings in miliseconds on each call.</div><blockquote type="cite"><div><div><br></div><div>To be able to use this you need Graphite [2]. You also need statsd</div></div></blockquote></div></blockquote><div>running,</div><blockquote type="cite"><div><div>preferably on the local machine since there potentially many small UDP</div><div>packets are being sent out. Please also note that we have not yet tested</div></div></blockquote><div>this</div><blockquote type="cite"><div><div>with production workloads.</div><blockquote type="cite"><div><div><br></div><div>[1] - <a href="https://github.com/spilgames/swprobe">https://github.com/spilgames/swprobe</a></div><div>[2] - <a href="http://graphite.wikidot.com">http://graphite.wikidot.com</a>/</div><div>[3] - <a href="https://github.com/etsy/statsd">https://github.com/etsy/statsd</a></div><div><br></div><div>Best regards,</div><div><br></div><div>--</div><div>Jasper Capel</div><div>Lead Infrastructure Engineer</div><div><br></div><div>W <a href="http://www.spilgames.com">http://www.spilgames.com</a> | S jwcapel-spil</div><div><br></div><div><br></div><div><br></div><div>_______________________________________________</div><div>Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a></div><div>Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a></div><div>Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a></div><div>More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a></div></div></blockquote><div><br></div><div><br></div><div>_______________________________________________</div><div>Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a></div><div>Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a></div><div>Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a></div><div>More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a></div></div></blockquote></div></blockquote><div><br></div><div><br></div><div>_______________________________________________</div><div>Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a></div><div>Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a></div><div>Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a></div><div>More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>