<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I’ve lamented for awhile that while swift/statsd provide a wealth of information, it’s in a somewhat difficult to use format. Specifically you have to connect to a socket and listen for messages. Furthermore if you’re listening, nobody
else can. I do realize there is a mechanism to send the data to graphite, but what if I’m not a graphite user OR want to look at the data at a finer granularity than is being sent to graphite?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What I’ve put together and would love to get some feedback on is a tool I’m calling ‘statsdtee’, specifically because you can configure statsd to send to the port it wants to listen on (configurable of course) and statsdtee will then process
it locally AND tee it out another socket, making it possible to forward the data on to graphite and still allow local processing.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Local processing consists of calculating rolling counters and writing them to a file that looks much like most /proc entries, such as this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><i>$cat /tmp/statsdtee<o:p></o:p></i></p>
<p class="MsoNormal"><i>V1.0 1403633349.159516<o:p></o:p></i></p>
<p class="MsoNormal"><i>accaudt 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>accreap 0 0 0 0 0 0 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>accrepl 0 0 2100 0 0 0 1391 682 0 2100<o:p></o:p></i></p>
<p class="MsoNormal"><i>accsrvr 1 0 0 0 0 2072 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>conaudt 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>conrepl 0 0 2892 0 0 0 1997 1107 0 2892<o:p></o:p></i></p>
<p class="MsoNormal"><i>consrvr 2700 0 0 1 1 992 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>consync 541036 0 11 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>conupdt 0 17 17889<o:p></o:p></i></p>
<p class="MsoNormal"><i>objaudt 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>objexpr 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>objrepl 0 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i>objsrvr 117190 16325 0 43068 9 996 5 0 6904<o:p></o:p></i></p>
<p class="MsoNormal"><i>objupdt 0 0 0 1704 0<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal">In this format we’re looking at data for account, container and object services. There is a similar one for proxy. The reason for the names on each line is what to report on is configurable in a conf file down to the granularity of a
single line, thereby making it possible to report less information, though I’m not sure if one would really do that or not.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To make this mechanism really simple and avoid using internal timers, I’m simply looking at the time of each record and every time the value of the second changes, write out the current counters. I could change it to every 10<sup>th</sup>
of second but am thinking that really isn’t necessary. I could also drive it off a timer interrupt, but again I’m not sure that would really buy you anything.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My peeve with /proc is you never know what each field means and so there is a second format in which headers are included and they look like this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><i>$ cat /tmp/statsdtee<o:p></o:p></i></p>
<p class="MsoNormal"><i>V1.0 1403633339.410722<o:p></o:p></i></p>
<p class="MsoNormal"><i># errs pass fail<o:p></o:p></i></p>
<p class="MsoNormal"><i>accaudt 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># errs cfail cdel cremain cposs_remain ofail odel oremain oposs_remain<o:p></o:p></i></p>
<p class="MsoNormal"><i>accreap 0 0 0 0 0 0 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># diff diff_cap nochg hasmat rsync rem_merge attmpt fail remov succ<o:p></o:p></i></p>
<p class="MsoNormal"><i>accrepl 0 0 2100 0 0 0 1391 682 0 2100<o:p></o:p></i></p>
<p class="MsoNormal"><i># put get post del head repl errs<o:p></o:p></i></p>
<p class="MsoNormal"><i>accsrvr 1 0 0 0 0 2069 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># errs pass fail<o:p></o:p></i></p>
<p class="MsoNormal"><i>conaudt 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># diff diff_cap nochg hasmat rsync rem_merge attmpt fail remov succ<o:p></o:p></i></p>
<p class="MsoNormal"><i>conrepl 0 0 2793 0 0 0 1934 1083 0 2793<o:p></o:p></i></p>
<p class="MsoNormal"><i># put get post del head repl errs<o:p></o:p></i></p>
<p class="MsoNormal"><i>consrvr 2700 0 0 1 1 976 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># skip fail sync del put<o:p></o:p></i></p>
<p class="MsoNormal"><i>consync 536193 0 11 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># succ fail no_chg<o:p></o:p></i></p>
<p class="MsoNormal"><i>conupdt 0 17 17889<o:p></o:p></i></p>
<p class="MsoNormal"><i># quar errs<o:p></o:p></i></p>
<p class="MsoNormal"><i>objaudt 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># obj errs<o:p></o:p></i></p>
<p class="MsoNormal"><i>objexpr 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># part_del part_upd suff_hashes suff_sync<o:p></o:p></i></p>
<p class="MsoNormal"><i>objrepl 0 0 0 0<o:p></o:p></i></p>
<p class="MsoNormal"><i># put get post del head repl errs quar async_pend<o:p></o:p></i></p>
<p class="MsoNormal"><i>objsrvr 117190 16325 0 43068 9 996 5 0 6904<o:p></o:p></i></p>
<p class="MsoNormal"><i># errs quar succ fail unlk<o:p></o:p></i></p>
<p class="MsoNormal"><i>objupdt 0 0 0 1704 0<o:p></o:p></i></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The important thing to remember about rolling counters is as many people who wish can read them simultaneously and be assured nobody is stepping on each other since they never get zeroed! You simply read a sample, wait awhile and read
another. The result is the change in the counters over that interval and anyone can use any interval they choose.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So how useful people think this is? Personally I think it’s very useful…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The next step is how to calculate the numbers I’m reporting. While statsd reports a lot of timing information, none of that really fits this model as all I want are counts. So when I see a GET timing record, I count it as 1 GET. Seems
to work so far. IS this a legitimate thing to be doing? Feels right and from the preliminary testing I’ve been doing it seems pretty accurate.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One thing I’ve found missing is more detailed error information. For example I can tell how many errors there were but I can’t tell how many of each type there were. Is this something that can easily be added? I’ve found in our environment
it can be useful when there’s an increase in the number of errors on a particular server, knowing the type can be quite useful.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">While I’m not currently counting everything, such as device specific data which would significantly increase the volume of output, I think I have covered quite a lot in my model.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Comments?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-mark<o:p></o:p></p>
</div>
</body>
</html>