<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Jorge,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Good discussion so far + glad to have you back
</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1F497D">J</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I am not a big fan of using logs for billing information since ultimately (at least at HP) we need to pump it into ceilometer. So I am envisioning either the
amphora (via a proxy) to pump it straight into that system or we collect it on the controller and pump it from there.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Allowing/enabling logging creates some requirements on the hardware, mainly, that they can handle the IO coming from logging. Some operators might choose to
hook up very cheap and non performing disks which might not be able to deal with the log traffic. So I would suggest that there is some rate limiting on the log output to help with that.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">German<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Jorge Miramontes [mailto:jorge.miramontes@RACKSPACE.COM]
<br>
<b>Sent:</b> Wednesday, October 22, 2014 6:51 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [Neutron][LBaaS][Octavia] Usage Requirements<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Hey Stephen (and Robert),<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">For real-time usage I was thinking something similar to what you are proposing. Using logs for this would be overkill IMO so your suggestions were what I was
thinking of starting with.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">As far as storing logs is concerned I was definitely thinking of offloading these onto separate storage devices. Robert, I totally hear you on the scalability
part as our current LBaaS setup generates TB of request logs. I'll start planning out a spec and then I'll let everyone chime in there. I just wanted to get a general feel for the ideas I had mentioned. I'll also bring it up in today's meeting.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Cheers,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">--Jorge<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">From:
</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Stephen Balukoff <<a href="mailto:sbalukoff@bluebox.net">sbalukoff@bluebox.net</a>><br>
<b>Reply-To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Date: </b>Wednesday, October 22, 2014 4:04 AM<br>
<b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [openstack-dev] [Neutron][LBaaS][Octavia] Usage Requirements<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Hi Jorge!
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Welcome back, eh! You've been missed.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Anyway, I just wanted to say that your proposal sounds great to me, and it's good to finally be closer to having concrete requirements for logging, eh. Once this
discussion is nearing a conclusion, could you write up the specifics of logging into a specification proposal document?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Regarding the discussion itself: I think we can ignore UDP for now, as there doesn't seem to be high demand for it, and it certainly won't be supported in v 0.5
of Octavia (and maybe not in v1 or v2 either, unless we see real demand).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Regarding the 'real-time usage' information: I have some ideas regarding getting this from a combination of iptables and / or the haproxy stats interface. Were
you thinking something different that involves on-the-fly analysis of the logs or something? (I tend to find that logs are great for non-real time data, but can often be lacking if you need, say, a gauge like 'currently open connections' or something.)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">One other thing: If there's a chance we'll be storing logs on the amphorae themselves, then we need to have log rotation as part of the configuration here. It
would be silly to have an amphora failure just because its ephemeral disk fills up, eh.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Stephen<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">On Wed, Oct 15, 2014 at 4:03 PM, Jorge Miramontes <<a href="mailto:jorge.miramontes@rackspace.com" target="_blank">jorge.miramontes@rackspace.com</a>> wrote:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Hey Octavia folks!<br>
<br>
<br>
First off, yes, I'm still alive and kicking. :)<br>
<br>
I,d like to start a conversation on usage requirements and have a few<br>
suggestions. I advocate that, since we will be using TCP and HTTP/HTTPS<br>
based protocols, we inherently enable connection logging for load<br>
balancers for several reasons:<br>
<br>
1) We can use these logs as the raw and granular data needed to track<br>
usage. With logs, the operator has flexibility as to what usage metrics<br>
they want to bill against. For example, bandwidth is easy to track and can<br>
even be split into header and body data so that the provider can choose if<br>
they want to bill on header data or not. Also, the provider can determine<br>
if they will bill their customers for failed requests that were the fault<br>
of the provider themselves. These are just a few examples; the point is<br>
the flexible nature of logs.<br>
<br>
2) Creating billable usage from logs is easy compared to other options<br>
like polling. For example, in our current LBaaS iteration at Rackspace we<br>
bill partly on "average concurrent connections". This is based on polling<br>
and is not as accurate as it possibly can be. It's very close, but it<br>
doesn't get more accurate that the logs themselves. Furthermore, polling<br>
is more complex and uses up resources on the polling cadence.<br>
<br>
3) Enabling logs for all load balancers can be used for debugging, support<br>
and audit purposes. While the customer may or may not want their logs<br>
uploaded to swift, operators and their support teams can still use this<br>
data to help customers out with billing and setup issues. Auditing will<br>
also be easier with raw logs.<br>
<br>
4) Enabling logs for all load balancers will help mitigate uncertainty in<br>
terms of capacity planning. Imagine if every customer suddenly enabled<br>
logs without it ever being turned on. This could produce a spike in<br>
resource utilization that will be hard to manage. Enabling logs from the<br>
start means we are certain as to what to plan for other than the nature of<br>
the customer's traffic pattern.<br>
<br>
Some Cons I can think of (please add more as I think the pros outweigh the<br>
cons):<br>
<br>
1) If we every add UDP based protocols then this model won't work. < 1% of<br>
our load balancers at Rackspace are UDP based so we are not looking at<br>
using this protocol for Octavia. I'm more of a fan of building a really<br>
good TCP/HTTP/HTTPS based load balancer because UDP load balancing solves<br>
a different problem. For me different problem == different product.<br>
<br>
2) I'm assuming HA Proxy. Thus, if we choose another technology for the<br>
amphora then this model may break.<br>
<br>
<br>
Also, and more generally speaking, I have categorized usage into three<br>
categories:<br>
<br>
1) Tracking usage - this is usage that will be used my operators and<br>
support teams to gain insight into what load balancers are doing in an<br>
attempt to monitor potential issues.<br>
2) Billable usage - this is usage that is a subset of tracking usage used<br>
to bill customers.<br>
3) Real-time usage - this is usage that should be exposed via the API so<br>
that customers can make decisions that affect their configuration (ex.<br>
"Based off of the number of connections my web heads can handle when<br>
should I add another node to my pool?").<br>
<br>
These are my preliminary thoughts, and I'd love to gain insight into what<br>
the community thinks. I have built about 3 usage collection systems thus<br>
far (1 with Brandon) and have learned a lot. Some basic rules I have<br>
discovered with collecting usage are:<br>
<br>
1) Always collect granular usage as it "paints a picture" of what actually<br>
happened. Massaged/un-granular usage == lost information.<br>
2) Never imply, always be explicit. Implications usually stem from bad<br>
assumptions.<br>
<br>
<br>
Last but not least, we need to store every user and system load balancer<br>
event such as creation, updates, suspension and deletion so that we may<br>
bill on things like uptime and serve our customers better by knowing what<br>
happened and when.<br>
<br>
<br>
Cheers,<br>
--Jorge<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">--
<br>
Stephen Balukoff <br>
Blue Box Group, LLC <br>
(800)613-4305 x807 <o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>