Open Stack

On Wed, 22 Oct 2014 11:29:27 AM Robert van Leeuwen wrote:
> > I,d like to start a conversation on usage requirements and have a few
> > suggestions. I advocate that, since we will be using TCP and HTTP/HTTPS
> > based protocols, we inherently enable connection logging for load
> 
> > balancers for several reasons:
> Just request from the operator side of things:
> Please think about the scalability when storing all logs.
> 
> e.g. we are currently logging http requests to one load balanced application
> (that would be a fit for LBAAS) It is about 500 requests per second, which
> adds up to 40GB per day (in elasticsearch.) Please make sure whatever
> solution is chosen it can cope with machines doing 1000s of requests per
> second...

And to take this further, what happens during DoS attack (either syn flood or 
full connections)?  How do we ensure that we don't lose our logging system 
and/or amplify the DoS attack?

One solution is sampling, with a tunable knob for the sampling rate - perhaps 
tunable per-vip.  This still increases linearly with attack traffic, unless you 
use time-based sampling (1-every-N-seconds rather than 1-every-N-packets).

One of the advantages of (eg) polling the number of current sessions is that 
the cost of that monitoring is essentially fixed regardless of the number of 
connections passing through.  Numerous other metrics (rate of new connections, 
etc) also have this property and could presumably be used for accurate billing 
- without amplifying attacks.

I think we should be careful about whether we want logging or metrics for more 
accurate billing.  Both are useful, but full logging is only really required 
for ad-hoc debugging (important! but different).

-- 
 - Gus