Open Stack

Hi Jorge,

This thread as started mostly to try to determine what people are actually
after, as far as stats collection goes. Surprisingly (to me at least), it
doesn't look like we need all that much to meet the needs of those who have
responded to this thread thus far.

In any case, my sights are set on preparation for next week's hack-a-thon,
and in any case I don't see stats gathering as a high priority for this
group right now. (Though basic stats will need to happen before we can
consider the service "minimally viable," IMO.) In any case, I'm seeing
nothing anyone is asking for, stats-wise, which is going to cause problems
for us as we continue on our current course developing LBaaS.

And having this thread to look back on when it comes time to actually solve
the stats problem will be a good thing, eh.

Stephen


On Fri, Jun 6, 2014 at 11:35 AM, Jorge Miramontes <
jorge.miramontes at rackspace.com> wrote:

> Hey Stephen,
>
> What we really care about are the following:
>
> - Inbound bandwidth (bytes)
> - Outbound bandwidth (bytes)
> - "Instance" Uptime (requires create/delete events)
>
> Just to note our current LBaaS implementation at Rackspace keeps track of
> when features are enabled/disabled. For example, we have markers for when
> SSL is turned on/off, markers for when we suspend/unsuspend load
> balancers, etc. Some of this stuff is used for tracking purposes, some of
> it is used for billing purposes and some of it used for both purposes. We
> also keep track of all user initiated API requests to help us out when
> issues arise.
>
> From my experience building usage collection systems just know it is not a
> trivial task, especially if we need to track events. One good tip is to be
> as explicit as possible and as granular as possible. Being implicit causes
> bad things to happen. Also, if we didn't have UDP as a protocol I would
> recommend using Hadoop's map reduce functionality to get accurate
> statistics by map-reducing request logs.
>
> I would not advocate tracking per node statistics as the user can track
> that information by themselves if they really want to. We currently, don't
> have any customers that have asked for this feature.
>
> If you want to tackle the usage collection problem for Neutron LBaaS I
> would be glad to help as I've got quite a bit of experience in this
> subject matter.
>
> Cheers,
> --Jorge
>
>
>
>
> From:  <Eichberger>, German <german.eichberger at hp.com>
> Reply-To:  "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev at lists.openstack.org>
> Date:  Tuesday, June 3, 2014 5:20 PM
> To:  "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev at lists.openstack.org>
> Subject:  Re: [openstack-dev] [Neutron][LBaaS] Requirements around
> statistics and  billing
>
>
> >Hi Stephen,
> >
> >We would like all those numbers as well
> >J
> >
> >Additionally, we measure:
> >·
> >When a lb instance was created, deleted, etc.
> >·
> >For monitoring we ³ping² a load balancers health check and report/act on
> >the results
> >·
> >For user¹s troubleshooting we make the haproxy logs available. Which
> >contain connection information like from, to, duration, protocol, status
> >(though
> > we frequently have been told that this is not really useful for
> >debuggingŠ) and of course having that more gui-fied would be neat
> >
> >German
> >
> >
> >
> >From: Stephen Balukoff [mailto:sbalukoff at bluebox.net]
> >
> >Sent: Tuesday, May 27, 2014 8:22 PM
> >To: OpenStack Development Mailing List (not for usage questions)
> >Subject: [openstack-dev] [Neutron][LBaaS] Requirements around statistics
> >and billing
> >
> >
> >Hi folks!
> >
> >
> >We have yet to have any kind of meaningful discussion on this list around
> >load balancer stats (which, I presume to include data that will
> >eventually need to be consumed by a billing system). I'd like to get the
> >discussion started here,
> > as this will have significant meaning for how we both make this data
> >available to users, and how we implement back-end systems to be able to
> >provide this data.
> >
> >
> >
> >So!  What kinds of data are people looking for, as far as load balancer
> >statistics.
> >
> >
> >
> >For our part, as an absolute minimum we need the following per
> >loadbalancer + listener combination:
> >
> >
> >
> >* Total bytes transferred in for a given period
> >
> >* Total bytes transferred out for a given period
> >
> >
> >
> >Our product and billing people I'm sure would like the following as well:
> >
> >
> >
> >* Some kind of peak connections / second data (95th percentile or average
> >over a period, etc.)
> >
> >* Total connections for a given period
> >
> >* Total HTTP / HTTPS requests served for a given period
> >
> >
> >
> >And the people who work on UIs and put together dashboards would like:
> >
> >
> >
> >* Current requests / second (average for last X seconds, either
> >on-demand, or simply dumped regularly).
> >
> >* Current In/Out bytes throughput
> >
> >
> >
> >And our monitoring people would like this:
> >
> >
> >
> >* Errors / second
> >
> >* Current connections / second and bytes throughput secant slope (ie.
> >like derivative but easier to calculate from digital data) for last X
> >seconds (ie. detecting massive spikes or drops in traffic, potentially
> >useful for detecting a problem
> > before it becomes critical)
> >
> >
> >
> >And some of our users would like all of the above data per pool, and not
> >just for loadbalancer + listener. Some would also like to see it per
> >member (though I'm less inclined to make this part of our standard).
> >
> >
> >
> >I'm also interested in hearing vendor capabilities here, as it doesn't
> >make sense to design stats that most can't implement, and I imagine
> >vendors also have valuable data on what their customer ask for / what
> >stats are most useful in troubleshooting.
> >
> >
> >
> >What other statistics data for load balancing are meaningful and
> >hopefully not too arduous to calculate? What other data are your users
> >asking for or accustomed to seeing?
> >
> >
> >
> >Thanks,
> >
> >Stephen
> >
> >
> >
> >--
> >Stephen Balukoff
> >Blue Box Group, LLC
> >(800)613-4305 x807
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140609/0594a70d/attachment.html>