<div dir="ltr"><p dir="ltr">Puhh, that is is going to be endless :D</p><p dir="ltr"><br></p><p>I want to see:</p><p>Operating System / Hardware: system load, cpu user, cpu system, memory usage, swap, swap activity, user/free disk space, nic bw, nic packets, disk bw, disk iops, zombie procs, hdd smart, RAID status<br>Docker: cpu / memory per container, volume usage<br>OpenStack: # free FIPs, # instances, # volumes, # networks, # routers, # networks<br>HAproxy: # sessions per listener, http status codes per service<br>RabbitMQ: message rates, # messages in queue<br>Ceph: placement group states, available capacity, pool bw / iops, disk latency, journal latency<br></p><p>I want alerts for:</p><p>- node goes down<br>- ceph: mon goes down, osd goes down, pgs stuck for more than X seconds<br>- service goes down<br>- container up/down<br>- mariadb/galera: cluster health<br>- rabbitmq: cluster health</p><p>That's just from the top of my head :) The reason why I don't want alerts on everything is that most solutions work with static thresholds which is mostly useless. I prefer walking through my dashboards every morning checking stuff myself.</p><p>cheers,</p><p>Mathias</p><p><br></p>
<div class="gmail_quote">Am 24.07.2016 17:16 schrieb "Michał Jastrzębski" <<a href="mailto:inc007@gmail.com" target="_blank">inc007@gmail.com</a>>:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Guys, thanks for all that!<br>
<br>
Can we for a second abstract this discussion from technology and start<br>
by lining up scenerios we want to achieve. Then put a software that<br>
will allow us to achieve all/most of scenerios with least amount of<br>
work/maintenance?<br>
<br>
So my scenerios:<br>
<br>
I want to see when health of docker service<br>
I want to see when message queue becomes saturated<br>
I want to see when RAM exceeds 70%<br>
I want to see when my network causes tons of retransmissions<br>
I want to see when one of nodes is down<br>
<br>
Did I miss anything? Which software stack would allow me to see these things?<br>
<br>
Cheers,<br>
Michal<br>
<br>
On 24 July 2016 at 09:09, Mathias Ewald <<a href="mailto:mewald@evoila.de" target="_blank">mewald@evoila.de</a>> wrote:<br>
> I think Sensu is the best monitoring approach out there atm. Nagios / Icinga<br>
> are way to static and scale badly imho. The kind of checks you proposed are<br>
> quite interesting. I would suggest to wrap a sensu check around Tempest but<br>
> that's going to far for the first cycle.<br>
><br>
> The two stacks (Sensu + Unchiwa and TICK) only really overlap in metrics<br>
> collection which can be done via Sensu and Telegraf. I don't know if it<br>
> makes sense to have both ... I definitely think we need Sensu though simply<br>
> to monitor service availability and other thresholds and events which aren't<br>
> covered in TICK as not everything is time series data and to have the<br>
> alerting. Only with Sensu we don't have insight into performance and trends,<br>
> with TICK only we lack alerting on events and non-performance metric data<br>
> (Is Keystone up? etc)<br>
><br>
> I think it won't hurt to develop theses two stacks in parallel and maybe<br>
> we'll join them together in a chain as I described earlier.<br>
><br>
> 2016-07-24 14:25 GMT+02:00 Dave Walker <<a href="mailto:email@daviey.com" target="_blank">email@daviey.com</a>>:<br>
>><br>
>> Thanks Mathias,<br>
>><br>
>> I'm not tied to Sensu.. anything can really fill that gap in my mind.<br>
>> You've done a good job at outlining the steps involved. I created a<br>
>> blueprint with the steps I had in mind[0]<br>
>><br>
>> For this cycle, I wanted to keep it simple so it was easily achievable. I<br>
>> only planned to have some basic up/down for each node and throw the<br>
>> performance data on the floor.<br>
>><br>
>> I wanted to include the option to include local configs, as json blobs.<br>
>> Some of the things I was thinking as local config:<br>
>> - daily checkouts, can instances be built with networking<br>
>> - remaining resources count (ie, does each subnet have X remaining ip<br>
>> addresses available)<br>
>> - Is Ceph healthy?<br>
>><br>
>> So, these things aren't really performance over time interesting.. which<br>
>> means the intention does differ. However, I do agree that both stacks could<br>
>> achieve both objectives.<br>
>><br>
>> I've essentially got much of this working locally, but would require about<br>
>> a day of cleaning up for submission... but if your work can achieve the<br>
>> objectives above, i'm happy to discontinue... or help make your stack<br>
>> pluggable.<br>
>><br>
>> [0] <a href="https://blueprints.launchpad.net/kolla/+spec/sensu" rel="noreferrer" target="_blank">https://blueprints.launchpad.net/kolla/+spec/sensu</a><br>
>><br>
>> --<br>
>> Kind Regards,<br>
>> Dave Walker<br>
>><br>
>> On 24 July 2016 at 11:56, Mathias Ewald <<a href="mailto:mewald@evoila.de" target="_blank">mewald@evoila.de</a>> wrote:<br>
>>><br>
>>> Monitoring is a difficult topic as the number of options regarding the<br>
>>> toolset and mechanisms are very high. We had some chats about it in IRC that<br>
>>> discovered even more options than I thought existed :D I believe Dave's view<br>
>>> on Sensu is generally correct in that Sensu is more directed to monitoring<br>
>>> in the form of "if X running/working" but of course has the ability to<br>
>>> transport metrics, too, but lacks the good dashboarding capabilities for<br>
>>> performance data. One set up I could images is<br>
>>><br>
>>> 1. Sensu Client to collect checks and metrics<br>
>>> 2. RabbitMQ for transport<br>
>>> 3. Sensu Server to receive, evaluate, alarm and write metrics to InfluxDB<br>
>>> 4. Uchiwa as a Dashboard to Sensu<br>
>>> 5. InfluxDB to store metrics<br>
>>> 6. Grafana to dashboard metrics<br>
>>><br>
>>> So Sensu could be used as a replacement for (or in addition to) a metrics<br>
>>> collection daemon like Collectd or what I decided to use: Telegraf. For my<br>
>>> implementation, this means I will add a parameter to make Telegraf optional.<br>
>>> This way, someone else may implement the rest of the stack and the user can<br>
>>> decide which one to use.<br>
>>><br>
>>> What do you think?<br>
>>><br>
>>> Mathias<br>
>>><br>
>>><br>
>>><br>
>>> 2016-07-23 21:51 GMT+02:00 Stephen Hindle <<a href="mailto:shindle@llnw.com" target="_blank">shindle@llnw.com</a>>:<br>
>>>><br>
>>>> My understanding was Sensu could produce metrics ?<br>
>>>> And Kapacitor can do alerting for the TICK stack stuff mewald is<br>
>>>> doing...<br>
>>>> I really don't see them as that different ?<br>
>>>><br>
>>>><br>
>>>> On Fri, Jul 22, 2016 at 5:19 PM, Dave Walker <<a href="mailto:email@daviey.com" target="_blank">email@daviey.com</a>> wrote:<br>
>>>> > Yes, this is my thought.<br>
>>>> ><br>
>>>> > The scope of the Sensu work is: "Is this thing working?" (with the<br>
>>>> > reference<br>
>>>> > being up/down)<br>
>>>> > But the scope of the Grafana and friends is, "How hard is this<br>
>>>> > working?"<br>
>>>> > (but no alerting)<br>
>>>> ><br>
>>>> > They are certainly complementary.... However, Sensu can throw data at<br>
>>>> > a<br>
>>>> > Grafana stack (aiui).. but I fear that is too much to achieve this<br>
>>>> > cycle.<br>
>>>> ><br>
>>>> > --<br>
>>>> > Kind Regards,<br>
>>>> > Dave Walker<br>
>>>> ><br>
>>>> > On 23 July 2016 at 00:11, Fox, Kevin M <<a href="mailto:Kevin.Fox@pnnl.gov" target="_blank">Kevin.Fox@pnnl.gov</a>> wrote:<br>
>>>> >><br>
>>>> >> I think those are two different, complementary things.<br>
>>>> >><br>
>>>> >> One's metrics and the other is monitoring. You probably want both at<br>
>>>> >> the<br>
>>>> >> same time.<br>
>>>> >><br>
>>>> >> Thanks,<br>
>>>> >> Kevin<br>
>>>> >> ________________________________________<br>
>>>> >> From: Steven Dake (stdake) [<a href="mailto:stdake@cisco.com" target="_blank">stdake@cisco.com</a>]<br>
>>>> >> Sent: Friday, July 22, 2016 3:52 PM<br>
>>>> >> To: OpenStack Development Mailing List (not for usage questions)<br>
>>>> >> Subject: Re: [openstack-dev] [kolla] Monitoring tooling<br>
>>>> >><br>
>>>> >> Thanks for pointing that out. Brain out to lunch today it appears :(<br>
>>>> >><br>
>>>> >> I think choices are a good thing even though they increase our<br>
>>>> >> implementation footprint. Anyone opposed to implementing both with<br>
>>>> >> something in globals.yml like<br>
>>>> >> monitoring: grafana or<br>
>>>> >> monitoring: sensu<br>
>>>> >><br>
>>>> >> Comments questions or concerns welcome.<br>
>>>> >><br>
>>>> >> Regards<br>
>>>> >> -steve<br>
>>>> >><br>
>>>> >> On 7/22/16, 3:42 PM, "Stephen Hindle" <<a href="mailto:shindle@llnw.com" target="_blank">shindle@llnw.com</a>> wrote:<br>
>>>> >><br>
>>>> >> >Don't forget mewalds implementation as well - we now have 2<br>
>>>> >> > monitoring<br>
>>>> >> >options for kolla :-)<br>
>>>> >> ><br>
>>>> >> >On Fri, Jul 22, 2016 at 3:15 PM, Steven Dake (stdake)<br>
>>>> >> > <<a href="mailto:stdake@cisco.com" target="_blank">stdake@cisco.com</a>><br>
>>>> >> >wrote:<br>
>>>> >> >> Hi folks,<br>
>>>> >> >><br>
>>>> >> >> At the midcycle we decided to push off implementing Monitoring<br>
>>>> >> >> until<br>
>>>> >> >>post<br>
>>>> >> >> Newton. The rationale for this decision was that the core review<br>
>>>> >> >> team<br>
>>>> >> >>has<br>
>>>> >> >> enough on their plates and nobody was super keen to implement any<br>
>>>> >> >>monitoring<br>
>>>> >> >> solution given our other priorities.<br>
>>>> >> >><br>
>>>> >> >> Like all good things, communities produce new folks that want to<br>
>>>> >> >> do new<br>
>>>> >> >> things, and Sensu was proposed as Kolla's monitoring solution<br>
>>>> >> >> (atleast<br>
>>>> >> >>the<br>
>>>> >> >> first one). A developer that has done some good work has shown up<br>
>>>> >> >> to<br>
>>>> >> >>do the<br>
>>>> >> >> job as well :) I have heard good things about Sensu, minus the<br>
>>>> >> >> fact<br>
>>>> >> >>that it<br>
>>>> >> >> is implemented in Ruby and I fear it may end up causing our gate a<br>
>>>> >> >> lot<br>
>>>> >> >>of<br>
>>>> >> >> hassle.<br>
>>>> >> >><br>
>>>> >> >> <a href="https://review.openstack.org/#/c/341861/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/341861/</a><br>
>>>> >> >><br>
>>>> >> >><br>
>>>> >> >> Anyway I think we can work through the gate problem.<br>
>>>> >> >><br>
>>>> >> >> Does anyone have any better suggestion? I'd like to unblock<br>
>>>> >> >> Dave's<br>
>>>> >> >> work<br>
>>>> >> >> which is blocked on a 2 pending a complete discussion of our<br>
>>>> >> >> monitoring<br>
>>>> >> >> solution. Note we may end up implementing more than one down the<br>
>>>> >> >> road<br>
>>>> >> >> <br>
>>>> >> >> Sensu is just where the original interest was.<br>
>>>> >> >><br>
>>>> >> >> Please provide feedback, even if you don't have a preference,<br>
>>>> >> >> whether<br>
>>>> >> >>your a<br>
>>>> >> >> core reviewer or not.<br>
>>>> >> >><br>
>>>> >> >> My take is we can merge this work in non-prioirty order, and if it<br>
>>>> >> >>makes the<br>
>>>> >> >> end of the cycle fantastic if not, we can release it in Ocatta.<br>
>>>> >> >><br>
>>>> >> >> Regards<br>
>>>> >> >> -steve<br>
>>>> >> >><br>
>>>> >> >><br>
>>>> >> >><br>
>>>> >><br>
>>>> >> >><br>
>>>> >> >> >> >>_________________________________________________________________________<br>
>>>> >> >>_<br>
>>>> >> >> OpenStack Development Mailing List (not for usage questions)<br>
>>>> >> >> Unsubscribe:<br>
>>>> >> >><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> >> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>> >> >><br>
>>>> >> ><br>
>>>> >> ><br>
>>>> >> ><br>
>>>> >> >--<br>
>>>> >> >Stephen Hindle - Senior Systems Engineer<br>
>>>> >> ><a href="tel:480.807.8189" value="+14808078189" target="_blank">480.807.8189</a> <a href="tel:480.807.8189" value="+14808078189" target="_blank">480.807.8189</a><br>
>>>> >> ><a href="http://www.limelight.com" rel="noreferrer" target="_blank">www.limelight.com</a> Delivering Faster Better<br>
>>>> >> ><br>
>>>> >> >Join the conversation<br>
>>>> >> ><br>
>>>> >> >at Limelight Connect<br>
>>>> >> ><br>
>>>> >> >--<br>
>>>> >> >The information in this message may be confidential. It is intended<br>
>>>> >> >solely<br>
>>>> >> >for<br>
>>>> >> >the addressee(s). If you are not the intended recipient, any<br>
>>>> >> > disclosure,<br>
>>>> >> >copying or distribution of the message, or any action or omission<br>
>>>> >> > taken<br>
>>>> >> >by<br>
>>>> >> >you<br>
>>>> >> >in reliance on it, is prohibited and may be unlawful. Please<br>
>>>> >> > immediately<br>
>>>> >> >contact the sender if you have received this message in error.<br>
>>>> >> ><br>
>>>> >> ><br>
>>>> >><br>
>>>> >> ><br>
>>>> >> > > >__________________________________________________________________________<br>
>>>> >> >OpenStack Development Mailing List (not for usage questions)<br>
>>>> >> >Unsubscribe:<br>
>>>> >> > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> >> ><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>> >><br>
>>>> >><br>
>>>> >><br>
>>>> >> __________________________________________________________________________<br>
>>>> >> OpenStack Development Mailing List (not for usage questions)<br>
>>>> >> Unsubscribe:<br>
>>>> >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>> >><br>
>>>> >><br>
>>>> >> __________________________________________________________________________<br>
>>>> >> OpenStack Development Mailing List (not for usage questions)<br>
>>>> >> Unsubscribe:<br>
>>>> >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>> ><br>
>>>> ><br>
>>>> ><br>
>>>> ><br>
>>>> > __________________________________________________________________________<br>
>>>> > OpenStack Development Mailing List (not for usage questions)<br>
>>>> > Unsubscribe:<br>
>>>> > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>> ><br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> Stephen Hindle - Senior Systems Engineer<br>
>>>> <a href="tel:480.807.8189" value="+14808078189" target="_blank">480.807.8189</a> <a href="tel:480.807.8189" value="+14808078189" target="_blank">480.807.8189</a><br>
>>>> <a href="http://www.limelight.com" rel="noreferrer" target="_blank">www.limelight.com</a> Delivering Faster Better<br>
>>>><br>
>>>> Join the conversation<br>
>>>><br>
>>>> at Limelight Connect<br>
>>>><br>
>>>> --<br>
>>>> The information in this message may be confidential. It is intended<br>
>>>> solely<br>
>>>> for<br>
>>>> the addressee(s). If you are not the intended recipient, any<br>
>>>> disclosure,<br>
>>>> copying or distribution of the message, or any action or omission taken<br>
>>>> by<br>
>>>> you<br>
>>>> in reliance on it, is prohibited and may be unlawful. Please<br>
>>>> immediately<br>
>>>> contact the sender if you have received this message in error.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> __________________________________________________________________________<br>
>>>> OpenStack Development Mailing List (not for usage questions)<br>
>>>> Unsubscribe:<br>
>>>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Mobil: <a href="tel:%2B49%20176%2010567592" value="+4917610567592" target="_blank">+49 176 10567592</a><br>
>>> E-Mail: <a href="mailto:mewald@evoila.de" target="_blank">mewald@evoila.de</a><br>
>>><br>
>>> evoila GmbH<br>
>>> Wilhelm-Theodor-Römheld-Str. 34<br>
>>> 55130 Mainz<br>
>>> Germany<br>
>>><br>
>>> Geschäftsführer: Johannes Hiemer<br>
>>><br>
>>> Amtsgericht Mainz HRB 42719<br>
>>><br>
>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte<br>
>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail<br>
>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und<br>
>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte<br>
>>> Weitergabe dieser Mail ist nicht gestattet.<br>
>>><br>
>>> This e-mail may contain confidential and/or privileged information. If<br>
>>> You are not the intended recipient (or have received this e-mail in error)<br>
>>> please notify the sender immediately and destroy this e-mail. Any<br>
>>> unauthorised copying, disclosure or distribution of the material in this<br>
>>> e-mail is strictly forbidden.<br>
>>><br>
>>><br>
>>> __________________________________________________________________________<br>
>>> OpenStack Development Mailing List (not for usage questions)<br>
>>> Unsubscribe:<br>
>>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>><br>
>><br>
>> __________________________________________________________________________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
><br>
><br>
> --<br>
> Mobil: <a href="tel:%2B49%20176%2010567592" value="+4917610567592" target="_blank">+49 176 10567592</a><br>
> E-Mail: <a href="mailto:mewald@evoila.de" target="_blank">mewald@evoila.de</a><br>
><br>
> evoila GmbH<br>
> Wilhelm-Theodor-Römheld-Str. 34<br>
> 55130 Mainz<br>
> Germany<br>
><br>
> Geschäftsführer: Johannes Hiemer<br>
><br>
> Amtsgericht Mainz HRB 42719<br>
><br>
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte<br>
> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail<br>
> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und<br>
> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte<br>
> Weitergabe dieser Mail ist nicht gestattet.<br>
><br>
> This e-mail may contain confidential and/or privileged information. If You<br>
> are not the intended recipient (or have received this e-mail in error)<br>
> please notify the sender immediately and destroy this e-mail. Any<br>
> unauthorised copying, disclosure or distribution of the material in this<br>
> e-mail is strictly forbidden.<br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div>
</div>