<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>I'm not using influxdb, just basic configuration generated by openstack ansible, which enables file storage by default.</p>
<p><br>
</p>
<p>The reason for bumping those values was to process a lot of measures and 400 seems a high number at that time.</p>
<p><br>
</p>
<p>I did use the below values without any impact<br>
</p>
<p>metric_processing_delay = 0</p>
<p>metric_reporting_delay = 1 <br>
</p>
<p>metric_cleanul_delay = 10</p>
<br>
<p>I'm opened to apply any configuration modification to my setup in order to resolve my issue without any code modification(that i did).
<br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Alex Krzos <akrzos@redhat.com><br>
<b>Sent:</b> Tuesday, March 28, 2017 8:19:58 PM<br>
<b>To:</b> Ionut Biru - Fleio<br>
<b>Cc:</b> openstack-operators@lists.openstack.org<br>
<b>Subject:</b> Re: [Openstack-operators] scaling gnocchi metricd</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">This is interesting, thanks for sharing. I assume your using an<br>
influxdb storage driver correct? I have also wondered if there was a<br>
specific reason for the TASKS_PER_WORKER and BLOCK_SIZE values.<br>
<br>
Also did you have to adjust your metric_processing_delay?<br>
<br>
<br>
Alex Krzos | Performance Engineering<br>
Red Hat<br>
Desk: 919-754-4280<br>
Mobile: 919-909-6266<br>
<br>
<br>
On Tue, Mar 28, 2017 at 3:28 PM, Ionut Biru - Fleio <ionut@fleio.com> wrote:<br>
> Hello,<br>
><br>
><br>
> I do have a cloud under administration, my setup is fairly basic, I have<br>
> deployed openstack using Openstack Ansible, currently I'm a Newton and<br>
> planning to upgrade on Ocata.<br>
><br>
><br>
> I'm having a problem with gnocchi metricd falling behind on processing<br>
> metrics.<br>
><br>
><br>
> Gnocchi config: <a href="https://paste.xinu.at/f73A/">https://paste.xinu.at/f73A/</a><br>
><br>
><br>
> In I'm using default workers number(cpu count) the number of "storage/total<br>
> number of measures to process" keeps growing, last time I had 300k in queue.<br>
> In seems that the tasks are not rescheduled in order to process them all in<br>
> time and it processing couples of metrics after they are received from<br>
> ceilometer and after that they are kept in queue and I only have 10 compute<br>
> nodes with about 70 instances.<br>
><br>
><br>
> In order to process I had to set up workers to a very high number (100) and<br>
> keep restarting metricd in order for them to be processed but this method is<br>
> very cpu and memory intensive and luckily I found another method that works<br>
> quite well.<br>
><br>
><br>
> <a href="https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/cli.py?h=stable/3.1#n154">
https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/cli.py?h=stable/3.1#n154</a><br>
><br>
><br>
> I have modified TASKS_PER_WORKER and BLOCK_SIZE to 400 and now metricd keeps<br>
> processing them.<br>
><br>
><br>
> I'm not sure yet if is a bug or not but my question is, how do you guys<br>
> scale gnocchi metricd in order to process a lot of resources and metrics?<br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> OpenStack-operators@lists.openstack.org<br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
><br>
</div>
</span></font>
</body>
</html>