<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I recently got into trouble with a large backlog. What I found was at some point the backlog got too large for gnocchi to effectivly function. When using ceph list of metric objects is kept in a omap object which normally is a quick and efficient way to store this list. However, at some point the list grows too large for it to be managed by the leveldb which implements the omap k/v store. I finally moved to some ssd’s to get enough iops for leveldb/omap to function. What I’m guessing is that if you are using ceph the increased number of metrics grabbed per pass reduced the number of times a now expensive operation is performed. Indications are that the new bluestore should make omap scale much better but isn’t slated to go stable for a few months with the release of Luminous.<div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 28, 2017, at 2:28 PM, Ionut Biru - Fleio <<a href="mailto:ionut@fleio.com" class="">ionut@fleio.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif;" class=""><div style="margin-top: 0px; margin-bottom: 0px;" class="">Hello,</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">I do have a cloud under administration,<span class="Apple-converted-space"> </span><span class="">my setup is fairly basic, I have deployed openstack using Openstack Ansible, currently I'm a Newton and planning to upgrade on Ocata.</span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">I'm having a problem with gnocchi metricd falling behind on processing metrics.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Gnocchi config:<span class="Apple-converted-space"> </span><a href="https://paste.xinu.at/f73A/" class="OWAAutoLink" id="LPlnk129397" previewremoved="true">https://paste.xinu.at/f73A/</a><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">In I'm using default workers number(cpu count) the number of<span class="Apple-converted-space"> </span><span class="">"storage/total number of measures to process</span>" keeps growing, last time I had 300k in queue. In seems that the tasks are not rescheduled in order to process them all in time and it processing couples of metrics after they are received from ceilometer and after that they are kept in queue and I only have 10 compute nodes with about 70 instances.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">In order to process I had to set up workers to a very high number (100) and keep restarting metricd in order for them to be processed but this method is very cpu and memory intensive and luckily I found another method that works quite well.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><a href="https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/cli.py?h=stable/3.1#n154" class="OWAAutoLink" id="LPlnk662575" previewremoved="true">https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/cli.py?h=stable/3.1#n154</a></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span class="">I have modified TASKS_PER_WORKER and BLOCK_SIZE to 400 and<span class="Apple-converted-space"> </span></span>now metricd keeps processing them.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">I'm not sure yet if is a bug or not but my question is, how do you guys scale gnocchi metricd in order to process a lot of resources and metrics?<br class=""><span class=""><br class=""></span></div></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">OpenStack-operators mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:OpenStack-operators@lists.openstack.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">OpenStack-operators@lists.openstack.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></div></blockquote></div><br class=""></div></body></html>