<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I think it was somewhere around the 2M mark.<div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 30, 2017, at 8:33 AM, Alex Krzos <<a href="mailto:akrzos@redhat.com" class="">akrzos@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On Tue, Mar 28, 2017 at 3:55 PM, Mike Lowe <</span><a href="mailto:jomlowe@iu.edu" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">jomlowe@iu.edu</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">> wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">I recently got into trouble with a large backlog. What I found was at some<br class="">point the backlog got too large for gnocchi to effectivly function. When<br class="">using ceph list of metric objects is kept in a omap object which normally is<br class="">a quick and efficient way to store this list. However, at some point the<br class="">list grows too large for it to be managed by the leveldb which implements<br class="">the omap k/v store.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Can you share at what number of keys were stored in the omap object</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">when this became a problem.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Thanks,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Alex</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">I finally moved to some ssd’s to get enough iops for<br class="">leveldb/omap to function. What I’m guessing is that if you are using ceph<br class="">the increased number of metrics grabbed per pass reduced the number of times<br class="">a now expensive operation is performed. Indications are that the new<br class="">bluestore should make omap scale much better but isn’t slated to go stable<br class="">for a few months with the release of Luminous.<br class=""><br class=""><br class="">On Mar 28, 2017, at 2:28 PM, Ionut Biru - Fleio <<a href="mailto:ionut@fleio.com" class="">ionut@fleio.com</a>> wrote:<br class=""><br class="">Hello,<br class=""><br class="">I do have a cloud under administration, my setup is fairly basic, I have<br class="">deployed openstack using Openstack Ansible, currently I'm a Newton and<br class="">planning to upgrade on Ocata.<br class=""><br class="">I'm having a problem with gnocchi metricd falling behind on processing<br class="">metrics.<br class=""><br class="">Gnocchi config: <a href="https://paste.xinu.at/f73A/" class="">https://paste.xinu.at/f73A/</a><br class=""><br class="">In I'm using default workers number(cpu count) the number of "storage/total<br class="">number of measures to process" keeps growing, last time I had 300k in queue.<br class="">In seems that the tasks are not rescheduled in order to process them all in<br class="">time and it processing couples of metrics after they are received from<br class="">ceilometer and after that they are kept in queue and I only have 10 compute<br class="">nodes with about 70 instances.<br class=""><br class="">In order to process I had to set up workers to a very high number (100) and<br class="">keep restarting metricd in order for them to be processed but this method is<br class="">very cpu and memory intensive and luckily I found another method that works<br class="">quite well.<br class=""><br class=""><a href="https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/cli.py?h=stable/3.1#n154" class="">https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/cli.py?h=stable/3.1#n154</a><br class=""><br class="">I have modified TASKS_PER_WORKER and BLOCK_SIZE to 400 and now metricd keeps<br class="">processing them.<br class=""><br class="">I'm not sure yet if is a bug or not but my question is, how do you guys<br class="">scale gnocchi metricd in order to process a lot of resources and metrics?<br class=""><br class="">_______________________________________________<br class="">OpenStack-operators mailing list<br class="">OpenStack-operators@lists.openstack.org<br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators<br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">OpenStack-operators mailing list<br class=""><a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></blockquote></div></blockquote></div><br class=""></div></body></html>