
<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">We had some issues related to this too, we ended up changing our collect_statistics_interval to 30 seconds as opposed to the default which is 5 I think.<br class=""><div class=""><br class=""></div><div class="">We also upgraded to 3.6.2 and that version is very buggy and wouldn’t recommend anyone to use it. It has a memory leak and some other nasty bugs we encountered.</div><div class=""><br class=""></div><div class="">3.6.1 on the other hand is very stable for us and we’ve been using it in production for several months now. </div><div class=""><br class=""></div><div class="">Sam</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 6 Jul 2016, at 3:50 AM, Alexey Lebedev <<a href="mailto:alebedev@mirantis.com" class="">alebedev@mirantis.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">Hi Joshua,<br class=""><br class=""></div><div class="">Does this happen with `<span class="">rates_mode` set to `none` and tuned `collect_statistics_interval`? Like in <a href="https://bugs.launchpad.net/fuel/+bug/1510835" class="">https://bugs.launchpad.net/fuel/+bug/1510835</a><br class=""><br class="">High connection/channel churn during upgrade can cause such issues.<br class=""><br class=""></span></div><div class="">BTW, soon-to-be-released rabbitmq 3.6.3 contains several improvements related to management plugin statistics handling. And almost every version before that also contained some related fixes. And I think that upstream devs response will have some mention of upgrade =)<br class=""><br class="">Best,<br class=""></div></div>Alexey<div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Jul 5, 2016 at 8:02 PM, Joshua Harlow <span dir="ltr" class=""><<a href="mailto:harlowja@fastmail.com" target="_blank" class="">harlowja@fastmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi ops and dev-folks,<br class="">

<br class="">

We over at godaddy (running rabbitmq with openstack) have been hitting a issue that has been causing the `rabbit_mgmt_db` consuming nearly all the processes memory (after a given amount of time),<br class="">

<br class="">

We've been thinking that this bug (or bugs?) may have existed for a while and our dual-version-path (where we upgrade the control plane and then slowly/eventually upgrade the compute nodes to the same version) has somehow triggered this memory leaking bug/issue since it has happened most prominently on our cloud which was running nova-compute at kilo and the other services at liberty (thus using the versioned objects code path more frequently due to needing translations of objects).<br class="">

<br class="">

The rabbit we are running is 3.4.0 on CentOS Linux release 7.2.1511 with kernel 3.10.0-327.4.4.el7.x86_64 (do note that upgrading to 3.6.2 seems to make the issue go away),<br class="">

<br class="">

# rpm -qa | grep rabbit<br class="">

<br class="">

rabbitmq-server-3.4.0-1.noarch<br class="">

<br class="">

The logs that seem relevant:<br class="">

<br class="">

```<br class="">

**********************************************************<br class="">

*** Publishers will be blocked until this alarm clears ***<br class="">

**********************************************************<br class="">

<br class="">

=INFO REPORT==== 1-Jul-2016::16:37:46 ===<br class="">

accepting AMQP connection <0.23638.342> (<a href="http://127.0.0.1:51932/" rel="noreferrer" target="_blank" class="">127.0.0.1:51932</a> -> <a href="http://127.0.0.1:5671/" rel="noreferrer" target="_blank" class="">127.0.0.1:5671</a>)<br class="">

<br class="">

=INFO REPORT==== 1-Jul-2016::16:37:47 ===<br class="">

vm_memory_high_watermark clear. Memory used:29910180640 allowed:47126781542<br class="">

```<br class="">

<br class="">

This happens quite often, the crashes have been affecting our cloud over the weekend (which made some dev/ops not so happy especially due to the july 4th mini-vacation),<br class="">

<br class="">

Looking to see if anyone else has seen anything similar?<br class="">

<br class="">

For those interested this is the upstream bug/mail that I'm also seeing about getting confirmation from the upstream users/devs (which also has erlang crash dumps attached/linked),<br class="">

<br class="">

<a href="https://groups.google.com/forum/#!topic/rabbitmq-users/FeBK7iXUcLg" rel="noreferrer" target="_blank" class="">https://groups.google.com/forum/#!topic/rabbitmq-users/FeBK7iXUcLg</a><br class="">

<br class="">

Thanks,<br class="">

<br class="">

-Josh<br class="">

<br class="">

__________________________________________________________________________<br class="">

OpenStack Development Mailing List (not for usage questions)<br class="">

Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" rel="noreferrer" target="_blank" class="">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="">

<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="">

</blockquote></div><br class=""><br clear="all" class=""><br class="">-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class="">Best,<div class="">Alexey</div></div></div>

</div></div>

__________________________________________________________________________<br class="">OpenStack Development Mailing List (not for usage questions)<br class="">Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" class="">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class=""></div></blockquote></div><br class=""></div></body></html>