Open Stack

Same here, running 3.6.5 for (some) of the rabbit clusters.

It's been stable over the last month (fingers crossed!), though:
* gave up on stats collection (set to 60000 which makes it not so useful)
* can still make it very sick with a couple of misconfigured clients
(rabbit_retry_interval=1 and rabbit_retry_backoff=60 currently
everywhere).

Some data from the neutron rabbit cluster (3 vm nodes, not all infra
currently talks to neutron):

* connections: ~8k
* memory used per node: 2.5GB, 1.7GB, 0.1GB (the last one is less used
due to a previous net partition i believe)
* rabbit hiera configuration
rabbitmq::cluster_partition_handling: 'autoheal'
rabbitmq::config_kernel_variables:
  inet_dist_listen_min: 41055
  inet_dist_listen_max: 41055
rabbitmq::config_variables:
  collect_statistics_interval: 60000
  reverse_dns_lookups: true
  vm_memory_high_watermark: 0.8
rabbitmq::environment_variables:
  SERVER_ERL_ARGS: "'+K true +A 128 +P 1048576'"
rabbitmq::tcp_keepalive: true
rabbitmq::tcp_backlog: 4096

* package versions

erlang-kernel-18.3.4.4-1
rabbitmq-server-3.6.5-1

It's stable enough to keep scaling it up in the next couple months and
see how it goes.

Cheers,
  Ricardo

On Mon, Jan 9, 2017 at 3:54 AM, Sam Morrison <sorrison at gmail.com> wrote:
> We’ve been running 3.6.5 for sometime now and it’s working well.
>
> 3.6.1 - 3.6.3 are unusable, we had lots of issues with stats DB and other
> weirdness.
>
> Our setup is a 3 physical node cluster with around 9k connections, average
> around the 300 messages/sec delivery. We have the stats sample rate set to
> default and it is working fine.
>
> Yes we did have to restart the cluster to upgrade.
>
> Cheers,
> Sam
>
>
>
> On 6 Jan 2017, at 5:26 am, Matt Fischer <matt at mattfischer.com> wrote:
>
> MIke,
>
> I did a bunch of research and experiments on this last fall. We are running
> Rabbit 3.5.6 on our main cluster and 3.6.5 on our Trove cluster which has
> significantly less load (and criticality). We were going to upgrade to 3.6.5
> everywhere but in the end decided not to, mainly because there was little
> perceived benefit at the time. Our main issue is unchecked memory growth at
> random times. I ended up making several config changes to the stats
> collector and then we also restart it after every deploy and that solved it
> (so far).
>
> I'd say these were my main reasons for not going to 3.6 for our control
> nodes:
>
> In 3.6.x they re-wrote the stats processor to make it parallel. In every 3.6
> release since then, Pivotal has fixed bugs in this code. Then finally they
> threw up their hands and said "we're going to make a complete rewrite in
> 3.7/4.x" (you need to look through issues on Github to find this discussion)
> Out of the box with the same configs 3.6.5 used more memory than 3.5.6,
> since this was our main issue, I consider this a negative.
> Another issue is the ancient version of erlang we have with Ubuntu Trusty
> (which we are working on) which made upgrades more complex/impossible
> depending on the version.
>
> Given those negatives, the main one being that I didn't think there would be
> too many more fixes to the parallel statsdb collector in 3.6, we decided to
> stick with 3.5.6. In the end the devil we know is better than the devil we
> don't and I had no evidence that 3.6.5 would be an improvement.
>
> I did decide to leave Trove on 3.6.5 because this would give us some bake-in
> time if 3.5.x became untenable we'd at least have had it up and running in
> production and some data on it.
>
> If statsdb is not a concern for you, I think this changes the math and maybe
> you should use 3.6.x. I would however recommend at least going to 3.5.6,
> it's been better than 3.3/3.4 was.
>
> No matter what you do definitely read all the release notes. There are some
> upgrades which require an entire cluster shutdown. The upgrade to 3.5.6 did
> not require this IIRC.
>
> Here's the hiera for our rabbit settings which I assume you can translate:
>
> rabbitmq::cluster_partition_handling: 'autoheal'
> rabbitmq::config_variables:
>   'vm_memory_high_watermark': '0.6'
>   'collect_statistics_interval': 30000
> rabbitmq::config_management_variables:
>   'rates_mode': 'none'
> rabbitmq::file_limit: '65535'
>
> Finally, if you do upgrade to 3.6.x please report back here with your
> results at scale!
>
>
> On Thu, Jan 5, 2017 at 8:49 AM, Mike Dorman <mdorman at godaddy.com> wrote:
>>
>> We are looking at upgrading to the latest RabbitMQ in an effort to ease
>> some cluster failover issues we’ve been seeing.  (Currently on 3.4.0)
>>
>>
>>
>> Anyone been running 3.6.x?  And what has been your experience?  Any
>> gottchas to watch out for?
>>
>>
>>
>> Thanks,
>>
>> Mike
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>