Open Stack

Hi Abel,

We were running Havana on Precise before. We moved to Icehouse on Trusty. So new kombu etc. too.
We also moved to new rabbit servers so all the queues were fresh etc.

Joe: Yeah we have metadata_workers set to 2 also, a little gotcha for the icehouse upgrade.

Cheers,
Sam



On 3 Sep 2014, at 6:28 am, Abel Lopez <alopgeek at gmail.com> wrote:

> What release were you running before Icehouse?
> I’m curious if you purged/deleted queues during the upgrade.
> Might be useful to start fresh with your rabbit, like completely trash your mensia during a maintenance window (obviously with your services stopped) so they recreate the queues at startup.
> Also, was kombu upgraded along with your openstack release?
> 
> On Aug 25, 2014, at 4:17 PM, Sam Morrison <sorrison at gmail.com> wrote:
> 
>> Hi,
>> 
>> Since upgrading to Icehouse we have seen increased issues with messaging relating to RabbitMQ.
>> 
>> 1. We often get reply_xxxxxx queues starting to fill up with unacked messages. To fix this we need to restart the offending service. Usually nova-api or nova-compute.
>> 
>> 2. If you kill a node so as to force an *ungraceful* disconnect of rabbit the connection “object?” still sticks around in rabbit. Starting the service again means there are now 2 consumers. The new one and the phantom old one. This then leads to messages piling up in the unacked queue. This feels like a rabbit bug to me but just thought I’d mention it here too.
>> 
>> 
>> We have have a setup that includes icehouse computes and havana computes in the same cloud and we only see this on the icehouse computes. This is using Trusty and RabbitMQ 3.3.4
>> 
>> 
>> Has anyone seen anything like this too?
>> 
>> Thanks,
>> Sam
>> 
>> 
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>