Open Stack

It seems that I as chasing down the wrong rabbit hole…

Our controller node runs on a VM and has had sever hiccups the last few days (pausing for up to 10 seconds from time to time). We have moved the VM to another host, the pauses disappeared, and I can delete as many VMs at a time as I like…

cheers
jc

-- 
SWITCH
Jens-Christian Fischer, Peta Solutions
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
jens-christian.fischer at switch.ch
http://www.switch.ch

http://www.switch.ch/stories

On 11.07.2014, at 15:45, Brebner, Gavin <gavin.brebner at hp.com> wrote:

>  
> we change /etc/security/limits.conf (actually add an entry to /etc/security/limits.conf.d) for the rabbit user.
>  
> We just do that for the non-compute nodes.
>  
>                 Gavin
>  
>  
> From: Jens-Christian Fischer [mailto:jens-christian.fischer at switch.ch] 
> Sent: Friday, July 11, 2014 3:41 PM
> To: Brebner, Gavin
> Cc: openstack at lists.openstack.org
> Subject: Re: [Openstack] Unable to delete instances
>  
> There is just one queue that is building up: notifications.info - there are no consumers for it.
>  
> I have upped the ulimit for the rabbitmq server (by changing /etc/default/rabbitmq-server) but that hasn’t made any difference.
>  
> Are you also upping the limits on the compute nodes? 
>  
> cheers
> jc
>  
> -- 
> SWITCH
> Jens-Christian Fischer, Peta Solutions
> Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
> phone +41 44 268 15 15, direct +41 44 268 15 71
> jens-christian.fischer at switch.ch
> http://www.switch.ch
> 
> http://www.switch.ch/stories
>  
> On 11.07.2014, at 12:50, Brebner, Gavin <gavin.brebner at hp.com> wrote:
> 
> 
> Delete issues in my experience are typically linked to messaging getting stuck (or dead hosts). I would have a look at the
> RMQ queue lengths and see if any are building up.
>  
> Have you adjusted your open file descriptor limits ? The default Linux limit (1024) is NOT enough in our experience for any reasonable
> sized system. We’ve seen that cause messaging outages.
>  
>                 Gavin
>  
> From: Jens-Christian Fischer [mailto:jens-christian.fischer at switch.ch] 
> Sent: Friday, July 11, 2014 12:32 PM
> To: Thomas Bernard
> Cc: openstack at lists.openstack.org
> Subject: Re: [Openstack] Unable to delete instances
>  
> Did you ever come to the bottom of this?
>  
> We are seeing the same thing with a brand new Icehouse / Ceph installation (using the stock distributions)
>  
> Restarting nova-compute on the compute nodes usually makes the instances go away (or resetting the state, and then restarting nova-compute)
>  
> We see this problem intermittent but frequent enough to be really annoying.
>  
> The log files have so far not shown anything conclusive: I got an “Unknown auth strategy” error, but according to https://bugs.launchpad.net/python-neutronclient/+bug/1297309 this is a red herring.
>  
> Anyone else running into this problem or having solved it?
>  
> cheers
> Jens-Christian
> -- 
> SWITCH
> Jens-Christian Fischer, Peta Solutions
> Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
> phone +41 44 268 15 15, direct +41 44 268 15 71
> jens-christian.fischer at switch.ch
> http://www.switch.ch
> 
> http://www.switch.ch/stories
>  
> On 22.05.2014, at 16:19, Thomas Bernard <tbe at atolcd.com> wrote:
> 
> 
> 
> Hi,
> 
> I just finished my upgrade to icehouse, everything works just fine but... i can't delete my instances!
> 
> When i delete instance (from cli or from horizon) : 
> 
> * The VM goes to "deleting"state
> * Vm are destroy from KVM
> * But nova list & horizon continue to display the "deleting" state
> 
> Some time after, the Vm state goes to "ERROR" and in nova-compute.log i can see : "Unauthorized: Unknown auth strategy\n" (from neutronclient)
> 
> Any idea?
> 
> Thx!
> 
> 
> 
> 
> 
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140714/2212e077/attachment.html>