Open Stack

Hello everyone.

I'm testing Icehouse on Ubuntu 14.04 nodes, and seems like it works ok,
except for one thing - cinder randomly gives me unpleasant surprises which
from a perspective of nova appears as:

...
Max retries exceeded with url:
/v1/a5286f8e7e2440ab9e8fcc120d59b872/volumes/fc10ef37-43ce-4b49-a943-f9cda7f97fac/action
(Caused by <class 'httplib.BadStatusLine'>: '')
...

at approximately the same time I see following stuff in cinder-volume.log:

2014-05-20 13:08:15.139 16400 ERROR oslo.messaging._drivers.impl_rabbit [-]
Failed to consume message from queue: Socket closed
...
stacktrace goes here
...
2014-05-20 13:08:15.139 16400 TRACE oslo.messaging._drivers.impl_rabbit
IOError: Socket closed
2014-05-20 13:08:15.139 16400 TRACE oslo.messaging._drivers.impl_rabbit
2014-05-20 13:08:15.141 16400 INFO oslo.messaging._drivers.impl_rabbit [-]
Reconnecting to AMQP server on 127.0.0.1:5672
2014-05-20 13:08:15.142 16400 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2014-05-20 13:08:16.184 16400 INFO oslo.messaging._drivers.impl_rabbit [-]
Connected to AMQP server on 127.0.0.1:5672

That's the only suspicious thing I see in logs.

127.0.0.1:5672 is HAProxy balancing between 2 controllers with rabbitmq.
HAProxy config for Rabbit:

listen rabbitmq-cluster
bind 127.0.0.1:5672
        mode    tcp
option  tcpka
 balance roundrobin
        server controller01 10.1.2.231:5672 check inter 5000 rise 2 fall 3
        server controller02 10.1.2.232:5672 check inter 5000 rise 2 fall 3

I suppose that the reason might be HAProxy dropping rabbitmq connections
due to timeouts or something, but I have client and server timeouts set
much higher then the interval when separate stacktraces appear in log.

Have anyone faced these troubles? Any suggestions regarding optimization of
haproxy conf or something?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140520/f8346cc4/attachment.html>