<div dir="ltr">Hello guys,<div><br></div><div> Last saturday I needed to turn off my controller node because it was needed to perform a energy maintenance at my lab. After turning on the controller again I could fix the network problems, but I can't fix the cinder one, I'm using Grizzly at Ubuntu Server 12.04 and when I try to attach my storage I got this error:</div>
<div><br></div><div><div>cinder-api.log:</div><div><br></div><div>2013-11-05 11:23:49 INFO [cinder.api.middleware.fault] <a href="http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action">http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action</a> returned with HTTP 500<br>
</div><div>2013-11-05 11:23:53 INFO [cinder.api.openstack.wsgi] POST <a href="http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action">http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action</a></div>
<div>2013-11-05 11:24:29 INFO [cinder.api.openstack.wsgi] GET <a href="http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/detail">http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/detail</a></div>
<div>2013-11-05 11:24:29 AUDIT [cinder.api.v1.volumes] vol=<cinder.db.sqlalchemy.models.Volume object at 0x38a2250></div><div>2013-11-05 11:24:29 AUDIT [cinder.api.v1.volumes] vol=<cinder.db.sqlalchemy.models.Volume object at 0x373f5d0></div>
<div>2013-11-05 11:24:29 AUDIT [cinder.api.v1.volumes] vol=<cinder.db.sqlalchemy.models.Volume object at 0x37a9890></div><div>2013-11-05 11:24:29 INFO [cinder.api.openstack.wsgi] <a href="http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/detail">http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/detail</a> returned with HTTP 200</div>
<div>2013-11-05 11:24:53 ERROR [cinder.api.middleware.fault] Caught error: Timeout while waiting on RPC response.</div><div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line 73, in __call__</div>
<div> return req.get_response(self.application)</div><div> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send</div><div> application, catch_exc_info=False)</div><div> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application</div>
<div> app_iter = application(self.environ, start_response)</div><div> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div><div> return resp(environ, start_response)</div><div>
File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__</div><div> return self.app(env, start_response)</div><div> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div>
<div> return resp(environ, start_response)</div><div> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div><div> return resp(environ, start_response)</div><div> File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__</div>
<div> response = self.app(environ, start_response)</div><div> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div><div> return resp(environ, start_response)</div><div> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__</div>
<div> resp = self.call_func(req, *args, **self.kwargs)</div><div> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func</div><div> return self.func(req, *args, **kwargs)</div><div>
File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 803, in __call__</div>
<div> content_type, body, accept)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 851, in _process_stack</div><div> action_result = self.dispatch(meth, request, action_args)</div>
<div> File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 927, in dispatch</div><div> return method(req=request, **action_args)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/api/contrib/volume_actions.py", line 137, in _initialize_connection</div>
<div> connector)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 63, in wrapped</div><div> return func(self, context, target_obj, *args, **kwargs)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 493, in initialize_connection</div>
<div> connector)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/volume/rpcapi.py", line 117, in initialize_connection</div><div> volume['host']))</div><div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/proxy.py", line 80, in call</div>
<div> return rpc.call(context, self._get_topic(topic), msg, timeout)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/__init__.py", line 140, in call</div><div> return _get_impl().call(CONF, context, topic, msg, timeout)</div>
<div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/impl_kombu.py", line 798, in call</div><div> rpc_amqp.get_connection_pool(conf, Connection))</div><div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 613, in call</div>
<div> rv = list(rv)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 555, in __iter__</div><div> self.done()</div><div> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div> self.gen.next()</div><div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 552, in __iter__</div><div> self._iterator.next()</div><div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume</div>
<div> yield self.ensure(_error_callback, _consume)</div><div> File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/impl_kombu.py", line 566, in ensure</div><div> error_callback(e)</div><div>
File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback</div><div> raise rpc_common.Timeout()</div><div>Timeout: Timeout while waiting on RPC response.</div>
<div>2013-11-05 11:24:53 INFO [cinder.api.middleware.fault] <a href="http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action">http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action</a> returned with HTTP 500</div>
<div>2013-11-05 11:24:53 INFO [cinder.api.openstack.wsgi] POST <a href="http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action">http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action</a></div>
<div>2013-11-05 11:24:53 INFO [cinder.api.openstack.wsgi] <a href="http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action">http://192.168.3.1:8776/v1/d13839320f5d4194a4a3fe3b723d6144/volumes/954d2f1b-837b-4ba5-abfd-b3610597be5e/action</a> returned with HTTP 202</div>
</div><div><br></div><div>My cinder-volume.log and cinder-scheduler.log are empty. The attachment was working before rebooting the controller node.</div><div><br></div><div>Can anybody please help me to fix it?</div><div>
<br></div><div>Thank you all.</div><div><br></div><div>Guilherme.</div></div>