<div dir="ltr">Hi Alberto,<div><br></div><div>I have fix this issue already. The root cause is that the controller(where the cinder service located) block the 8776 port and the node can not access the cinder service. After enable the port on the IPtables, it works. </div>
<div><br></div><div style>thanks you. </div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Dec 18, 2012 at 2:00 PM, Lei Zhang <span dir="ltr"><<a href="mailto:zhang.lei.fly@gmail.com" target="_blank">zhang.lei.fly@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="line-height:1.2;color:rgb(34,34,34);font-size:1em;font-family:Helvetica,arial,freesans,clean,sans-serif;border:none">
<p style="margin:1em 0px">Hi Alberto,
</p>
<p style="margin:1em 0px">I found the error message from computer.log on the node. Did you meet this before?
</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;overflow:auto;margin:1em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:nowrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-top-left-radius:3px;border-top-right-radius:3px;border-bottom-right-radius:3px;border-bottom-left-radius:3px;display:inline;white-space:pre-wrap;border-top-left-radius:3px;border-top-right-radius:3px;border-bottom-right-radius:3px;border-bottom-left-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block;padding:0.5em;color:rgb(51,51,51);background-color:rgb(248,248,255);background-repeat:initial initial">2012-12-18 00:57:42 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'project_name': u'admin', 'user_id': u'9e0e905b3305477587d5f85889b43955', 'roles': [u'admin'], 'timestamp': u'2012-12-18T05:57:33.137372', 'auth_token': '<SANITIZED>', 'remote_address': u'172.16.0.10', 'quota_class': None, 'is_admin': True, 'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b" target="_blank">http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b</a>', u'region': u'RegionOne', u'id': u'b83ae23e3eca4411b726c754df1890b5', u'serviceName': u'cinder', u'internalURL': u'<a href="http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b" target="_blank">http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b</a>', u'publicURL': u'<a href="http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b'%7D" target="_blank">http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b'}</a>], u'type': u'volume', u'name': u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://172.16.0.10:9292/v1" target="_blank">http://172.16.0.10:9292/v1</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://172.16.0.10:9292/v1" target="_blank">http://172.16.0.10:9292/v1</a>', u'internalURL': u'<a href="http://172.16.0.10:9292/v1" target="_blank">http://172.16.0.10:9292/v1</a>', u'id': u'fd308ef11b07483db2948b4ec36ddd9c'}], u'type': u'image', u'name': u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://172.16.0.10:8774/v2/87f394d760354056a584021eb6b6214b" target="_blank">http://172.16.0.10:8774/v2/87f394d760354056a584021eb6b6214b</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://172.16.0.10:8774/v2/87f394d760354056a584021eb6b6214b" target="_blank">http://172.16.0.10:8774/v2/87f394d760354056a584021eb6b6214b</a>', u'internalURL': u'<a href="http://172.16.0.10:8774/v2/87f394d760354056a584021eb6b6214b" target="_blank">http://172.16.0.10:8774/v2/87f394d760354056a584021eb6b6214b</a>', u'id': u'7acff35322ab45e6b81be3361265ec54'}], u'type': u'compute', u'name': u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://172.16.0.10:8773/services/Admin" target="_blank">http://172.16.0.10:8773/services/Admin</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://172.16.0.10:8773/services/Cloud" target="_blank">http://172.16.0.10:8773/services/Cloud</a>', u'internalURL': u'<a href="http://172.16.0.10:8773/services/Cloud" target="_blank">http://172.16.0.10:8773/services/Cloud</a>', u'id': u'5081d098d06446bcb1218a12fd849c8c'}], u'type': u'ec2', u'name': u'nova_ec2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://172.16.0.10:35357/v2.0" target="_blank">http://172.16.0.10:35357/v2.0</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://172.16.0.10:5000/v2.0" target="_blank">http://172.16.0.10:5000/v2.0</a>', u'internalURL': u'<a href="http://172.16.0.10:5000/v2.0" target="_blank">http://172.16.0.10:5000/v2.0</a>', u'id': u'0d5b09341c954c1297735341106bb288'}], u'type': u'identity', u'name': u'keystone'}], 'request_id': u'req-dde3b5c5-57d0-410e-9845-246df9baed23', 'instance_lock_checked': False, 'project_id': u'87f394d760354056a584021eb6b6214b', 'user_name': u'admin', 'read_deleted': u'no'} from (pid=2076) _safe_log /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py:195
2012-12-18 00:57:42 DEBUG nova.volume.cinder [req-dde3b5c5-57d0-410e-9845-246df9baed23 9e0e905b3305477587d5f85889b43955 87f394d760354056a584021eb6b6214b] Cinderclient connection created using URL: <a href="http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b" target="_blank">http://172.16.0.10:8776/v1/87f394d760354056a584021eb6b6214b</a> from (pid=2076) cinderclient /usr/lib/python2.6/site-packages/nova/volume/cinder.py:68
2012-12-18 00:57:43 DEBUG nova.utils [req-dde3b5c5-57d0-410e-9845-246df9baed23 9e0e905b3305477587d5f85889b43955 87f394d760354056a584021eb6b6214b] Got semaphore "compute_resources" for method "update_usage"... from (pid=2076) inner /usr/lib/python2.6/site-packages/nova/utils.py:713
2012-12-18 00:57:43 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 181, in decorated_function
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp pass
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 167, in decorated_function
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 202, in decorated_function
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 196, in decorated_function
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1945, in attach_volume
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp context, instance.get('uuid'), mountpoint)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1941, in attach_volume
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp mountpoint, instance)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1948, in _attach_volume
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp volume = self.volume_api.get(context, volume_id)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 144, in get
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp item = cinderclient(context).volumes.get(volume_id)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 147, in get
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp return self._get("/volumes/%s" % volume_id, "volume")
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinderclient/base.py", line 141, in _get
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp resp, body = self.api.client.get(url)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 138, in get
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp return self._cs_request(url, 'GET', **kwargs)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 126, in _cs_request
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp **kwargs)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 109, in request
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp raise exceptions.from_response(resp, body)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp BadRequest: n/a (HTTP 400)
2012-12-18 00:57:43 TRACE nova.openstack.common.rpc.amqp</code></pre>
</div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Mon, Dec 17, 2012 at 8:40 PM, Alberto Molina Coballes <span dir="ltr"><<a href="mailto:alb.molina@gmail.com" target="_blank">alb.molina@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Lei,<br>
<br>
I was facing a similar issue with nova-volume on essex, so I'm not<br>
sure if this can be useful to you ...<br>
<br>
Where are you looking for errors? I found volume attach errors at<br>
nova-compute.log of node where instance is running<br>
<br>
I hope this help you<br>
<span><font color="#888888"><br>
Alberto<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div></div></div><div>-- <br><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse">Lei Zhang</span></div><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><br>
</span></div><div><font face="arial, sans-serif">Blog: <a href="http://jeffrey4l.github.com" target="_blank">http://jeffrey4l.github.com</a></font></div><div>twitter/weibo: @jeffrey4l</div><br>
</div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse">Lei Zhang</span></div><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><br>
</span></div><div><font face="arial, sans-serif">Blog: <a href="http://jeffrey4l.github.com" target="_blank">http://jeffrey4l.github.com</a></font></div><div>twitter/weibo: @jeffrey4l</div><br>
</div></div>