<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 9, 2014 at 11:18 PM, Erich Weiler <span dir="ltr"><<a href="mailto:weiler@soe.ucsc.edu" target="_blank">weiler@soe.ucsc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":1wi" class="a3s" style="overflow:hidden">The cinder logs (all types) show nothing. The only evidence of strangeness I see in any logs are in the nova compute log on the server that the instance I'm trying to attach to is running on. I see this:<br>
<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 266, in decorated_function<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 309, in decorated_function<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/openstack/<u></u>common/excutils.py", line 68, in __exit__<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 296, in decorated_function<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 4130, in attach_volume<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher bdm.destroy(context)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/openstack/<u></u>common/excutils.py", line 68, in __exit__<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 4127, in attach_volume<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher return self._attach_volume(context, instance, driver_bdm)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 4148, in _attach_volume<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher self.volume_api.unreserve_<u></u>volume(context, bdm.volume_id)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/volume/cinder.<u></u>py", line 173, in wrapper<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher res = method(self, ctx, volume_id, *args, **kwargs)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/nova/volume/cinder.<u></u>py", line 249, in unreserve_volume<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher cinderclient(context).volumes.<u></u>unreserve(volume_id)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/v1/<u></u>volumes.py", line 293, in unreserve<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher return self._action('os-unreserve', volume)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/v1/<u></u>volumes.py", line 250, in _action<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher return self.api.client.post(url, body=body)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/client.<u></u>py", line 210, in post<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher return self._cs_request(url, 'POST', **kwargs)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/client.<u></u>py", line 199, in _cs_request<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher raise exceptions.ConnectionError(<u></u>msg)<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher ConnectionError: Unable to establish connection: [Errno 101] ENETUNREACH<br>
2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher<br>
2014-04-09 22:08:29.998 6547 ERROR oslo.messaging._drivers.common [-] Returning exception Unable to establish connection: [Errno 101] ENETUNREACH to caller<br>
2014-04-09 22:08:29.998 6547 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-<u></u>packages/oslo/messaging/rpc/<u></u>dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.6/site-<u></u>packages/oslo/messaging/rpc/<u></u>dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/oslo/messaging/rpc/<u></u>dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/openstack/<u></u>common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 280, in decorated_function\n pass\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/openstack/<u></u>common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 266, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 309, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/openstack/<u></u>common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 296, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 4130, in attach_volume\n bdm.destroy(context)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/openstack/<u></u>common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 4127, in attach_volume\n return self._attach_volume(context, instance, driver_bdm)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/compute/manager.<u></u>py", line 4148, in _attach_volume\n self.volume_api.unreserve_<u></u>volume(context, bdm.volume_id)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/volume/cinder.<u></u>py", line 173, in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/nova/volume/cinder.<u></u>py", line 249, in unreserve_volume\n cinderclient(context).volumes.<u></u>unreserve(volume_id)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/v1/<u></u>volumes.py", line 293, in unreserve\n return self._action(\'os-unreserve\', volume)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/v1/<u></u>volumes.py", line 250, in _action\n return self.api.client.post(url, body=body)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/client.<u></u>py", line 210, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/usr/lib/python2.6/site-<u></u>packages/cinderclient/client.<u></u>py", line 199, in _cs_request\n raise exceptions.ConnectionError(<u></u>msg)\n', 'ConnectionError: Unable to establish connection: [Errno 101] ENETUNREACH\n']<br>
<br>
So clearly nova is having trouble attaching the volume, and I see several "Unable to establish connection" messages. Does that mean it can't connect to the iSCSI port on the cinder server? No firewall there. Or something else it can't connect to?</div>
</blockquote></div><br><div class="gmail_default" style="font-family:'courier new',monospace">Based on what you have here it actually looks like you're not even getting to the point of setting up the iSCSI piece and logging in to the target to open a session.</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">The ENETUNREACH error is actually coming from cinderclient, it's not able to get the request issued out to the cinder API server is what it *looks* like. It would appear you've successfully used cinderclient from *somewhere* else to create the volume that you're trying to attach. Maybe verify that you can hit the Cinder API address from the compute node that's trying to do the attach? I believe you might want to do a "keystone endpoint-list" as an admin, get the URL for the Cinder endpoints and make sure you can hit it from the compute node your instance is on.</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">Thanks,</div><div class="gmail_default" style="font-family:'courier new',monospace">
John</div><br></div></div>