Open Stack

Ha, right you are, my man.  I had a typo in my internal URL endpoint for 
cinder.  After fixing that, and also specifying:

"cinder_catalog_info=volume:cinder:internalURL"

in my nova.conf on my compute nodes, it started working.  Thanks a bunch!

-erich

On 04/09/14 23:22, John Griffith wrote:
>
> On Wed, Apr 9, 2014 at 11:18 PM, Erich Weiler <weiler at soe.ucsc.edu
> <mailto:weiler at soe.ucsc.edu>> wrote:
>
>     The cinder logs (all types) show nothing.  The only evidence of
>     strangeness I see in any logs are in the nova compute log on the
>     server that the instance I'm trying to attach to is running on.  I
>     see this:
>
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py",
>     line 266, in decorated_function
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     return function(self, context, *args, **kwargs)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py",
>     line 309, in decorated_function
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     e, sys.exc_info())
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File
>     "/usr/lib/python2.6/site-__packages/nova/openstack/__common/excutils.py",
>     line 68, in __exit__
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     six.reraise(self.type_, self.value, self.tb)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py",
>     line 296, in decorated_function
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     return function(self, context, *args, **kwargs)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py",
>     line 4130, in attach_volume
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     bdm.destroy(context)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File
>     "/usr/lib/python2.6/site-__packages/nova/openstack/__common/excutils.py",
>     line 68, in __exit__
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     six.reraise(self.type_, self.value, self.tb)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py",
>     line 4127, in attach_volume
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     return self._attach_volume(context, instance, driver_bdm)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py",
>     line 4148, in _attach_volume
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     self.volume_api.unreserve___volume(context, bdm.volume_id)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/volume/cinder.__py",
>     line 173, in wrapper
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     res = method(self, ctx, volume_id, *args, **kwargs)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/nova/volume/cinder.__py",
>     line 249, in unreserve_volume
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     cinderclient(context).volumes.__unreserve(volume_id)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File
>     "/usr/lib/python2.6/site-__packages/cinderclient/v1/__volumes.py",
>     line 293, in unreserve
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     return self._action('os-unreserve', volume)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File
>     "/usr/lib/python2.6/site-__packages/cinderclient/v1/__volumes.py",
>     line 250, in _action
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     return self.api.client.post(url, body=body)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/cinderclient/client.__py",
>     line 210, in post
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     return self._cs_request(url, 'POST', **kwargs)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     File "/usr/lib/python2.6/site-__packages/cinderclient/client.__py",
>     line 199, in _cs_request
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     raise exceptions.ConnectionError(__msg)
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     ConnectionError: Unable to establish connection: [Errno 101] ENETUNREACH
>     2014-04-09 22:08:29.995 6547 TRACE oslo.messaging.rpc.dispatcher
>     2014-04-09 22:08:29.998 6547 ERROR oslo.messaging._drivers.common
>     [-] Returning exception Unable to establish connection: [Errno 101]
>     ENETUNREACH to caller
>     2014-04-09 22:08:29.998 6547 ERROR oslo.messaging._drivers.common
>     [-] ['Traceback (most recent call last):\n', '  File
>     "/usr/lib/python2.6/site-__packages/oslo/messaging/rpc/__dispatcher.py",
>     line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File
>     "/usr/lib/python2.6/site-__packages/oslo/messaging/rpc/__dispatcher.py",
>     line 176, in _dispatch\n    return self._do_dispatch(endpoint,
>     method, ctxt, args)\n', '  File
>     "/usr/lib/python2.6/site-__packages/oslo/messaging/rpc/__dispatcher.py",
>     line 122, in _do_dispatch\n    result = getattr(endpoint,
>     method)(ctxt, **new_args)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     399, in decorated_function\n    return function(self, context,
>     *args, **kwargs)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/exception.py", line 88, in
>     wrapped\n    payload)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/openstack/__common/excutils.py",
>     line 68, in __exit__\n    six.reraise(self.type_, self.value,
>     self.tb)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/exception.py", line 71, in
>     wrapped\n    return f(self, context, *args, **kw)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     280, in decorated_function\n    pass\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/openstack/__common/excutils.py",
>     line 68, in __exit__\n    six.reraise(self.type_, self.value,
>     self.tb)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     266, in decorated_function\n    return function(self, context,
>     *args, **kwargs)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     309, in decorated_function\n    e, sys.exc_info())\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/openstack/__common/excutils.py",
>     line 68, in __exit__\n    six.reraise(self.type_, self.value,
>     self.tb)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     296, in decorated_function\n    return function(self, context,
>     *args, **kwargs)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     4130, in attach_volume\n    bdm.destroy(context)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/openstack/__common/excutils.py",
>     line 68, in __exit__\n    six.reraise(self.type_, self.value,
>     self.tb)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     4127, in attach_volume\n    return self._attach_volume(context,
>     instance, driver_bdm)\n', '  File
>     "/usr/lib/python2.6/site-__packages/nova/compute/manager.__py", line
>     4148, in _attach_volume\n
>       self.volume_api.unreserve___volume(context, bdm.volume_id)\n', '
>       File "/usr/lib/python2.6/site-__packages/nova/volume/cinder.__py",
>     line 173, in wrapper\n    res = method(self, ctx, volume_id, *args,
>     **kwargs)\n', ' File
>     "/usr/lib/python2.6/site-__packages/nova/volume/cinder.__py", line
>     249, in unreserve_volume\n
>     cinderclient(context).volumes.__unreserve(volume_id)\n', '  File
>     "/usr/lib/python2.6/site-__packages/cinderclient/v1/__volumes.py",
>     line 293, in unreserve\n    return self._action(\'os-unreserve\',
>     volume)\n', ' File
>     "/usr/lib/python2.6/site-__packages/cinderclient/v1/__volumes.py",
>     line 250, in _action\n    return self.api.client.post(url,
>     body=body)\n', ' File
>     "/usr/lib/python2.6/site-__packages/cinderclient/client.__py", line
>     210, in post\n    return self._cs_request(url, \'POST\',
>     **kwargs)\n', '  File
>     "/usr/lib/python2.6/site-__packages/cinderclient/client.__py", line
>     199, in _cs_request\n    raise exceptions.ConnectionError(__msg)\n',
>     'ConnectionError: Unable to establish connection: [Errno 101]
>     ENETUNREACH\n']
>
>     So clearly nova is having trouble attaching the volume, and I see
>     several "Unable to establish connection" messages.  Does that mean
>     it can't connect to the iSCSI port on the cinder server?  No
>     firewall there.  Or something else it can't connect to?
>
>
> Based on what you have here it actually looks like you're not even
> getting to the point of setting up the iSCSI piece and logging in to the
> target to open a session.
>
> The ENETUNREACH error is actually coming from cinderclient, it's not
> able to get the request issued out to the cinder API server is what it
> *looks* like.  It would appear you've successfully used cinderclient
> from *somewhere* else to create the volume that you're trying to attach.
>   Maybe verify that you can hit the Cinder API address from the compute
> node that's trying to do the attach?  I believe you might want to do a
> "keystone endpoint-list" as an admin, get the URL for the Cinder
> endpoints and make sure you can hit it from the compute node your
> instance is on.
>
> Thanks,
> John
>