[Openstack] Cinder issues...
Erich Weiler
weiler at soe.ucsc.edu
Tue Apr 8 22:27:44 UTC 2014
Hey Y'all,
Thanks a bunch for all the help so far by the way - I'm nearly done with
standing up my POC OpenStack system.
This is Icehouse RDO on RedHat BTW.
I'm hvaing this odd thing happening with cinder. In horizon, I can see
the cinder volume storage and even create a volume, no problem. I try
to attach the storage to a VM and it sits in "Attaching..." in Horizon,
and never seems to attach. I can also not delete it using horizon or
the command line:
# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID | Status | Display Name |
Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| e34524ea-cd2f-41e8-b37e-ac15456275d7 | attaching | test-volume-1 | 10
| None | false | |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
# cinder delete e34524ea-cd2f-41e8-b37e-ac15456275d7
Delete for volume e34524ea-cd2f-41e8-b37e-ac15456275d7 failed: Invalid
volume: Volume status must be available or error, but current status is:
attaching (HTTP 400) (Request-ID: req-1d2351e0-48e8-4b23-8c9b-d6f2d0b21718)
ERROR: Unable to delete any of the specified volumes.
On the compute node that has the VM, I see this in the nova logs:
2014-04-08 14:42:12.440 14756 WARNING nova.network.neutronv2 [-] Using
neutron_admin_tenant_name for authentication is deprecated and will be
removed in the next release. Use neutron_admin_tenant_id instead.
2014-04-08 14:42:17.167 14756 WARNING nova.virt.disk.vfs.guestfs
[req-8c668c6c-9184-4f31-891d-93b7ddb1c25b
f8fdf7f84ad34c439c4075b5e3720211 f7e61747885045d8b266a161310c0094]
Failed to close augeas aug_close: do_aug_close: you must call 'aug-init'
first to initialize Augeas
2014-04-08 14:42:44.971 14756 ERROR nova.compute.manager
[req-59b34801-a0b8-4113-987f-95b134f1e181
f8fdf7f84ad34c439c4075b5e3720211 f7e61747885045d8b266a161310c0094]
[instance: 3aef6998-0aa4-40f5-a5be-469edefe98fc] Failed to attach
e34524ea-cd2f-41e8-b37e-ac15456275d7 at /dev/vdb
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] Traceback (most recent call last):
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3886,
in _attach_volume
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] do_check_attach=False,
do_driver_attach=True)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 44,
in wrapped
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] ret_val = method(obj, context,
*args, **kwargs)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 215,
in attach
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] volume =
volume_api.get(context, self.volume_id)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 174, in
wrapper
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] res = method(self, ctx,
volume_id, *args, **kwargs)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 207, in get
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] item =
cinderclient(context).volumes.get(volume_id)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 196,
in get
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] return self._get("/volumes/%s"
% volume_id, "volume")
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/cinderclient/base.py", line 145, in _get
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] resp, body =
self.api.client.get(url)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 207, in get
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] return self._cs_request(url,
'GET', **kwargs)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 199, in
_cs_request
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] raise
exceptions.ConnectionError(msg)
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc] ConnectionError: Unable to
establish connection: [Errno 101] ENETUNREACH
2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
3aef6998-0aa4-40f5-a5be-469edefe98fc]
2014-04-08 14:42:44.977 14756 ERROR root [-] Original exception being
dropped: ['Traceback (most recent call last):\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3886,
in _attach_volume\n do_check_attach=False, do_driver_attach=True)\n',
' File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py",
line 44, in wrapped\n ret_val = method(obj, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 215,
in attach\n volume = volume_api.get(context, self.volume_id)\n', '
File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 174,
in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n',
' File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line
207, in get\n item = cinderclient(context).volumes.get(volume_id)\n',
' File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py",
line 196, in get\n return self._get("/volumes/%s" % volume_id,
"volume")\n', ' File
"/usr/lib/python2.6/site-packages/cinderclient/base.py", line 145, in
_get\n resp, body = self.api.client.get(url)\n', ' File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 207, in
get\n return self._cs_request(url, \'GET\', **kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 199, in
_cs_request\n raise exceptions.ConnectionError(msg)\n',
'ConnectionError: Unable to establish connection: [Errno 101]
ENETUNREACH\n']
2014-04-08 14:42:45.130 14756 ERROR oslo.messaging.rpc.dispatcher [-]
Exception during message handling: Unable to establish connection:
[Errno 101] ENETUNREACH
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
Traceback (most recent call last):
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 133, in _dispatch_and_reply
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 176, in _dispatch
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 122, in _do_dispatch
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 360, in
decorated_function
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
return function(self, context, *args, **kwargs)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
payload)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
return f(self, context, *args, **kw)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 244, in
decorated_function
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher pass
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 230, in
decorated_function
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
return function(self, context, *args, **kwargs)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 272, in
decorated_function
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 259, in
decorated_function
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
return function(self, context, *args, **kwargs)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3876,
in attach_volume
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
bdm.destroy(context)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210, in post
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
return self._cs_request(url, 'POST', **kwargs)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 199, in
_cs_request
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
raise exceptions.ConnectionError(msg)
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
ConnectionError: Unable to establish connection: [Errno 101] ENETUNREACH
2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
2014-04-08 14:42:45.133 14756 ERROR oslo.messaging._drivers.common [-]
Returning exception Unable to establish connection: [Errno 101]
ENETUNREACH to caller
2014-04-08 14:42:45.134 14756 ERROR oslo.messaging._drivers.common [-]
['Traceback (most recent call last):\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 133, in _dispatch_and_reply\n incoming.message))\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 176, in _dispatch\n return self._do_dispatch(endpoint, method,
ctxt, args)\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
**new_args)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 360, in
decorated_function\n return function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in
wrapped\n payload)\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in
wrapped\n return f(self, context, *args, **kw)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 244, in
decorated_function\n pass\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 230, in
decorated_function\n return function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 272, in
decorated_function\n e, sys.exc_info())\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 259, in
decorated_function\n return function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3876,
in attach_volume\n bdm.destroy(context)\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3873,
in attach_volume\n return self._attach_volume(context, instance,
driver_bdm)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3894,
in _attach_volume\n self.volume_api.unreserve_volume(context,
bdm.volume_id)\n', ' File
"/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 174, in
wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', '
File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 250,
in unreserve_volume\n
cinderclient(context).volumes.unreserve(volume_id)\n', ' File
"/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 293,
in unreserve\n return self._action(\'os-unreserve\', volume)\n', '
File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line
250, in _action\n return self.api.client.post(url, body=body)\n', '
File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line
210, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', '
File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line
199, in _cs_request\n raise exceptions.ConnectionError(msg)\n',
'ConnectionError: Unable to establish connection: [Errno 101]
ENETUNREACH\n']
I see several messages in there about "Unable to attach /dev/vdb" and
also some "unable to establish connection". Any ideas where I went wrong?
Also, is there a way I can delete the volume that is in "attaching"
status? I terminated the VM successfully, but the volume still sits in
"attaching" status...
Hopefully I'm missing something basic... ;)
Thanks for any ideas!!
cheers,
erich
More information about the Openstack
mailing list