[Openstack] Cinder issues...
Remo Mattei
remo at italy1.com
Tue Apr 8 23:28:04 UTC 2014
I just found it
force-delete
> Arg, I think I see the issue... I created my cinder endpoints to point to thew wrong internal hosts. I fixed the endpoints, but the volume still sits in "Attaching" state, so I can't touch it.
>
> Should I manually tweak mysql to fix this? I see:
>
> mysql> use cinder;
>
> mysql> select * from volumes\G
> *************************** 1. row ***************************
> created_at: 2014-04-08 21:36:17
> updated_at: 2014-04-08 21:36:17
> deleted_at: 2014-04-08 21:37:52
> deleted: 1
> id: 756ca5f0-bcdf-40e9-a62c-26a648be753f
> ec2_id: NULL
> user_id: f8fdf7f84ad34c439c4075b5e3720211
> project_id: f7e61747885045d8b266a161310c0094
> host: NULL
> size: 10
> availability_zone: nova
> instance_uuid: NULL
> mountpoint: NULL
> attach_time: NULL
> status: deleted
> attach_status: detached
> scheduled_at: NULL
> launched_at: NULL
> terminated_at: NULL
> display_name: test-volume-1
> display_description: Just testing yo.
> provider_location: NULL
> provider_auth: NULL
> snapshot_id: NULL
> volume_type_id: NULL
> source_volid: NULL
> bootable: 0
> attached_host: NULL
> provider_geometry: NULL
> _name_id: NULL
> encryption_key_id: NULL
> migration_status: NULL
> *************************** 2. row ***************************
> created_at: 2014-04-08 21:38:52
> updated_at: 2014-04-08 21:42:44
> deleted_at: NULL
> deleted: 0
> id: e34524ea-cd2f-41e8-b37e-ac15456275d7
> ec2_id: NULL
> user_id: f8fdf7f84ad34c439c4075b5e3720211
> project_id: f7e61747885045d8b266a161310c0094
> host: genome-cloudstore
> size: 10
> availability_zone: nova
> instance_uuid: NULL
> mountpoint: NULL
> attach_time: NULL
> status: attaching
> attach_status: detached
> scheduled_at: 2014-04-08 21:38:52
> launched_at: 2014-04-08 21:38:53
> terminated_at: NULL
> display_name: test-volume-1
> display_description: Just testing.
> provider_location: NULL
> provider_auth: NULL
> snapshot_id: NULL
> volume_type_id: NULL
> source_volid: NULL
> bootable: 0
> attached_host: NULL
> provider_geometry: NULL
> _name_id: NULL
> encryption_key_id: NULL
> migration_status: NULL
> 2 rows in set (0.00 sec)
>
> I can change the value of 'status' from 'attaching' to 'error'... In theory I can delete it then. Any reason why I shouldn't do that?
>
> Thanks,
> -erich
>
>
> On 04/08/14 15:27, Erich Weiler wrote:
>> Hey Y'all,
>>
>> Thanks a bunch for all the help so far by the way - I'm nearly done with
>> standing up my POC OpenStack system.
>>
>> This is Icehouse RDO on RedHat BTW.
>>
>> I'm hvaing this odd thing happening with cinder. In horizon, I can see
>> the cinder volume storage and even create a volume, no problem. I try
>> to attach the storage to a VM and it sits in "Attaching..." in Horizon,
>> and never seems to attach. I can also not delete it using horizon or
>> the command line:
>>
>> # cinder list
>> +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
>>
>> | ID | Status | Display Name |
>> Size | Volume Type | Bootable | Attached to |
>> +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
>>
>> | e34524ea-cd2f-41e8-b37e-ac15456275d7 | attaching | test-volume-1 | 10
>> | None | false | |
>> +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
>>
>>
>> # cinder delete e34524ea-cd2f-41e8-b37e-ac15456275d7
>> Delete for volume e34524ea-cd2f-41e8-b37e-ac15456275d7 failed: Invalid
>> volume: Volume status must be available or error, but current status is:
>> attaching (HTTP 400) (Request-ID: req-1d2351e0-48e8-4b23-8c9b-d6f2d0b21718)
>> ERROR: Unable to delete any of the specified volumes.
>>
>> On the compute node that has the VM, I see this in the nova logs:
>>
>> 2014-04-08 14:42:12.440 14756 WARNING nova.network.neutronv2 [-] Using
>> neutron_admin_tenant_name for authentication is deprecated and will be
>> removed in the next release. Use neutron_admin_tenant_id instead.
>> 2014-04-08 14:42:17.167 14756 WARNING nova.virt.disk.vfs.guestfs
>> [req-8c668c6c-9184-4f31-891d-93b7ddb1c25b
>> f8fdf7f84ad34c439c4075b5e3720211 f7e61747885045d8b266a161310c0094]
>> Failed to close augeas aug_close: do_aug_close: you must call 'aug-init'
>> first to initialize Augeas
>> 2014-04-08 14:42:44.971 14756 ERROR nova.compute.manager
>> [req-59b34801-a0b8-4113-987f-95b134f1e181
>> f8fdf7f84ad34c439c4075b5e3720211 f7e61747885045d8b266a161310c0094]
>> [instance: 3aef6998-0aa4-40f5-a5be-469edefe98fc] Failed to attach
>> e34524ea-cd2f-41e8-b37e-ac15456275d7 at /dev/vdb
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] Traceback (most recent call last):
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3886,
>> in _attach_volume
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] do_check_attach=False,
>> do_driver_attach=True)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 44,
>> in wrapped
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] ret_val = method(obj, context,
>> *args, **kwargs)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 215,
>> in attach
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] volume =
>> volume_api.get(context, self.volume_id)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 174, in
>> wrapper
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] res = method(self, ctx,
>> volume_id, *args, **kwargs)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 207, in get
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] item =
>> cinderclient(context).volumes.get(volume_id)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 196,
>> in get
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] return self._get("/volumes/%s"
>> % volume_id, "volume")
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/cinderclient/base.py", line 145, in _get
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] resp, body =
>> self.api.client.get(url)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 207, in get
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] return self._cs_request(url,
>> 'GET', **kwargs)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] File
>> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 199, in
>> _cs_request
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] raise
>> exceptions.ConnectionError(msg)
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc] ConnectionError: Unable to
>> establish connection: [Errno 101] ENETUNREACH
>> 2014-04-08 14:42:44.971 14756 TRACE nova.compute.manager [instance:
>> 3aef6998-0aa4-40f5-a5be-469edefe98fc]
>> 2014-04-08 14:42:44.977 14756 ERROR root [-] Original exception being
>> dropped: ['Traceback (most recent call last):\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3886,
>> in _attach_volume\n do_check_attach=False, do_driver_attach=True)\n',
>> ' File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py",
>> line 44, in wrapped\n ret_val = method(obj, context, *args,
>> **kwargs)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 215,
>> in attach\n volume = volume_api.get(context, self.volume_id)\n', '
>> File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 174,
>> in wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n',
>> ' File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line
>> 207, in get\n item = cinderclient(context).volumes.get(volume_id)\n',
>> ' File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py",
>> line 196, in get\n return self._get("/volumes/%s" % volume_id,
>> "volume")\n', ' File
>> "/usr/lib/python2.6/site-packages/cinderclient/base.py", line 145, in
>> _get\n resp, body = self.api.client.get(url)\n', ' File
>> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 207, in
>> get\n return self._cs_request(url, \'GET\', **kwargs)\n', ' File
>> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 199, in
>> _cs_request\n raise exceptions.ConnectionError(msg)\n',
>> 'ConnectionError: Unable to establish connection: [Errno 101]
>> ENETUNREACH\n']
>> 2014-04-08 14:42:45.130 14756 ERROR oslo.messaging.rpc.dispatcher [-]
>> Exception during message handling: Unable to establish connection:
>> [Errno 101] ENETUNREACH
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> Traceback (most recent call last):
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
>> line 133, in _dispatch_and_reply
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> incoming.message))
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
>> line 176, in _dispatch
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher return
>> self._do_dispatch(endpoint, method, ctxt, args)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
>> line 122, in _do_dispatch
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher result
>> = getattr(endpoint, method)(ctxt, **new_args)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 360, in
>> decorated_function
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher return
>> function(self, context, *args, **kwargs)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher payload)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
>> line 68, in __exit__
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> six.reraise(self.type_, self.value, self.tb)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher return
>> f(self, context, *args, **kw)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 244, in
>> decorated_function
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher pass
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
>> line 68, in __exit__
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> six.reraise(self.type_, self.value, self.tb)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 230, in
>> decorated_function
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher return
>> function(self, context, *args, **kwargs)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 272, in
>> decorated_function
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher e,
>> sys.exc_info())
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
>> line 68, in __exit__
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> six.reraise(self.type_, self.value, self.tb)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 259, in
>> decorated_function
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher return
>> function(self, context, *args, **kwargs)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3876,
>> in attach_volume
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> bdm.destroy(context)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210, in
>> post
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher return
>> self._cs_request(url, 'POST', **kwargs)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher File
>> "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 199, in
>> _cs_request
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher raise
>> exceptions.ConnectionError(msg)
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> ConnectionError: Unable to establish connection: [Errno 101] ENETUNREACH
>> 2014-04-08 14:42:45.130 14756 TRACE oslo.messaging.rpc.dispatcher
>> 2014-04-08 14:42:45.133 14756 ERROR oslo.messaging._drivers.common [-]
>> Returning exception Unable to establish connection: [Errno 101]
>> ENETUNREACH to caller
>> 2014-04-08 14:42:45.134 14756 ERROR oslo.messaging._drivers.common [-]
>> ['Traceback (most recent call last):\n', ' File
>> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
>> line 133, in _dispatch_and_reply\n incoming.message))\n', ' File
>> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
>> line 176, in _dispatch\n return self._do_dispatch(endpoint, method,
>> ctxt, args)\n', ' File
>> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
>> line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
>> **new_args)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 360, in
>> decorated_function\n return function(self, context, *args,
>> **kwargs)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in
>> wrapped\n payload)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
>> line 68, in __exit__\n six.reraise(self.type_, self.value,
>> self.tb)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in
>> wrapped\n return f(self, context, *args, **kw)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 244, in
>> decorated_function\n pass\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
>> line 68, in __exit__\n six.reraise(self.type_, self.value,
>> self.tb)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 230, in
>> decorated_function\n return function(self, context, *args,
>> **kwargs)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 272, in
>> decorated_function\n e, sys.exc_info())\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
>> line 68, in __exit__\n six.reraise(self.type_, self.value,
>> self.tb)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 259, in
>> decorated_function\n return function(self, context, *args,
>> **kwargs)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3876,
>> in attach_volume\n bdm.destroy(context)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
>> line 68, in __exit__\n six.reraise(self.type_, self.value,
>> self.tb)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3873,
>> in attach_volume\n return self._attach_volume(context, instance,
>> driver_bdm)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3894,
>> in _attach_volume\n self.volume_api.unreserve_volume(context,
>> bdm.volume_id)\n', ' File
>> "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 174, in
>> wrapper\n res = method(self, ctx, volume_id, *args, **kwargs)\n', '
>> File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 250,
>> in unreserve_volume\n
>> cinderclient(context).volumes.unreserve(volume_id)\n', ' File
>> "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 293,
>> in unreserve\n return self._action(\'os-unreserve\', volume)\n', '
>> File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line
>> 250, in _action\n return self.api.client.post(url, body=body)\n', '
>> File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line
>> 210, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', '
>> File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line
>> 199, in _cs_request\n raise exceptions.ConnectionError(msg)\n',
>> 'ConnectionError: Unable to establish connection: [Errno 101]
>> ENETUNREACH\n']
>>
>> I see several messages in there about "Unable to attach /dev/vdb" and
>> also some "unable to establish connection". Any ideas where I went wrong?
>>
>> Also, is there a way I can delete the volume that is in "attaching"
>> status? I terminated the VM successfully, but the volume still sits in
>> "attaching" status...
>>
>> Hopefully I'm missing something basic... ;)
>>
>> Thanks for any ideas!!
>>
>> cheers,
>> erich
>>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
> !DSPAM:1,5344824b244361176810642!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140408/21b7af86/attachment.html>
More information about the Openstack
mailing list