Open Stack

Hello,

For the volume management, I have a separate node which runs cinder.
Whenever I reboot this machine, the existing volumes get "inactive" and I cannot attach them to any VM. Even when I lvchange them to active, I cannot attach them either.

And here is compute log when I try to attach a volume to VM:

2013-10-21 16:33:38 AUDIT nova.compute.manager [req-edf41883-2d87-47f7-a113-c71d94595df1 3800b26a95c44e7abd1c080f7695560a 4c2340f60b7c48a0a2504eeefe2ff74a] [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8] Attaching volume 370ccf0c-8d12-4e88-8cbc-2b12d9a4595f to /dev/vdb
2013-10-21 16:33:39 WARNING nova.virt.libvirt.volume [req-edf41883-2d87-47f7-a113-c71d94595df1 3800b26a95c44e7abd1c080f7695560a 4c2340f60b7c48a0a2504eeefe2ff74a] ISCSI volume not yet found at: vdb. Will rescan & retry.  Try number: 0
2013-10-21 16:33:40 WARNING nova.virt.libvirt.volume [req-edf41883-2d87-47f7-a113-c71d94595df1 3800b26a95c44e7abd1c080f7695560a 4c2340f60b7c48a0a2504eeefe2ff74a] ISCSI volume not yet found at: vdb. Will rescan & retry.  Try number: 1
2013-10-21 16:33:44 WARNING nova.virt.libvirt.volume [req-edf41883-2d87-47f7-a113-c71d94595df1 3800b26a95c44e7abd1c080f7695560a 4c2340f60b7c48a0a2504eeefe2ff74a] ISCSI volume not yet found at: vdb. Will rescan & retry.  Try number: 2
2013-10-21 16:33:54 ERROR nova.compute.manager [req-edf41883-2d87-47f7-a113-c71d94595df1 3800b26a95c44e7abd1c080f7695560a 4c2340f60b7c48a0a2504eeefe2ff74a] [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8] Failed to attach volume 370ccf0c-8d12-4e88-8cbc-2b12d9a4595f at /dev/vdb
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8] Traceback (most recent call last):
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2046, in _attach_volume
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     mountpoint)
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     temp_level, payload)
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     self.gen.next()
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     return f(*args, **kw)
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 642, in attach_volume
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     mount_device)
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 634, in volume_driver_method
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     return method(connection_info, *args, **kwargs)
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 760, in inner
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     retval = f(*args, **kwargs)
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 165, in connect_volume
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]     % (host_device))
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8] NovaException: iSCSI device not found at /dev/disk/by-path/ip-10.1.0.13:3260-iscsi-iqn.2010-10.org.openstack:volume-370ccf0c-8d12-4e88-8cbc-2b12d9a4595f-lun-1
2013-10-21 16:33:54 23470 TRACE nova.compute.manager [instance: b6a6ddcd-c785-40bc-9fe6-f68b682fb2f8]
2013-10-21 16:33:55 23470 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kw)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 175, in decorated_function
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     pass
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 161, in decorated_function
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 196, in decorated_function
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     kwargs['instance']['uuid'], e, sys.exc_info())
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 190, in decorated_function
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2020, in attach_volume
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     context, instance.get('uuid'), mountpoint)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2016, in attach_volume
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     mountpoint, instance)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2055, in _attach_volume
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     connector)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2046, in _attach_volume
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     mountpoint)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kw)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 642, in attach_volume
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     mount_device)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 634, in volume_driver_method
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     return method(connection_info, *args, **kwargs)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 760, in inner
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 165, in connect_volume
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp     % (host_device))
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp NovaException: iSCSI device not found at /dev/disk/by-path/ip-10.1.0.13:3260-iscsi-iqn.2010-10.org.openstack:volume-370ccf0c-8d12-4e88-8cbc-2b12d9a4595f-lun-1
2013-10-21 16:33:55 23470 TRACE nova.openstack.common.rpc.amqp


Thank you,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20131031/3a619e25/attachment.html>