Hi I'm running (trying) a openstack (version victoria) with a Dell Unity storage unit. When I'm create a instance, everything work fine, the block volume are created on the Unity, the mount (with iscsi) work on the compute and the instance boot normaly. But if I shutdown a instance and try to restart it, it's failed. It's just like the block volume cannot be mount. I check on the unity eveyrthing are ok In the log of the compute /var/log/nova/nova-compute.log I can see : Any idea ? Regards 2021-05-21 21:39:15.849 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] LUN 5 on iSCSI portal 10.15.23.252:3260 not found on sysfs after logging in. 2021-05-21 21:39:15.952 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found : os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:15.960 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. : os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:16.231 2663 INFO nova.compute.manager [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] Successfully reverted task state from powering-on on failure for instance. 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in wrapped 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server _emit_exception_notification( 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 188, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. " 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 159, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1456, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3152, in start_instance 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._power_on(context, instance) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3120, in _power_on 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.driver.power_on(context, instance, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3429, in power_on 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._hard_reboot(context, instance, network_info, block_device_info, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3294, in _hard_reboot 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server xml = self._get_guest_xml(context, instance, network_info, disk_info, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6364, in _get_guest_xml 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server conf = self._get_guest_config(instance, network_info, image_meta, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6006, in _get_guest_config 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server storage_configs = self._get_guest_storage_config(context, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4750, in _get_guest_storage_config 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._connect_volume(context, connection_info, instance) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in _connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server vol_driver.connect_volume(connection_info, instance) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server device_info = self.connector.connect_volume(connection_info['data']) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._cleanup_connection(connection_properties, force=True) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return self._connect_single_volume(connection_properties) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise attempt.get() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 591, in _connect_single_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise exception.VolumeDeviceNotFound(device='') 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server 2021-05-21 21:43:45.556 2663 WARNING nova.compute.manager [req-6d91e265-b6b2-4522-826d-176e0d24a2d3 - - - - -] While synchronizing instance power states, found 1 instances in the database and 0 instances on the hypervisor. 2021-05-21 21:43:45.598 2663 INFO nova.compute.manager [-] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] During _sync_instance_power_state the DB power_state (4) does not match the vm_power_state from the hypervisor (0). Updating power_state in the DB to match the hypervisor. -- Albert SHIH Observatoire de Paris France xmpp: jas@obspm.fr Heure local/Local time: Fri May 21 11:43:30 PM CEST 2021