I would recommend turning up the debug logs for nova on the compute and trying again. It could be something where the ISCSI session is not properly closed when you shutdown the VM and it creates issues when the VM is started again. On Fri, May 21, 2021 at 5:57 PM Albert Shih <Albert.Shih@obspm.fr> wrote:

Hi

I'm running (trying) a openstack (version victoria) with a Dell Unity storage unit.

When I'm create a instance, everything work fine, the block volume are created on the Unity, the mount (with iscsi) work on the compute and the instance boot normaly.

But if I shutdown a instance and try to restart it, it's failed.

It's just like the block volume cannot be mount. I check on the unity eveyrthing are ok

In the log of the compute

/var/log/nova/nova-compute.log

I can see :

Any idea ?

Regards

2021-05-21 21:39:15.849 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] LUN 5 on iSCSI portal 10.15.23.252:3260 not found on sysfs after logging in. 2021-05-21 21:39:15.952 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found : os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:15.960 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. : os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:16.231 2663 INFO nova.compute.manager [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] Successfully reverted task state from powering-on on failure for instance. 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in wrapped 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server _emit_exception_notification( 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 188, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. " 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 159, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1456, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in decorated_function 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3152, in start_instance 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._power_on(context, instance) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3120, in _power_on 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.driver.power_on(context, instance, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3429, in power_on 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._hard_reboot(context, instance, network_info, block_device_info, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3294, in _hard_reboot 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server xml = self._get_guest_xml(context, instance, network_info, disk_info, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6364, in _get_guest_xml 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server conf = self._get_guest_config(instance, network_info, image_meta, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6006, in _get_guest_config 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server storage_configs = self._get_guest_storage_config(context, 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4750, in _get_guest_storage_config 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._connect_volume(context, connection_info, instance) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in _connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server vol_driver.connect_volume(connection_info, instance) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server device_info = self.connector.connect_volume(connection_info['data']) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._cleanup_connection(connection_properties, force=True) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return self._connect_single_volume(connection_properties) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise attempt.get() 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 591, in _connect_single_volume 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise exception.VolumeDeviceNotFound(device='') 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server os_brick.exception.VolumeDeviceNotFound: Volume device not found at . 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server 2021-05-21 21:43:45.556 2663 WARNING nova.compute.manager [req-6d91e265-b6b2-4522-826d-176e0d24a2d3 - - - - -] While synchronizing instance power states, found 1 instances in the database and 0 instances on the hypervisor. 2021-05-21 21:43:45.598 2663 INFO nova.compute.manager [-] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] During _sync_instance_power_state the DB power_state (4) does not match the vm_power_state from the hypervisor (0). Updating power_state in the DB to match the hypervisor.

-- Albert SHIH Observatoire de Paris France xmpp: jas@obspm.fr Heure local/Local time: Fri May 21 11:43:30 PM CEST 2021