I would recommend turning up the debug logs for nova on the compute and trying again. It could be something where the ISCSI session is not properly closed when you shutdown the VM and it creates issues when the VM is started again.

On Fri, May 21, 2021 at 5:57 PM Albert Shih <Albert.Shih@obspm.fr> wrote:
Hi


I'm running (trying) a openstack (version victoria) with  a Dell Unity
storage unit.

When I'm create a instance, everything work fine, the block volume are
created on the Unity, the mount (with iscsi) work on the compute and the
instance boot normaly.

But if I shutdown a instance and try to restart it, it's failed.

It's just like the block volume cannot be mount. I check on the unity
eveyrthing are ok

In the log of the compute

  /var/log/nova/nova-compute.log

I can see  :

Any idea ?

Regards



2021-05-21 21:39:15.849 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] LUN 5 on iSCSI portal 10.15.23.252:3260 not found on sysfs after logging in.
2021-05-21 21:39:15.952 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found
: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:15.960 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:16.231 2663 INFO nova.compute.manager [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] Successfully reverted task state from powering-on on failure for instance.
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in wrapped
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     _emit_exception_notification(
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 188, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     LOG.warning("Failed to revert task state for instance. "
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1456, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     compute_utils.add_instance_fault_from_exc(context,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3152, in start_instance
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3120, in _power_on
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.driver.power_on(context, instance,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3429, in power_on
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3294, in _hard_reboot
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     xml = self._get_guest_xml(context, instance, network_info, disk_info,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6364, in _get_guest_xml
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     conf = self._get_guest_config(instance, network_info, image_meta,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6006, in _get_guest_config
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     storage_configs = self._get_guest_storage_config(context,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4750, in _get_guest_storage_config
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._connect_volume(context, connection_info, instance)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in _connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     vol_driver.connect_volume(connection_info, instance)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     device_info = self.connector.connect_volume(connection_info['data'])
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._cleanup_connection(connection_properties, force=True)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return self._connect_single_volume(connection_properties)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return r.call(f, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise attempt.get()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 591, in _connect_single_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise exception.VolumeDeviceNotFound(device='')
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
2021-05-21 21:43:45.556 2663 WARNING nova.compute.manager [req-6d91e265-b6b2-4522-826d-176e0d24a2d3 - - - - -] While synchronizing instance power states, found 1 instances in the database and 0 instances on the hypervisor.
2021-05-21 21:43:45.598 2663 INFO nova.compute.manager [-] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] During _sync_instance_power_state the DB power_state (4) does not match the vm_power_state from the hypervisor (0). Updating power_state in the DB to match the hypervisor.

--
Albert SHIH
Observatoire de Paris
France
xmpp: jas@obspm.fr
Heure local/Local time:
Fri May 21 11:43:30 PM CEST 2021