<div dir="ltr">I would recommend turning up the debug logs for nova on the compute and trying again. It could be something where the ISCSI session is not properly closed when you shutdown the VM and it creates issues when the VM is started again.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 21, 2021 at 5:57 PM Albert Shih <<a href="mailto:Albert.Shih@obspm.fr">Albert.Shih@obspm.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi<br>
<br>
<br>
I'm running (trying) a openstack (version victoria) with a Dell Unity<br>
storage unit.<br>
<br>
When I'm create a instance, everything work fine, the block volume are<br>
created on the Unity, the mount (with iscsi) work on the compute and the<br>
instance boot normaly.<br>
<br>
But if I shutdown a instance and try to restart it, it's failed.<br>
<br>
It's just like the block volume cannot be mount. I check on the unity<br>
eveyrthing are ok<br>
<br>
In the log of the compute<br>
<br>
/var/log/nova/nova-compute.log<br>
<br>
I can see :<br>
<br>
Any idea ?<br>
<br>
Regards<br>
<br>
<br>
<br>
2021-05-21 21:39:15.849 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] LUN 5 on iSCSI portal <a href="http://10.15.23.252:3260" rel="noreferrer" target="_blank">10.15.23.252:3260</a> not found on sysfs after logging in.<br>
2021-05-21 21:39:15.952 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found<br>
: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .<br>
2021-05-21 21:39:15.960 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.<br>
: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .<br>
2021-05-21 21:39:16.231 2663 INFO nova.compute.manager [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] Successfully reverted task state from powering-on on failure for instance.<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server Traceback (most recent call last):<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in wrapped<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server _emit_exception_notification(<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise()<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 188, in decorated_function<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise()<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 159, in decorated_function<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1456, in decorated_function<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise()<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in decorated_function<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3152, in start_instance<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._power_on(context, instance)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3120, in _power_on<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.driver.power_on(context, instance,<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3429, in power_on<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._hard_reboot(context, instance, network_info, block_device_info,<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3294, in _hard_reboot<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server xml = self._get_guest_xml(context, instance, network_info, disk_info,<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6364, in _get_guest_xml<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server conf = self._get_guest_config(instance, network_info, image_meta,<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6006, in _get_guest_config<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server storage_configs = self._get_guest_storage_config(context,<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4750, in _get_guest_storage_config<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._connect_volume(context, connection_info, instance)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in _connect_volume<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server vol_driver.connect_volume(connection_info, instance)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server device_info = self.connector.connect_volume(connection_info['data'])<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self._cleanup_connection(connection_properties, force=True)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server self.force_reraise()<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_volume<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return self._connect_single_volume(connection_properties)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise attempt.get()<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2])<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise value<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False)<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 591, in _connect_single_volume<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server raise exception.VolumeDeviceNotFound(device='')<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server os_brick.exception.VolumeDeviceNotFound: Volume device not found at .<br>
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server<br>
2021-05-21 21:43:45.556 2663 WARNING nova.compute.manager [req-6d91e265-b6b2-4522-826d-176e0d24a2d3 - - - - -] While synchronizing instance power states, found 1 instances in the database and 0 instances on the hypervisor.<br>
2021-05-21 21:43:45.598 2663 INFO nova.compute.manager [-] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] During _sync_instance_power_state the DB power_state (4) does not match the vm_power_state from the hypervisor (0). Updating power_state in the DB to match the hypervisor.<br>
<br>
--<br>
Albert SHIH<br>
Observatoire de Paris<br>
France<br>
xmpp: <a href="mailto:jas@obspm.fr" target="_blank">jas@obspm.fr</a><br>
Heure local/Local time:<br>
Fri May 21 11:43:30 PM CEST 2021<br>
<br>
</blockquote></div>