Open Stack

Hi


I'm running (trying) a openstack (version victoria) with  a Dell Unity
storage unit.

When I'm create a instance, everything work fine, the block volume are
created on the Unity, the mount (with iscsi) work on the compute and the
instance boot normaly.

But if I shutdown a instance and try to restart it, it's failed.

It's just like the block volume cannot be mount. I check on the unity
eveyrthing are ok

In the log of the compute

  /var/log/nova/nova-compute.log

I can see  :

Any idea ?

Regards



2021-05-21 21:39:15.849 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] LUN 5 on iSCSI portal 10.15.23.252:3260 not found on sysfs after logging in.
2021-05-21 21:39:15.952 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found
: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:15.960 2663 WARNING os_brick.initiator.connectors.iscsi [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:16.231 2663 INFO nova.compute.manager [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] Successfully reverted task state from powering-on on failure for instance.
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server [req-42f8147b-b006-4af7-8dfb-31c39b46b728 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27 223c116cac324ae19fd74877a6c06d27] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in wrapped
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     _emit_exception_notification(
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 188, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     LOG.warning("Failed to revert task state for instance. "
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1456, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     compute_utils.add_instance_fault_from_exc(context,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in decorated_function
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3152, in start_instance
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3120, in _power_on
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.driver.power_on(context, instance,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3429, in power_on
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3294, in _hard_reboot
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     xml = self._get_guest_xml(context, instance, network_info, disk_info,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6364, in _get_guest_xml
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     conf = self._get_guest_config(instance, network_info, image_meta,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6006, in _get_guest_config
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     storage_configs = self._get_guest_storage_config(context,
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4750, in _get_guest_storage_config
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._connect_volume(context, connection_info, instance)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in _connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     vol_driver.connect_volume(connection_info, instance)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     device_info = self.connector.connect_volume(connection_info['data'])
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self._cleanup_connection(connection_properties, force=True)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return self._connect_single_volume(connection_properties)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return r.call(f, *args, **kwargs)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise attempt.get()
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise value
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 591, in _connect_single_volume
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise exception.VolumeDeviceNotFound(device='')
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
2021-05-21 21:43:45.556 2663 WARNING nova.compute.manager [req-6d91e265-b6b2-4522-826d-176e0d24a2d3 - - - - -] While synchronizing instance power states, found 1 instances in the database and 0 instances on the hypervisor.
2021-05-21 21:43:45.598 2663 INFO nova.compute.manager [-] [instance: ec79c0d6-259d-466f-94da-def4ce318d3a] During _sync_instance_power_state the DB power_state (4) does not match the vm_power_state from the hypervisor (0). Updating power_state in the DB to match the hypervisor.

--
Albert SHIH
Observatoire de Paris
France
xmpp: jas at obspm.fr
Heure local/Local time:
Fri May 21 11:43:30 PM CEST 2021