[victoria] Reboot problem

Laurent Dumont laurentfdumont at gmail.com
Sat May 22 14:27:52 UTC 2021 

I would recommend turning up the debug logs for nova on the compute and
trying again. It could be something where the ISCSI session is not properly
closed when you shutdown the VM and it creates issues when the VM is
started again.

On Fri, May 21, 2021 at 5:57 PM Albert Shih <Albert.Shih at obspm.fr> wrote:

> Hi
>
>
> I'm running (trying) a openstack (version victoria) with  a Dell Unity
> storage unit.
>
> When I'm create a instance, everything work fine, the block volume are
> created on the Unity, the mount (with iscsi) work on the compute and the
> instance boot normaly.
>
> But if I shutdown a instance and try to restart it, it's failed.
>
> It's just like the block volume cannot be mount. I check on the unity
> eveyrthing are ok
>
> In the log of the compute
>
>   /var/log/nova/nova-compute.log
>
> I can see  :
>
> Any idea ?
>
> Regards
>
>
>
> 2021-05-21 21:39:15.849 2663 WARNING os_brick.initiator.connectors.iscsi
> [req-42f8147b-b006-4af7-8dfb-31c39b46b728
> 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96
> 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27
> 223c116cac324ae19fd74877a6c06d27] LUN 5 on iSCSI portal 10.15.23.252:3260
> not found on sysfs after logging in.
> 2021-05-21 21:39:15.952 2663 WARNING os_brick.initiator.connectors.iscsi
> [req-42f8147b-b006-4af7-8dfb-31c39b46b728
> 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96
> 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27
> 223c116cac324ae19fd74877a6c06d27] Couldn't find iSCSI nodes because
> iscsiadm err: iscsiadm: No records found
> : os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
> 2021-05-21 21:39:15.960 2663 WARNING os_brick.initiator.connectors.iscsi
> [req-42f8147b-b006-4af7-8dfb-31c39b46b728
> 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96
> 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27
> 223c116cac324ae19fd74877a6c06d27] iscsiadm stderr output when getting
> sessions: iscsiadm: No active sessions.
> : os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
> 2021-05-21 21:39:16.231 2663 INFO nova.compute.manager
> [req-42f8147b-b006-4af7-8dfb-31c39b46b728
> 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96
> 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27
> 223c116cac324ae19fd74877a6c06d27] [instance:
> ec79c0d6-259d-466f-94da-def4ce318d3a] Successfully reverted task state from
> powering-on on failure for instance.
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
> [req-42f8147b-b006-4af7-8dfb-31c39b46b728
> 868dde297576ce232570ea549928d02d58f543007825d199eb48f30a10139c96
> 416ac2de9ed74316920e9cbe3d376bb3 - 223c116cac324ae19fd74877a6c06d27
> 223c116cac324ae19fd74877a6c06d27] Exception during message handling:
> os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server Traceback
> (most recent call last):
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in
> _process_incoming
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     res =
> self.dispatcher.dispatch(message)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line
> 276, in dispatch
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line
> 196, in _do_dispatch
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     result =
> func(ctxt, **new_args)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in
> wrapped
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  _emit_exception_notification(
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self.force_reraise()
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  six.reraise(self.type_, self.value, self.tb)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise
> value
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in
> wrapped
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> f(self, context, *args, **kw)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 188, in
> decorated_function
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  LOG.warning("Failed to revert task state for instance. "
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self.force_reraise()
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  six.reraise(self.type_, self.value, self.tb)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise
> value
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 159, in
> decorated_function
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> function(self, context, *args, **kwargs)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1456, in
> decorated_function
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> function(self, context, *args, **kwargs)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in
> decorated_function
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  compute_utils.add_instance_fault_from_exc(context,
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self.force_reraise()
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  six.reraise(self.type_, self.value, self.tb)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise
> value
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in
> decorated_function
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> function(self, context, *args, **kwargs)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3152, in
> start_instance
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self._power_on(context, instance)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3120, in
> _power_on
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self.driver.power_on(context, instance,
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3429, in
> power_on
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self._hard_reboot(context, instance, network_info, block_device_info,
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3294, in
> _hard_reboot
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     xml =
> self._get_guest_xml(context, instance, network_info, disk_info,
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6364, in
> _get_guest_xml
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     conf =
> self._get_guest_config(instance, network_info, image_meta,
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6006, in
> _get_guest_config
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  storage_configs = self._get_guest_storage_config(context,
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4750, in
> _get_guest_storage_config
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self._connect_volume(context, connection_info, instance)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1623, in
> _connect_volume
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  vol_driver.connect_volume(connection_info, instance)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line
> 64, in connect_volume
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  device_info = self.connector.connect_volume(connection_info['data'])
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/os_brick/utils.py", line 137, in
> trace_logging_wrapper
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> f(*args, **kwargs)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359,
> in inner
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> f(*args, **kwargs)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py",
> line 519, in connect_volume
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self._cleanup_connection(connection_properties, force=True)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  self.force_reraise()
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  six.reraise(self.type_, self.value, self.tb)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise
> value
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py",
> line 513, in connect_volume
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> self._connect_single_volume(connection_properties)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/os_brick/utils.py", line 61, in _wrapper
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     return
> r.call(f, *args, **kwargs)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/retrying.py", line 212, in call
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise
> attempt.get()
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
>  six.reraise(self.value[0], self.value[1], self.value[2])
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise
> value
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     attempt =
> Attempt(fn(*args, **kwargs), attempt_number, False)
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py",
> line 591, in _connect_single_volume
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server     raise
> exception.VolumeDeviceNotFound(device='')
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
> os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
> 2021-05-21 21:39:16.237 2663 ERROR oslo_messaging.rpc.server
> 2021-05-21 21:43:45.556 2663 WARNING nova.compute.manager
> [req-6d91e265-b6b2-4522-826d-176e0d24a2d3 - - - - -] While synchronizing
> instance power states, found 1 instances in the database and 0 instances on
> the hypervisor.
> 2021-05-21 21:43:45.598 2663 INFO nova.compute.manager [-] [instance:
> ec79c0d6-259d-466f-94da-def4ce318d3a] During _sync_instance_power_state the
> DB power_state (4) does not match the vm_power_state from the hypervisor
> (0). Updating power_state in the DB to match the hypervisor.
>
> --
> Albert SHIH
> Observatoire de Paris
> France
> xmpp: jas at obspm.fr
> Heure local/Local time:
> Fri May 21 11:43:30 PM CEST 2021
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210522/47c3e3ae/attachment-0001.html>

More information about the openstack-discuss mailing list