Re: [cinder] Error when creating backups from iscsi volume

9 Mar 2023

      On 06/03, Rishat Azizov wrote:
...
Hi,
It works with smaller volumes.
multipath.conf attached to thist email.
Cinder version - 18.2.0 Wallaby
Hi,

After giving it some thought I think I may know what is going on.

If you have DEBUG logs enabled in cinder-backup when it fails, how many
calls do you see in the cinder-backup to "multipath -f" from os-brick,
only one or do you see more?

Cheers,
Gorka.
...
пн, 6 мар. 2023 г. в 17:35, Gorka Eguileor <geguileo@redhat.com>:
...
On 16/02, Rishat Azizov wrote:
...
Hello!
We have an error with creating backups from iscsi volume. Usually, this
happens with large backups over 100GB.
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
[req-f6619913-6f96-4226-8d75-2da3fca722f1
23de1b92e7674cf59486f07ac75b886b
a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message
handling:
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error
while
running command.
Command: multipath -f 3624a93705842cfae35d7483200015ec6
Exit code: 1
Stdout: ''
Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a
multipath device\n'
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Traceback
(most recent call last):
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line
165,
in _process_incoming
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     res =
self.dispatcher.dispatch(message)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line
309, in dispatch
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
self._do_dispatch(endpoint, method, ctxt, args)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line
229, in _do_dispatch
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     result =
func(ctxt, **new_args)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
func(self, *args, **kwargs)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in
create_backup
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
volume_utils.update_backup_error(backup, str(err))
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in
__exit__
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
self.force_reraise()
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     raise
self.value
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in
create_backup
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     updates
=
self._run_backup(context, backup, volume)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in
_run_backup
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
ignore_errors=True)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066,
in
_detach_device
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
force=force, ignore_errors=ignore_errors)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in
trace_logging_wrapper
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
f(*args, **kwargs)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line
360,
in inner
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
f(*args, **kwargs)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
...
line 880, in disconnect_volume
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
is_disconnect_call=True)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
...
line 942, in _cleanup_connection
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
self._linuxscsi.flush_multipath_device(multipath_name)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line
382, in flush_multipath_device
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
root_helper=self._root_helper)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in
_execute
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     result =
self.__execute(*args, **kwargs)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line
172, in execute
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
execute_root(*cmd, **kwargs)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line
247,
in _wrap
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
self.channel.remote_call(name, args, kwargs)
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in
remote_call
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     raise
exc_type(*result[2])
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error
while
running command.
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command:
multipath -f 3624a93705842cfae35d7483200015ec6
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit code: 1
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: ''
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: 'Feb
16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath
device\n'
2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
Could you please help with this error?
Hi,
Does it work for smaller volumes or does it also fail?
What are your defaults in your /etc/multipath.conf file?
What Cinder release are you using?
Cheers,
Gorka.

...
defaults {
        user_friendly_names no
        find_multipaths yes
        enable_foreign "^$"
}
blacklist_exceptions {
        property "(SCSI_IDENT_|ID_WWN)"
}
blacklist {
}
devices {
  device {
        vendor "PURE"
        product "FlashArray"
        fast_io_fail_tmo 10
        path_grouping_policy "group_by_prio"
        failback "immediate"
        prio "alua"
        hardware_handler "1 alua"
        max_sectors_kb 4096
    }
}

Gorka Eguileor

tags

participants (1)