Open Stack

On 16/02, Rishat Azizov wrote:
> Hello!
>
> We have an error with creating backups from iscsi volume. Usually, this
> happens with large backups over 100GB.
>
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> [req-f6619913-6f96-4226-8d75-2da3fca722f1 23de1b92e7674cf59486f07ac75b886b
> a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message handling:
> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
> running command.
> Command: multipath -f 3624a93705842cfae35d7483200015ec6
> Exit code: 1
> Stdout: ''
> Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a
> multipath device\n'
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Traceback
> (most recent call last):
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165,
> in _process_incoming
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     res =
> self.dispatcher.dispatch(message)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 309, in dispatch
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 229, in _do_dispatch
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     result =
> func(ctxt, **new_args)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> func(self, *args, **kwargs)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in
> create_backup
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> volume_utils.update_backup_error(backup, str(err))
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in
> __exit__
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in
> force_reraise
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     raise
> self.value
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in
> create_backup
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     updates =
> self._run_backup(context, backup, volume)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in
> _run_backup
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> ignore_errors=True)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066, in
> _detach_device
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> force=force, ignore_errors=ignore_errors)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in
> trace_logging_wrapper
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> f(*args, **kwargs)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360,
> in inner
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> f(*args, **kwargs)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
> line 880, in disconnect_volume
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> is_disconnect_call=True)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
> line 942, in _cleanup_connection
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> self._linuxscsi.flush_multipath_device(multipath_name)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line
> 382, in flush_multipath_device
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> root_helper=self._root_helper)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in
> _execute
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     result =
> self.__execute(*args, **kwargs)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line
> 172, in execute
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> execute_root(*cmd, **kwargs)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 247,
> in _wrap
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> self.channel.remote_call(name, args, kwargs)
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in
> remote_call
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     raise
> exc_type(*result[2])
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
> running command.
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command:
> multipath -f 3624a93705842cfae35d7483200015ec6
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit code: 1
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: ''
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: 'Feb
> 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath device\n'
> 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
>
> Could you please help with this error?

Hi,

Does it work for smaller volumes or does it also fail?

What are your defaults in your /etc/multipath.conf file?

What Cinder release are you using?

Cheers,
Gorka.