Open Stack

On 06/03, Rishat Azizov wrote:
> Hi,
>
> It works with smaller volumes.
>
> multipath.conf attached to thist email.
>
> Cinder version - 18.2.0 Wallaby

Hi,

After giving it some thought I think I may know what is going on.

If you have DEBUG logs enabled in cinder-backup when it fails, how many
calls do you see in the cinder-backup to "multipath -f" from os-brick,
only one or do you see more?

Cheers,
Gorka.

>
> пн, 6 мар. 2023 г. в 17:35, Gorka Eguileor <geguileo at redhat.com>:
>
> > On 16/02, Rishat Azizov wrote:
> > > Hello!
> > >
> > > We have an error with creating backups from iscsi volume. Usually, this
> > > happens with large backups over 100GB.
> > >
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > [req-f6619913-6f96-4226-8d75-2da3fca722f1
> > 23de1b92e7674cf59486f07ac75b886b
> > > a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message
> > handling:
> > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error
> > while
> > > running command.
> > > Command: multipath -f 3624a93705842cfae35d7483200015ec6
> > > Exit code: 1
> > > Stdout: ''
> > > Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a
> > > multipath device\n'
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Traceback
> > > (most recent call last):
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line
> > 165,
> > > in _process_incoming
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     res =
> > > self.dispatcher.dispatch(message)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line
> > > 309, in dispatch
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> > > self._do_dispatch(endpoint, method, ctxt, args)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line
> > > 229, in _do_dispatch
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     result =
> > > func(ctxt, **new_args)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> > > func(self, *args, **kwargs)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in
> > > create_backup
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > volume_utils.update_backup_error(backup, str(err))
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in
> > > __exit__
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > self.force_reraise()
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in
> > > force_reraise
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     raise
> > > self.value
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in
> > > create_backup
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     updates
> > =
> > > self._run_backup(context, backup, volume)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in
> > > _run_backup
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > ignore_errors=True)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066,
> > in
> > > _detach_device
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > force=force, ignore_errors=ignore_errors)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in
> > > trace_logging_wrapper
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> > > f(*args, **kwargs)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line
> > 360,
> > > in inner
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> > > f(*args, **kwargs)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > >
> > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
> > > line 880, in disconnect_volume
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > is_disconnect_call=True)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > >
> > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py",
> > > line 942, in _cleanup_connection
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > self._linuxscsi.flush_multipath_device(multipath_name)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line
> > > 382, in flush_multipath_device
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > root_helper=self._root_helper)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in
> > > _execute
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     result =
> > > self.__execute(*args, **kwargs)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line
> > > 172, in execute
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> > > execute_root(*cmd, **kwargs)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line
> > 247,
> > > in _wrap
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     return
> > > self.channel.remote_call(name, args, kwargs)
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server   File
> > > "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in
> > > remote_call
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server     raise
> > > exc_type(*result[2])
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error
> > while
> > > running command.
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command:
> > > multipath -f 3624a93705842cfae35d7483200015ec6
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit code: 1
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: ''
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: 'Feb
> > > 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath
> > device\n'
> > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server
> > >
> > > Could you please help with this error?
> >
> > Hi,
> >
> > Does it work for smaller volumes or does it also fail?
> >
> > What are your defaults in your /etc/multipath.conf file?
> >
> > What Cinder release are you using?
> >
> > Cheers,
> > Gorka.
> >
> >

> defaults {
>         user_friendly_names no
>         find_multipaths yes
>         enable_foreign "^$"
> }
>
> blacklist_exceptions {
>         property "(SCSI_IDENT_|ID_WWN)"
> }
>
> blacklist {
> }
>
> devices {
>   device {
>         vendor "PURE"
>         product "FlashArray"
>         fast_io_fail_tmo 10
>         path_grouping_policy "group_by_prio"
>         failback "immediate"
>         prio "alua"
>         hardware_handler "1 alua"
>         max_sectors_kb 4096
>     }
> }