March 2023 - openstack-discuss - lists.openstack.org

[ptl] Need PTL volunteer for OpenStack Sahara
by Ghanshyam Mann 22 Mar '23

22 Mar '23

Hi Qiu, I am reaching out to you as you were PTL for OpenStack Sahara project in the last cycle. There is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list. Please check if you or anyone you know would like to lead this project. - https://etherpad.opendev.org/p/2023.2-leaderless Also, if anyone else would like to help leading this project, this is time to let TC knows. -gmann

1 0

Re: [cinder] Error when creating backups from iscsi volume
by Gorka Eguileor 22 Mar '23

22 Mar '23

On 11/03, Rishat Azizov wrote: > Hi, Gorka, > > Thanks. I see multiple "multipath -f" calls. Logs in attachments. > Hi, There are multiple things going on here: 1. There is a bug in os-brick, because the disconnect_volume should not fail, since it is being called with force=True and ignore_errors=True. The issues is that this call [1] is not wrapped in the ExceptionChainer context manager, and it should not even be a flush call, it should be a call to "multipathd remove map $map" instead. 2. The way multipath code is written [2][3], the error we see about "3624a93705842cfae35d7483200015fce is not a multipath device" means 2 different things: it is not a multipath or an error happened. So we don't really know what happened without enabling more verbose multipathd log levels. 3. The "multipath -f" call should not be failing in the first place, because the failure is happening on disconnecting the source volume, which has no data buffered to be written and therefore no reason to fail the flush (unless it's using a friendly name). I don't know if it could be happening that the first flush fails with a timeout (maybe because there is an extend operation happening), but multipathd keeps trying to flush it in the background and when it succeeds it removes the multipath device, which makes following calls fail. If that's the case we would need to change the retry from automatic [4] to manual and check in-between to see if the device has been removed in-between calls. The first issue is definitely a bug, the 2nd one is something that could be changed in the deployment to try to get additional information on the failure, and the 3rd one could be a bug. I'll see if I can find someone who wants to work on the 1st and 3rd points. Cheers, Gorka. [1]: https://github.com/openstack/os-brick/blob/e15edf6c17449899ec8401c37482f7cb… [2]: https://github.com/opensvc/multipath-tools/blob/db4804bc7393f2482448bdd8701… [3]: https://github.com/opensvc/multipath-tools/blob/db4804bc7393f2482448bdd8701… [4]: https://github.com/openstack/os-brick/blob/e15edf6c17449899ec8401c37482f7cb… > > чт, 9 мар. 2023 г. в 15:55, Gorka Eguileor <geguileo(a)redhat.com>: > > > On 06/03, Rishat Azizov wrote: > > > Hi, > > > > > > It works with smaller volumes. > > > > > > multipath.conf attached to thist email. > > > > > > Cinder version - 18.2.0 Wallaby > > > > Hi, > > > > After giving it some thought I think I may know what is going on. > > > > If you have DEBUG logs enabled in cinder-backup when it fails, how many > > calls do you see in the cinder-backup to "multipath -f" from os-brick, > > only one or do you see more? > > > > Cheers, > > Gorka. > > > > > > > > пн, 6 мар. 2023 г. в 17:35, Gorka Eguileor <geguileo(a)redhat.com>: > > > > > > > On 16/02, Rishat Azizov wrote: > > > > > Hello! > > > > > > > > > > We have an error with creating backups from iscsi volume. Usually, > > this > > > > > happens with large backups over 100GB. > > > > > > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > [req-f6619913-6f96-4226-8d75-2da3fca722f1 > > > > 23de1b92e7674cf59486f07ac75b886b > > > > > a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message > > > > handling: > > > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > > > while > > > > > running command. > > > > > Command: multipath -f 3624a93705842cfae35d7483200015ec6 > > > > > Exit code: 1 > > > > > Stdout: '' > > > > > Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a > > > > > multipath device\n' > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > Traceback > > > > > (most recent call last): > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line > > > > 165, > > > > > in _process_incoming > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server res > > = > > > > > self.dispatcher.dispatch(message) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > > line > > > > > 309, in dispatch > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > self._do_dispatch(endpoint, method, ctxt, args) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > > line > > > > > 229, in _do_dispatch > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > result = > > > > > func(ctxt, **new_args) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in > > wrapper > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > func(self, *args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 410, in > > > > > create_backup > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > volume_utils.update_backup_error(backup, str(err)) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, > > in > > > > > __exit__ > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > self.force_reraise() > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, > > in > > > > > force_reraise > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > raise > > > > > self.value > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 399, in > > > > > create_backup > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > updates > > > > = > > > > > self._run_backup(context, backup, volume) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 493, in > > > > > _run_backup > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > ignore_errors=True) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 1066, > > > > in > > > > > _detach_device > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > force=force, ignore_errors=ignore_errors) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in > > > > > trace_logging_wrapper > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > f(*args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", > > line > > > > 360, > > > > > in inner > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > f(*args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > > > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > > > line 880, in disconnect_volume > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > is_disconnect_call=True) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > > > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > > > line 942, in _cleanup_connection > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > self._linuxscsi.flush_multipath_device(multipath_name) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", > > line > > > > > 382, in flush_multipath_device > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > root_helper=self._root_helper) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in > > > > > _execute > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > result = > > > > > self.__execute(*args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", > > line > > > > > 172, in execute > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > execute_root(*cmd, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line > > > > 247, > > > > > in _wrap > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > self.channel.remote_call(name, args, kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, > > in > > > > > remote_call > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > raise > > > > > exc_type(*result[2]) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > > > while > > > > > running command. > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command: > > > > > multipath -f 3624a93705842cfae35d7483200015ec6 > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit > > code: 1 > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: > > '' > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: > > 'Feb > > > > > 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath > > > > device\n' > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > > > > > > Could you please help with this error? > > > > > > > > Hi, > > > > > > > > Does it work for smaller volumes or does it also fail? > > > > > > > > What are your defaults in your /etc/multipath.conf file? > > > > > > > > What Cinder release are you using? > > > > > > > > Cheers, > > > > Gorka. > > > > > > > > > > > > > defaults { > > > user_friendly_names no > > > find_multipaths yes > > > enable_foreign "^$" > > > } > > > > > > blacklist_exceptions { > > > property "(SCSI_IDENT_|ID_WWN)" > > > } > > > > > > blacklist { > > > } > > > > > > devices { > > > device { > > > vendor "PURE" > > > product "FlashArray" > > > fast_io_fail_tmo 10 > > > path_grouping_policy "group_by_prio" > > > failback "immediate" > > > prio "alua" > > > hardware_handler "1 alua" > > > max_sectors_kb 4096 > > > } > > > } > > > > > 2023-03-10 16:42:41.785 2878341 DEBUG cinder.backup.drivers.ceph [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Transferred chunk 1600 of 1600 (84387K/s) _transfer_data /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:426 > 2023-03-10 16:42:42.107 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting metadata type 'volume-base-metadata' _save_vol_base_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:79 > 2023-03-10 16:42:42.139 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Completed fetching metadata type 'volume-base-metadata' _save_vol_base_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:98 > 2023-03-10 16:42:42.139 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting metadata type 'volume-metadata' _save_vol_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:109 > 2023-03-10 16:42:42.147 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] No metadata type 'volume-metadata' available _save_vol_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:123 > 2023-03-10 16:42:42.148 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting metadata type 'volume-glance-metadata' _save_vol_glance_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:132 > 2023-03-10 16:42:42.156 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Completed fetching metadata type 'volume-glance-metadata' _save_vol_glance_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:145 > 2023-03-10 16:42:42.157 2878341 DEBUG cinder.backup.drivers.ceph [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Backing up metadata for volume 7e9beead-01a6-488e-bc2f-47093a8cdbd9. _backup_metadata /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:946 > 2023-03-10 16:42:42.251 2878341 DEBUG cinder.backup.drivers.ceph [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Backup '4f32e959-509d-49d9-9674-acc3776b4b6a' of volume 7e9beead-01a6-488e-bc2f-47093a8cdbd9 finished. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:1011 > 2023-03-10 16:42:42.252 2878341 DEBUG oslo_concurrency.processutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/dm-5 execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:42.613 2878341 DEBUG oslo_concurrency.processutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/dm-5" returned: 0 in 0.361s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:42.614 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] ==> disconnect_volume: call "{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7fe6925c3ac8>, {'target_discovered': False, 'discard': True, 'target_luns': [1, 1, 1, 1], 'target_iqns': ['iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'], 'target_portals': ['10.224.18.46:3260', '10.224.18.47:3260', '10.224.18.48:3260', '10.224.18.49:3260'], 'wwn': '3624a93705842cfae35d7483200015fce', 'qos_specs': {'total_bytes_sec': '524288000', 'read_iops_sec': '12800', 'write_iops_sec': '6400'}, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, {'type': 'block', 'scsi_wwn': '3624a93705842cfae35d7483200015fce', 'path': '/dev/dm-5', 'multipath_id': '3624a93705842cfae35d7483200015fce'}), 'kwargs': {'force': True, 'ignore_errors': True}}" trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:150 > 2023-03-10 16:42:42.616 2878341 DEBUG oslo_concurrency.lockutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > 2023-03-10 16:42:42.616 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting connected devices for (ips,iqns,luns)=[('10.224.18.46:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1), ('10.224.18.47:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1), ('10.224.18.48:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1), ('10.224.18.49:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1)] _get_connection_devices /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:821 > 2023-03-10 16:42:42.618 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:42.631 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:42.632 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('10.224.18.46:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n10.224.18.47:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n10.224.18.48:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n10.224.18.49:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:42:42.634 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:42.647 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:42.647 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('tcp: [234] 10.224.18.46:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\ntcp: [235] 10.224.18.47:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\ntcp: [236] 10.224.18.49:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\ntcp: [237] 10.224.18.48:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:42:42.649 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('-m', 'session'): stdout=tcp: [234] 10.224.18.46:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [235] 10.224.18.47:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [236] 10.224.18.49:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [237] 10.224.18.48:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > stderr= _run_iscsiadm_bare /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1155 > 2023-03-10 16:42:42.649 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsi session list stdout=tcp: [234] 10.224.18.46:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [235] 10.224.18.47:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [236] 10.224.18.49:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [237] 10.224.18.48:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > stderr= _run_iscsi_session /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1144 > 2023-03-10 16:42:42.653 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Resulting device map defaultdict(<function ISCSIConnector._get_connection_devices.<locals>.<lambda> at 0x7fe6921bf268>, {('10.224.18.46:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sdb'}, set()), ('10.224.18.47:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sda'}, set()), ('10.224.18.48:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sdc'}, set()), ('10.224.18.49:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sdd'}, set())}) _get_connection_devices /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:853 > 2023-03-10 16:42:42.654 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Removing multipathed devices sdc, sdd, sda, sdb remove_connection /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:308 > 2023-03-10 16:42:42.655 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flush multipath device 3624a93705842cfae35d7483200015fce flush_multipath_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:376 > 2023-03-10 16:42:42.657 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:46.675 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 4.019s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:46.676 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:42:46.676 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:46.685 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 20 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:42:55.994 2878341 DEBUG oslo_service.periodic_task [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Running periodic task BackupManager.publish_service_capabilities run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:211 > 2023-03-10 16:42:55.995 2878341 DEBUG cinder.manager [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python3.6/site-packages/cinder/manager.py:197 > 2023-03-10 16:43:06.707 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 20.031s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:06.707 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:43:06.707 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:06.714 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 40 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:43:46.753 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 40.046s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.754 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:474 > 2023-03-10 16:43:46.754 2880468 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140628271231200]: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:43:06 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' _process_cmd /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:490 > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 485, in _process_cmd > ret = func(*f_args, **f_kwargs) > File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 249, in _wrap > return func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root > return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute > on_completion=on_completion, *cmd, **kwargs) > File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute > cmd=sanitized_cmd) > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:43:06 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:43:46.757 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', ('', 'Mar 10 16:43:06 | 3624a93705842cfae35d7483200015fce is not a multipath device\n', 1, 'multipath -f 3624a93705842cfae35d7483200015fce', None)) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.758 2878341 WARNING os_brick.exception [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flushing 3624a93705842cfae35d7483200015fce failed: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > 2023-03-10 16:43:46.761 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sdc execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.772 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sdc" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.772 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.774 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sdc with /sys/block/sdc/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.774 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdc/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.814 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdc/device/delete" returned: 0 in 0.040s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.815 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.817 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sdd execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.828 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sdd" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.828 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.829 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sdd with /sys/block/sdd/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.831 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdd/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.869 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdd/device/delete" returned: 0 in 0.039s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.870 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.872 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sda execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.883 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sda" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.884 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.885 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sda with /sys/block/sda/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.887 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sda/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.929 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sda/device/delete" returned: 0 in 0.042s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.930 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.931 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sdb execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.942 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sdb" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.943 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.944 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.945 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.979 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.034s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.979 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.980 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Checking to see if SCSI volumes sdc, sdd, sda, sdb have been removed. wait_for_volumes_removal /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:83 > 2023-03-10 16:43:46.981 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] SCSI volumes sdc, sdd, sda, sdb have been removed. wait_for_volumes_removal /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:92 > 2023-03-10 16:43:46.982 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Disconnecting from: [('10.224.18.46:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'), ('10.224.18.47:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'), ('10.224.18.48:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'), ('10.224.18.49:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1')] _disconnect_connection /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1132 > 2023-03-10 16:43:46.983 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.996 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op update -n node.startup -v manual" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.996 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.997 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:46.998 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.026 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --logout" returned: 0 in 0.028s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.027 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260]\nLogout of [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.028 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260] > Logout of [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.029 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.041 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op delete" returned: 0 in 0.012s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.041 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.042 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.043 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.055 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op update -n node.startup -v manual" returned: 0 in 0.012s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.055 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.056 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.057 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.084 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --logout" returned: 0 in 0.026s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.084 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260]\nLogout of [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.086 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260] > Logout of [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.088 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.101 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op delete" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.102 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.103 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.104 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.114 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op update -n node.startup -v manual" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.114 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.115 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.116 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.140 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --logout" returned: 0 in 0.024s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.140 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260]\nLogout of [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.141 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260] > Logout of [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.142 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.153 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op delete" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.153 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.154 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.155 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.165 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op update -n node.startup -v manual" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.165 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.166 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.167 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.189 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --logout" returned: 0 in 0.022s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.190 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260]\nLogout of [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.191 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260] > Logout of [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.192 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.202 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op delete" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.202 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.203 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.204 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flushing again multipath 3624a93705842cfae35d7483200015fce now that we removed the devices. _cleanup_connection /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:941 > 2023-03-10 16:43:47.204 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flush multipath device 3624a93705842cfae35d7483200015fce flush_multipath_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:376 > 2023-03-10 16:43:47.205 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.215 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.216 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:43:47.216 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.221 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 20 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:43:55.994 2878341 DEBUG oslo_service.periodic_task [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Running periodic task BackupManager.publish_service_capabilities run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:211 > 2023-03-10 16:43:55.995 2878341 DEBUG cinder.manager [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python3.6/site-packages/cinder/manager.py:197 > 2023-03-10 16:44:07.242 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 20.026s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:44:07.242 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:44:07.242 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:44:07.249 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 40 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:44:47.287 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 40.045s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:44:47.287 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:474 > 2023-03-10 16:44:47.288 2880468 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140628271231200]: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' _process_cmd /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:490 > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 485, in _process_cmd > ret = func(*f_args, **f_kwargs) > File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 249, in _wrap > return func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root > return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute > on_completion=on_completion, *cmd, **kwargs) > File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute > cmd=sanitized_cmd) > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:44:47.288 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', ('', 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n', 1, 'multipath -f 3624a93705842cfae35d7483200015fce', None)) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:44:47.289 2878341 DEBUG oslo_concurrency.lockutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 124.674s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > 2023-03-10 16:44:47.289 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] <== disconnect_volume: exception (124674ms) ProcessExecutionError('', 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n', 1, 'multipath -f 3624a93705842cfae35d7483200015fce', None) trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:160 > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Traceback (most recent call last): > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in create_backup > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err)) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server self.force_reraise() > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server raise self.value > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in create_backup > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in _run_backup > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server ignore_errors=True) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066, in _detach_device > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server force=force, ignore_errors=ignore_errors) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 154, in trace_logging_wrapper > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 880, in disconnect_volume > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server is_disconnect_call=True) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 942, in _cleanup_connection > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server self._linuxscsi.flush_multipath_device(multipath_name) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line 382, in flush_multipath_device > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server root_helper=self._root_helper) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in _execute > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Command: multipath -f 3624a93705842cfae35d7483200015fce > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Exit code: 1 > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Stdout: '' > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server

3 4

OpenStack 2023.1 Antelope is officially released!
by Herve Beraud 22 Mar '23

22 Mar '23

Hello OpenStack community, I'm excited to announce the final releases for the components of OpenStack 2023.1 Antelope, which conclude the 2023.1 Antelope development cycle. You will find a complete list of all components, their latest versions, and links to individual project release notes documents listed on the new release site. https://releases.openstack.org/antelope/ Congratulations to all of the teams who have contributed to this release! Our next production cycle, 2023.2 Bobcat, has already started. We will meet at the Virtual Project Team Gathering, March 27-31, 2023, to plan the work for the upcoming cycle. I hope to see you there! Thanks, OpenStack Release Management team -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

6 5

OpenInfra Live episode: March 23, 2023, at 9 a.m. CT (14:00 UTC)
by Kristin Barrientos 22 Mar '23

22 Mar '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack community! Episode: OpenStack Antelope: A New Era The OpenStack community released Antelope, the 27th version of the world’s most widely deployed open source cloud infrastructure software, this week. Join us to learn about the latest from community leaders about what was delivered in Antelope and what we can expect in Bobcat, OpenStack's 28th release targeting October 2023. Speakers: Carlos Silva, Rajat Dhasmana, Sylvain Bauza, Jay Faulkner, Kendall Nelson Date and time: March 23, 2023, at 9 a.m. CT (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=YdLTUTyJ1eU LinkedIn: https://www.linkedin.com/events/7042534262494941185/comments/ WeChat: recording will be posted on OpenStack WeChat after the live stream Have an idea for a future episode? Share it now at ideas.openinfra.live. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[ptl][tc][ops][ptg] Operator + Developers interaction (operator-hours) slots in 2023.2 Bobcat PTG
by Ghanshyam Mann 22 Mar '23

22 Mar '23

Hello Everyone/PTL, To improve the interaction/feedback between operators and developers, one of the efforts is to schedule the 'operator-hour' in developers' events. We scheduled the 'operator-hour' in the last vPTG, which had mixed productivity feedback[1]. The TC discussed it and thinks we should continue the 'operator-hour' in March vPTG also. TC will not book the placeholder this time so that slots can be booked in the project room itself, and operators can join developers to have a joint discussion. But at the same time, we need to avoid slot conflict for operators. Every project needs to make sure its 'operator-hour' does not overlap with the related projects (integrated projects which might have common operators, for example. nova, cinder, neutron etc needs to avoid conflict) 'operator-hour'. Guidelines for the project team to book 'operator-hour' --------------------------------------------------------------------------------------- * Request in #openinfra-events IRC channel to register the new track 'operator-hour-<your project name>'. For example, 'operator-hour-nova' * Once the track is registered, find a spot in your project slots where no other project (which you think is related/integrated project and might have common operators) has already booked their operator-hour. Accordingly, book with the newly registered track 'operator-hour-<your project name>'. For example, #operator-hour-nova book essex-WedB1 . * Do not book more than one slot (1 hour) so that other projects will have enough slots open to book. If more discussion is needed on anything, it can be continued in project-specific slots. We request that every project book an 'operator hour' slot for operators to join your PTG session. For any query/conflict, ping TC in #openstack-tc or #openinfra-events IRC channel. [1] https://etherpad.opendev.org/p/Oct2022_PTGFeedback#L32 -gmann

2 3

[largescale-sig] Next meeting: March 22, 15utc
by Thierry Carrez 22 Mar '23

22 Mar '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday, the Antelope release day, in #openstack-operators on OFTC IRC, at 15UTC, our EU+US-friendly time. Since we currently are in DST hell, you should doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230322T15 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[ironic] Meeting cancelled April 10; PTL availabliity
by Jay Faulkner 22 Mar '23

22 Mar '23

After a quick poll of some Ironic contributors, it looks like most of us will be out April 10 for holiday or vacation. I'm cancelling the team meeting as we likely won't have quorum. Along those lines; I will be out the entire week of April 10. If you have anything that will need my attention specifically, please reach out before then. I trust any one of the PTLs-emeritus that are Ironic contributors to handle any event requiring PTL approval while I'm gone. Thanks, Jay Faulkner Ironic PTL

1 0

App Deployment to OpenStack (Alfresco Community)
by Konstantinos Asimakopoulos 22 Mar '23

22 Mar '23

Hello! I am new to OpenStack technology (cloud) and of course willing to dive into this interesting infrastructure. I would like to get information (step by step wiki) on how to deploy open source applications especially Alfresco Community Edition https://www.alfresco.com/ecm-software/alfresco-community-editions to openstack cloud and operate it globally. Is that possible and how? That would help me a lot! Regards Kostas

2 1

[kolla] next weekly meeting cancelled
by Michał Nasiadka 22 Mar '23

22 Mar '23

Hello Koalas, Next weekly meeting (29th March) is cancelled because of PTG on Mon/Tue/Thu - let’s meet there! Michal

1 0

[neutron] Neutron meetings cancelled next week
by Rodolfo Alonso Hernandez 22 Mar '23

22 Mar '23

Hello Neutrinos: The regular Neutron meetings (team, CI and drivers) will be cancelled next week because of the PTG. Join us during the next one! Regards.

1 0

[nova][neutron] Nova-Neutron PTG sessions
by Rodolfo Alonso Hernandez 22 Mar '23

22 Mar '23

Hello all: The Nova-Neutron PTG sessions will take place on Thursday, from 13UTC to 15UTC, in the Juno channel. Please check the agenda [1]. We have 3 topics: * (artom) delete_on_termination for Neutron ports * (dvo-plv) Blueprint: "Add support for Napatech LinkVirt SmartNICs" review * (ralonsoh, artom): https://bugs.launchpad.net/neutron/+bug/1986003 (How to handle the duplicated port binding activate request from Nova, both in Nova and Neutron) Regards. [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg

1 0

[neutron] Neutron PTG sessions schedule
by Rodolfo Alonso Hernandez 22 Mar '23

22 Mar '23

Hello all: Please check the Neutron sessions schedule for the PTG week [1]. All sessions will take place in Juno channel [2]. This is a summary of the sessions and days: * Tuesday (13UTC - 17UTC): retrospective, releases, migrations and project deprecations. * Wednesday (13UTC - 17UTC): new RFEs and operator hour. * Thursday (13UTC - 17UTC): Nova-Neutron sessions, ovn-bgp-agent roadmap and neutron-dynamic-routing RFE. * Friday (13UTC - 17UTC): open hour for core candidates!! If you have any questions, please reply to this email or ping me in IRC (#openstack-neutron, <ralonsoh>). Regards. [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg [2]https://ptg.opendev.org/ptg.html

1 0

[magnum] Secure RBAC implememtion
by Rico Lin 22 Mar '23

22 Mar '23

Hi magnum team I would like to make short report regarding the progress and trigger followup discussion Right now, with patch series https://review.opendev.org/c/openstack/magnum/+/874945 The patchset is a follow of tc goals: Consistent and Secure Default RBAC[1] We have now: * Implementation of Secure RBAC in project member and project reader for most APIs. And also add project scope check for APIs which is not design to run across multiple projects. * Unit test and functional test ready and passed for above features. The change of secure RBAC is currently default to false, so it will not affect on current running environments. And we should enable it in the following cycle. So what it does when not enable those configs are only provided deprecation warning. When enabled, we will requires project_reader role for perform any non-admin GET requests and project_member role for any other non-admin requests(PATCH, DELETE, POST, etc). And will also requires project scope token to allow perform those APIs. One of the patch we can discuss is to explicit set admin authorization to APIs in https://review.opendev.org/c/openstack/magnum/+/875625 This IMO, is an idea change to make sure we don't break admin operations on all APIs to avoid bugs like https://bugs.launchpad.net/neutron/+bug/1997089 , but if there are any other concerns, I would love to learn about it. The patch sets are ready, I think as we already in new developing cycle, would really like if anyone can help to review and landing them. Most of projects are already have these implementation in place, so now would be a good time for magnum to catch up with that goal. [1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… *Rico Lin*

1 0

[all][tc][policy] Canceling policy popup team next week meeting
by Ghanshyam Mann 22 Mar '23

22 Mar '23

Hello Everyone, Due to vPTG weeks, I am cancelling the policy pop-up next meeting scheduled for 28th Mar. https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[openstack][kolla-ansible]
by Nguyễn Hữu Khôi 22 Mar '23

22 Mar '23

Hello guys., I am using Xena(Ubuntu 20.04) and want to upgrade to zed. With my reading from https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html I will do it like that: Upgrade from Xena(20.04 container base) to Yoga(22.04 container base) then upgrade host to 22.04 >> Upgrade from Yoga to Zed.. *** Can you verify with me that we need kolla-ansible upgrade first then upgrade the host? Is that right? I just thought about mariadb, how we can verify schema upgrade and rabbitmq will crash with the new version? What will we do if something crashes like an upgrade database and rabbitmq failed? Thank you. Regards Nguyen Huu Khoi

1 0

PTG March 2023 Registration & Schedule
by Kendall Nelson 21 Mar '23

21 Mar '23

Hello Everyone! The March 2023 Project Teams Gathering is right around the corner (March 27-31) and the schedule is being setup by your team leads! Slots are going fast, so make sure to get your time booked ASAP if you haven't already! You can find the schedule and available slots on the PTGbot website [1]. The PTGbot site is the during-event website to keep track of what's being discussed and any last-minute schedule changes. It is driven via commands in the #openinfra-events IRC channel (on the OFTC network) where the PTGbot listens. If you have questions about the commands that you can give the bot, check out the documentation here[2]. Also, if you haven’t connected to IRC before, here are some docs on how to get setup![3] Lastly, please don't forget to register[4] (it is free after all!). Please let us know if you have any questions via email to ptg(a)openinfra.dev. Thanks! -Kendall (diablo_rojo) [1] PTGbot Site: https://ptg.opendev.org/ptg.html [2] PTGbot Documentation: https://github.com/openstack/ptgbot#open-infrastructure-ptg-bot [3] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [4] PTG Registration: https://openinfra-ptg.eventbrite.com

1 0

[horizon] Cancelling next two weekly meetings
by vishal manchanda 21 Mar '23

21 Mar '23

Hello Team, As agreed, during the last weekly meeting, we are canceling our weekly meeting on 22nd March and 29th March. The next weekly meeting will be on 5th April. See you at the PTG! Also, Please add topics for PTG discussion [1]. Thanks & regards, Vishal Manchanda [1] https://etherpad.opendev.org/p/horizon-bobcat-ptg

1 0

[vptg][ptg][openstack-ansible] Bobcat virtual Project Team Gathering and Operator Hours
by Dmitriy Rabotyagov 21 Mar '23

21 Mar '23

Hi everyone! I'm happy to inform you that the OpenStack-Ansible team is going to have a virtual PTG next Tuesday, on March 28 from 15:00 till 18:00 UTC in Kilo room [1]. Everyone who is interested in participating in further development or regarding project plans for the next releases are warmly welcome to join us. We're also continuing the tradition to have a project Operator Hours. So all operators or folks who are wondering about OpenStack-Ansible concept, designs or just want to share their experience with the project are warmly welcome to join us on Wednesday, March 29 from 17:00 till 18:00 UTC in Havana room [2] So add dates to your calendar and hope seeing/hearing everyone next week! [1] PTG room: https://www.openstack.org/ptg/rooms/kilo [2] Operator hours room: https://www.openstack.org/ptg/rooms/havana

1 0

[ironic] No meeting Monday 3/27/2023
by Jay Faulkner 21 Mar '23

21 Mar '23

Hello, Monday March 27, 2023 is when the vPTG is set to begin. I'm cancelling the Ironic weekly meeting to ensure any Ironic contributors can participate in sessions occurring Monday. Thanks! Jay Faulkner Ironic PTL

1 0

[tc][all] Technical Committee next weekly meeting on March 22 at 1600 UTC
by Nikolla, Kristi 21 Mar '23

21 Mar '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held tomorrow (March 22) at 1600 UTC on #openstack-tc on OFTC IRC. The meeting will be chaired by Kristi Nikolla. A copy of the agenda can be found below. Items can still be proposed by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting * Deciding on meeting time * Gate health check * 2023.2 cycle Leaderless projects ** https://etherpad.opendev.org/p/2023.2-leaderless * Virtual PTG Planning ** March 27-31, 2023, there's the Virtual PTG. ** https://etherpad.opendev.org/p/tc-2023-2-ptg * TC 2023.1 tracker status checks ** https://etherpad.opendev.org/p/tc-2023.1-tracker * Cleanup of PyPI maintainer list for OpenStack Projects ** Etherpad for audit and cleanup of additional PyPi maintainers *** https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup ** ML discussion *** https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031848… * Recurring tasks check ** Bare 'recheck' state *** https://etherpad.opendev.org/p/recheck-weekly-summary * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open There are no noted absences. Thank you, Kristi Nikolla

1 0

[CloudKitty] Virtual PTG March 2023
by Rafael Weingärtner 21 Mar '23

21 Mar '23

Hello everyone, As you probably heard our next PTG will be held virtually in March, 2023. We've marked March 31, at 13:00-15:00 UTC [1]. However, if you guys would like some other dates and/or time, just let me know. The room we selected is called "Bexar". I've also created an etherpad [2] to collect ideas/topics for the PTG sessions. If you have anything to discuss, please don't hesitate to write it there. [1] https://ptg.opendev.org/ptg.html [2] https://etherpad.opendev.org/p/march2023-ptg-cloudkitty -- Rafael Weingärtner

1 0

[Octavia][PTG] Bobcat PTG planning
by Gregory Thiemonge 21 Mar '23

21 Mar '23

Hi Folks, A reminder: the Octavia PTG will be on March 28th (14:00-18:00 UTC), There's a dedicated etherpad for this session: https://etherpad.opendev.org/p/bobcat-ptg-octavia Feel free to add your topic, Greg

1 0

[Glance]Weekly Meeting Cancelled for this week
by Pranali Deore 21 Mar '23

21 Mar '23

Hello, As discussed during last weekly meeting[1], Glance upstream weekly meeting for this week i.e., 23rd March, 2023 has been cancelled. See you all at PTG ! Thanks & Regards, Pranali [1]: https://meetings.opendev.org/irclogs/%23openstack-meeting/%23openstack-meet…

1 0

[neutron]
by Kamil Madac 21 Mar '23

21 Mar '23

Hi All, I'm in the process of planning a small public cloud based on OpenStack. I have quite experience with kolla-ansible deployments which use OVS networking and I have no issues with that. It works stable for my use cases (Vlan provider networks, DVR, tenant networks, floating IPs). For that new deployment I'm looking at OVN deployment which from what I read should be more performant (faster build of instances) and with ability to cover more networking features in OVN instead of needing external software like iptables/dnsmasq. Does anyone use OVN in production and what is your experience (pros/cons)? Is OVN mature enough to replace OVS in the production deployment (are there some basic features from OVS missing)? Thanks in advance for sharing the experience. -- Kamil Madac <https://kmadac.github.io/>

3 4

[ironic][ptg] vPTG scheduling
by Jay Faulkner 21 Mar '23

21 Mar '23

Hey all, The vPTG will be upon us soon, the week of March 27. I booked the following times on behalf of Ironic + BM SIG Operator hour, in accordance with what times worked in Antelope. It's my hope that since we've had little contributor turnover, these times continue to work. I'm completely open to having things moved around if it's more convenient to participants. I've booked the following times, all in Folsom: - Tuesday 1400 UTC - 1700 UTC - Wednesday 1300 UTC Operator hour: baremetal SIG - Wednesday 1400 UTC - 1600 UTC - Wednesday 2200 - 2300 UTC I propose that after the Ironic meeting on March 20, we shortly sync up in the Bobcat PTG etherpad (https://etherpad.opendev.org/p/ironic-bobcat-ptg) to pick topics and assign time. Again, this is all meant to be a suggestion, I'm happy to move things around but didn't want us to miss out on getting things booked. - Jay Faulkner Ironic PTL TC Member

2 6

[tacker] Cancelling next two IRC meetings
by Yasufumi Ogawa 21 Mar '23

21 Mar '23

Hi, I'd like to skip the next two IRC meetings due to a holiday tomorrow for many of us joining from Japan and next week for the bobcat vPTG. I'm looking forward to meet you guys on the vPTG! Cheers, Yasufumi

1 0

[neutron] Bug deputy report for week of March 13th
by Brian Haley 20 Mar '23

20 Mar '23

Hi, I was Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2011573 - [ovn-octavia-provider] Job pep8 failing due to bandit new lint rule - https://review.opendev.org/c/openstack/ovn-octavia-provider/+/877357 * https://bugs.launchpad.net/neutron/+bug/2011590 - Startup times for large OVN dbs is greatly increased by frozen_row() calls - https://review.opendev.org/c/openstack/neutron/+/877383 * https://bugs.launchpad.net/neutron/+bug/2011600 - functional test_get_datapath_id fails with neutron.common.utils.WaitTimeout: Timed out after 5 seconds - needs owner * https://bugs.launchpad.net/neutron/+bug/2011800 - ovn qos extension: update router does not remove no longer present qos rules - https://review.opendev.org/c/openstack/neutron/+/877603 (test only) - Needs code fix still Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/2011377 - test_agent_resync_on_non_existing_bridge failing intermittently sp - https://review.opendev.org/c/openstack/neutron/+/877535 * https://bugs.launchpad.net/neutron/+bug/2011724 - [OVN] Method "create_metadata_port" should pass the "fixed_ips" when creating the port - https://review.opendev.org/c/openstack/neutron/+/877528 * https://bugs.launchpad.net/neutron/+bug/2012104 - Neutron picking incorrect ovn records - Possibly related to https://bugs.launchpad.net/neutron/+bug/1951149 - OVN chassis deleted issue - does user need to manually clean-up? Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/2011687 - O flag is not enabled when ipv6_ra_mode is dhcpv6-stateful - O=1 is actually unnecessary in this case with M=1 based on the RFC, will need to figure out how to update code and/or docs to be in sync - https://review.opendev.org/c/openstack/neutron/+/877601 Misc bugs --------- * https://bugs.launchpad.net/neutron/+bug/2012144 - [OVN] adding/removing floating IPs neutron server errors about binding port - Mech driver shows ovn-bridge-mappings=[], but ovn-sbctl has them - Asked for more information Wishlist bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2011422 - [RFE] The Neutron agents should report the sync process status * https://bugs.launchpad.net/neutron/+bug/2012069 - [OVN] Flooding issue on provider networks with disabled port security - https://review.opendev.org/c/openstack/neutron/+/877675

1 0

[openstack-helm][tacker]
by TAKAHASHI TOSHIAKI(高橋　敏明) 20 Mar '23

20 Mar '23

Hi Openstack-helm team, I’m Toshiaki Takahashi, Tacker’s core developer. To my understanding, OpenStack-helm does not currently provide a Tacker’s helm chart. Recently there have been some requests to deploy Tacker with helm, and we would like to proceed with development of it if possible. Do we need to take any action such as to participate meeting of Openstack-helm project and propose our plan? Or, if OpenStack-helm is planning to have a PTG, I’d like to propose the plan at PTG, but is it planned? (I don't see any schedule at the moment). Regards, Toshiaki

2 3

[interop][ptg] Virtual Bobcat vPTG Planning
by Martin Kopec 20 Mar '23

20 Mar '23

Hello everyone, here is [1] our etherpad for the 2023.2 Bobcat PTG. Please, add your topics there if there is anything you would like to discuss / propose ... You can also vote for time slots for our session(s), so that they fit your schedule, at [2]. If you have any questions, feel free to reach out to me. [1] https://etherpad.opendev.org/p/bobcat-ptg-interop [2] https://framadate.org/2IPXOCvJNNoSGqHu Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[kolla-ansible][yoga][Magnum] Cannot attach cinder volume to pod
by wodel youchi 20 Mar '23

20 Mar '23

Hi, I am trying to attach a cinder volume to my pod, but it does not work. The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got : > Image: quay.io/k8scsi/csi-attacher:v2.0.0 > Image: quay.io/k8scsi/csi-provisioner:v1.4.0 > Image: quay.io/k8scsi/csi-snapshotter:v1.2.2 > Image: quay.io/k8scsi/csi-resizer:v0.3.0 > Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0 > Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0 > Image: > docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1 > Which 1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga : > Image: 10.0.0.165:4000/csi-attacher:v3.3.0 > Image: 10.0.0.165:4000/csi-provisioner:v3.0.0 > Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1 > Image: 10.0.0.165:4000/csi-resizer:v1.3.0 > Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 > Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0 > Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 > (cinder-csi-plugin:v1.23.0 which does not exists anymore) > 2 - And does not work, csi-cinder-controllerplugin keeps crashing. I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got : Volumes: html-volume: Type: Cinder (a Persistent Disk resource in OpenStack) VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1 FSType: ext4 ReadOnly: false SecretRef: nil kube-api-access-slqf4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- * Warning FailedMount 26m (x10 over 135m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting for the condition Warning FailedAttachVolume 3m39s (x40 over 146m) attachdetach-controller AttachVolume.Attach failed for volume "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning FailedMount 104s (x54 over 146m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the condition* "volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]} I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil> I0319 12:36:56.925658 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925687 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.947632 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:56.947646 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"} I0319 12:36:58.343821 1 connection.go:200] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}} I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil> I0319 12:36:58.343841 1 csi_handler.go:264] Attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.343848 1 util.go:38] Marking as attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: volumeattachments.storage.k8s.io <http://volumeattachments.storage.k8s.io> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io <http://storage.k8s.io>" at the cluster scope* I0319 12:36:58.348503 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348513 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348564 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:58.348567 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext: The I tried even the most updated images : > 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 > 10.0.0.165:4000/csi-provisioner:v3.4.0 > 10.0.0.165:4000/csi-resizer:v1.7.0 > 10.0.0.165:4000/csi-snapshotter:v6.2.1 > 10.0.0.165:4000/csi-attacher:v4.2.0 > 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0 > I had the same problem. Then I tried to use an older version of kubernetes : 1.21.11 with the older images shown above (following this link https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minu…) and it worked, the cinder volume was successfully mounted inside my nginx pod. - What is the meaning of the error I am having? - Is it magnum related or kubernetes related or both? Regards.

4 6

[Glance][PTG] Glance 2023.2 (Bobcat) vPTG Schedule
by Pranali Deore 20 Mar '23

20 Mar '23

Hello All, The 2023.2 (Bobcat) virtual PTG is going to start next week and we have created our PTG etherpad [1] and also added day wise topics along with timings we are going to discuss. Kindly let me know if you have any concerns with allotted time slots. Friday is reserved for any unplanned discussions. So please feel free to add your topics if you haven't added yet. As a reminder, these are the time slots for our discussion. Tuesday 28 MARCH 2023 1400 UTC to 1700 UTC Wednesday 29 MARCH 2023 1400 UTC to 1700 UTC Thursday 30 MARCH 2023 1400 UTC to 1700 UTC Friday 31 MARCH 2023 1400 UTC to 1700 UTC NOTE: We have scheduled glance operator hours on Thursday at 16:20 UTC(we can extend it if required), let us know your availability for the same. At the moment we don't have any sessions scheduled on Friday, if there are any last moment request(s)/topic(s) we will discuss that on Friday else we will conclude our PTG on Thursday 30th March. We will be using bluejeans for our discussion, kindly try to use it once before the actual discussion. The meeting URL is mentioned in etherpad [1] and will be the same throughout the PTG. [1]: https://etherpad.opendev.org/p/glance-bobcat-ptg Hope to see you there!! Thanks & Regards, Pranali Deore

1 0

[neutron] Extra routes
by Arnaud Morin 20 Mar '23

20 Mar '23

Hey all, When using DVR, is there any way to set extra-routes only on the snat network nodes? I want routes to apply only on north/south communication, not on east/west. I can't find something like this is API. Cheers,

1 0

[openstack][masakari] Ask about Masakari segment
by Nguyễn Hữu Khôi 19 Mar '23

19 Mar '23

Hello guys. I want to ask if I create two Masakari segments then instances will failover on only segment has a group of computes? Because I do test with this scenario, my instance failover on a different segment which has different compute hosts? Do I understand Masakari wrong? Thank you. Regards Nguyen Huu Khoi

2 3

Autoscaling in Kolla Ansible wallaby series
by Uday Dikshit 18 Mar '23

18 Mar '23

Hello Team I am looking forward to have an autoscaling feature on Kolla Ansible wallaby series openstack. I am using Senlin to create cluster, gnocchi for metrics and aodh for alarm. However I am facing issue with aodh alarm state as it gets stuck in the state in which it is. I also found once the load is hiking, the gnocchi metrics get inconsistent. Due to this also the alarm state sticks and do not trigger alarm. To solve this, i relied upon collectd. But collectd service does not push any metric from the Hypervisor to gnocchi. My objective is that, in case of a metric reaching the threshold, alarm should automatically trigger and cluster should scale up or down based on the load. Once the scale up or down task is successfully completed, alarm should return to normal state automatically. Does anybody have any experience with this use case or wanna propose any other solution? Thanks

1 0

[zun] allow zun to get information from blazar database
by James Leong 18 Mar '23

18 Mar '23

Hi all, I am using kolla-ansible for OpenStack deployment in the yoga version. Is It possible to allow zun to retrieve information from the blazar database in zun_api container? I have tried to include the blazar database connection information in the zun.conf file. However, when I try to use the newly added blazar information, I am getting the following error message. oslo_config.cfg.NoSuchOptError: no such option connectionTest in group [database] It seems like I have to set up some option somewhere else in the code. But I was not able to identify them. Thanks for your help James

1 0

[release] Release countdown for week R-0, March 20-24
by Herve Beraud 17 Mar '23

17 Mar '23

Development Focus ----------------- We will be releasing the coordinated OpenStack Antelope 2023.1 release next week, on March 22. Thanks to everyone involved in the Antelope 2023.1 cycle! We are now in pre-release freeze, so no new deliverable will be created until final release, unless a release-critical regression is spotted. Otherwise, teams attending the virtual PTG should start to plan what they will be discussing there, by creating and filling team etherpads. You can access the list of PTG etherpads at: http://ptg.openstack.org/etherpads.html General Information ------------------- On release day, the release team will produce final versions of deliverables following the cycle-with-rc release model, by re-tagging the commit used for the last RC. A patch doing just that will be proposed. PTLs and release liaisons should watch for that final release patch from the release team. While not required, we would appreciate having an ack from each team before we approve it on the 22nd, so that their approval is included in the metadata that goes onto the signed tag. Upcoming Deadlines & Dates -------------------------- Final Antelope 2023.1 release: March 22 Virtual PTG: March 27-31 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[horizon] Bobcat PTG Schedule
by vishal manchanda 17 Mar '23

17 Mar '23

Hello Team, Please Find the Schedule for Horizon Bobcat PTG in the eherpad [1]. Feel Free to add the topics you want to discuss in the PTG. Don't forget to register for PTG, if not done yet [2]. See you at the PTG! Thanks & Regards, Vishal Manchanda (irc: vishalmanchanda) [1] https://etherpad.opendev.org/p/horizon-bobcat-ptg [2] https://openinfra-ptg.eventbrite.com/

1 0

[kolla] Bobcat vPTG slots and topics
by Michał Nasiadka 17 Mar '23

17 Mar '23

Hello, Koalas! Allocated slots for Kolla sessions: 27-30 March 2023: Monday - 13.00 - 17.00 UTC (general, Kolla and Kolla-Ansible) Tuesday - 13.00 - 15.00 UTC (Kolla-Ansible) Tuesday - 15.00 - 17.00 UTC (Operator Hours Kolla) Thursday - 13.00 - 15.00 UTC (Kayobe) Please look at Kolla planning etherpad [1] and fill out topic proposals. Looking forward to meeting you! [1] https://etherpad.opendev.org/p/manila-bobcat-ptg-planning [2] https://ptg.opendev.org/ptg.html Thanks, mnasiadka

1 0

[kolla-ansible] migrate from OVS to OVN
by Nguyễn Hữu Khôi 17 Mar '23

17 Mar '23

Hello guys. Can we use kolla ansible to migrate from OVS to OVN? If then will it have downtime or impacts? Thank you much, Nguyen Huu Khoi

2 4

[magnum][kolla ansible]Ask about use higher magnum image on previous opentack version
by Nguyễn Hữu Khôi 17 Mar '23

17 Mar '23

Hello guys. I use Openstack Xena by using Kolla Ansible tool. due to some reason, I want to use Zed Magnum on my current system(Xena). Can I do this task by rebuilding Magnum images from code then retagging and replacing the Magnum container with new images? Any experience for this? Thank you very much. Nguyen Huu Khoi

1 0

Fwd: [kolla-ansible] Whether the cluster of two control nodes is available
by kaqiu pi 17 Mar '23

17 Mar '23

Hi， I'm a newer in kolla-ansibe. And I could deploy a cluster of two controll nodes by kolla-ansible. But I don't konw whether the cluster is anvailable? I would like to ask, when the number of control nodes is 2, the status of mariadb and rabbitmq clusters, are they safe and available for production? Thanks for any guidance. Good Luck

3 2

[outreachy][cinder] Running test in dev Enviornment
by Toheeb Oyekola 17 Mar '23

17 Mar '23

Hi everyone, I am having some error when i run "tox -e py3", here is a linke to the error https://paste.openstack.org/show/b5SAyZ35FdDuK2sRwTxD/.

2 2

Neutron
by Shrey Chauhan 16 Mar '23

16 Mar '23

2 1

[ironic] [infra] Cleaning up old IPA images from tarballs.o.o
by Dmitry Tantsur 16 Mar '23

16 Mar '23

Hi all, I would like to do a clean-up of old or wrongly created IPA images on the tarballs site. Before I do that, could you please check the proposed list to make sure we don't remove something we expect to be used? The list is https://paste.opendev.org/show/btDps0HFoYG9TKMmv1LB/ Dmitry

3 3

[neutron] PTG poll for meeting slots
by Rodolfo Alonso Hernandez 16 Mar '23

16 Mar '23

Hello Neutrinos: This is the link [1] (that I should have sent yesterday) to vote for the meeting slots during the PTG week. I think that 3 hours per day, from Tuesday to Friday, will be enough to cover the topics we need to discuss. Please feel free to send your vote here. Next week I'll close the poll and schedule the meetings. Thank you! [1]https://doodle.com/meeting/participate/id/eZ83mmvb

1 0

(OpenStack-horizon) unable to open horizon page after installing Open Stack
by Adivya Singh 16 Mar '23

16 Mar '23

Hi Team, I am unable to open Open OpenStack horizon page, after installation When i am opening the link , it says Haproxy service seems up and running, I have tried to Flush IP tables also, Seeing this might be causing the issue Port 443 is also listening. Any thoughts on this [image: image.png]

4 4

[Openstack] Lack of Balance solution such as Watcher.
by Nguyễn Hữu Khôi 16 Mar '23

16 Mar '23

I am building a private cloud.Everything is ok. But I cannot find a way to balance vm when It have heavy load which cause impact other vm on the compute node. I found solutions such as Watcher and Leveller. But they need to be done manually. Watcher is not good because It need cpu metric such as cpu load in Ceilometer which is removed so we cannot use it. Leveller is good, but It is obsolete. Nguyen Huu Khoi

4 7

[cinder] festival of feature reviews 17th March 2023
by Rajat Dhasmana 16 Mar '23

16 Mar '23

Hello Argonauts, We will be having our monthly festival of reviews tomorrow i.e. 17th March (Friday) from 1400-1600 UTC. We are close to the PTG but we still have backlog so good to have a head start in reviews for the next (2023.2) cycle. Following are some additional details: Date: 17th March, 2023 Time: 1400-1600 UTC Meeting link: https://bluejeans.com/556681290 etherpad: https://etherpad.opendev.org/p/cinder-festival-of-reviews See you there! Thanks Rajat Dhasmana

1 0

[cinder] Canceling upstream meeting 22nd March
by Rajat Dhasmana 16 Mar '23

16 Mar '23

Hello Argonauts, As discussed in this week's meeting[1], we will be canceling the Cinder upstream meeting next week i.e. 22nd March, 2023. Since we have RC2 this week, 2023.1 release next week and PTG after that, we don't expect many topics next week, but if you still have any, please add them to the PTG planning etherpad[2]. See you all at the PTG! [1] https://meetings.opendev.org/irclogs/%23openstack-meeting-alt/%23openstack-… [2] https://etherpad.opendev.org/p/bobcat-ptg-cinder-planning

1 0

Re: [manila] create snapshot from share not permitted
by Alvaro Soto 16 Mar '23

16 Mar '23

So you can test and get fresh log outputs, please do the following manila create <PARAMS> manila list manila show <PARAMS> Then go to https://pastebin.com/ and paste the CLI output and errors you see on the manila log files Friendly reminder: please click on 'reply all' so this thread can help more people Cheers! On Tue, Mar 14, 2023 at 1:53 AM garcetto <garcetto(a)gmail.com> wrote: > seems enabled...dont know why is not working... > > $ manila pool-list --detail > > +--------------------------------------+----------------------------------------------+ > | Property | Value > | > > +--------------------------------------+----------------------------------------------+ > | name | ostack-test@generic#GENERIC > | > | share_backend_name | GENERIC > | > | driver_handles_share_servers | True > | > | vendor_name | Open Source > | > | driver_version | 1.0 > | > | storage_protocol | NFS_CIFS > | > | total_capacity_gb | unknown > | > | free_capacity_gb | unknown > | > | reserved_percentage | 0 > | > | reserved_snapshot_percentage | 0 > | > | reserved_share_extend_percentage | 0 > | > | qos | False > | > | pools | None > | > | snapshot_support | True > | > | create_share_from_snapshot_support | True > | > | revert_to_snapshot_support | False > | > | mount_snapshot_support | False > | > | replication_domain | None > | > | filter_function | None > | > | goodness_function | None > | > | security_service_update_support | False > | > | network_allocation_update_support | False > | > | share_server_multiple_subnet_support | False > | > | max_shares_per_share_server | -1 > | > | max_share_server_size | -1 > | > | share_group_stats | {'consistent_snapshot_support': > None} | > | ipv4_support | True > | > | ipv6_support | False > | > | server_pools_mapping | > {'2242964f-be38-4f11-8c90-cdcfcd20c20a': []} | > | timestamp | 2023-03-13T09:13:06.331713 > | > > +--------------------------------------+----------------------------------------------+ > > > On Mon, Mar 13, 2023 at 11:45 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: > >> If you are inside this features support matrix >> >> >> https://docs.openstack.org/manila/latest/admin/share_back_ends_feature_supp… >> >> Examine your configuration as well: >> >> >> - >> >> snapshot_support indicates whether snapshots are supported for shares >> created on the pool/backend. When administrators do not set this capability >> as an extra-spec in a share type, the scheduler can place new shares of >> that type in pools without regard for whether snapshots are supported, and >> those shares will not support snapshots. >> >> >> https://docs.openstack.org/manila/latest/admin/capabilities_and_extra_specs… >> >> Cheers! >> >> On Mon, Mar 13, 2023 at 3:35 AM garcetto <garcetto(a)gmail.com> wrote: >> >>> good morning, >>> i am using manila and generic driver with dhss true, but cannot create >>> snapshot from shares, any help? where can i look at? >>> (cinder backend is a linux nfs server) >>> >>> thank you >>> >>> $ manila snapshot-create share-01 --name Snapshot1 >>> ERROR: Snapshots cannot be created for share >>> '2c8b1b3d-ef82-4372-94df-678539f0d843' since it does not have that >>> capability. (HTTP 422) (Request-ID: >>> req-cab23a46-37dc-4f2b-b26c-d6b21b7453ba) >>> >>> >> >> -- >> >> Alvaro Soto >> >> *Note: My work hours may not be your work hours. Please do not feel the >> need to respond during a time that is not convenient for you.* >> ---------------------------------------------------------- >> Great people talk about ideas, >> ordinary people talk about things, >> small people talk... about other people. >> > -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

2 1