March 2023 - openstack-discuss - lists.openstack.org

[ptl] Need PTL volunteer for OpenStack Sahara
by Ghanshyam Mann 22 Mar '23

22 Mar '23

Hi Qiu, I am reaching out to you as you were PTL for OpenStack Sahara project in the last cycle. There is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list. Please check if you or anyone you know would like to lead this project. - https://etherpad.opendev.org/p/2023.2-leaderless Also, if anyone else would like to help leading this project, this is time to let TC knows. -gmann

1 0

Re: [cinder] Error when creating backups from iscsi volume
by Gorka Eguileor 22 Mar '23

22 Mar '23

On 11/03, Rishat Azizov wrote: > Hi, Gorka, > > Thanks. I see multiple "multipath -f" calls. Logs in attachments. > Hi, There are multiple things going on here: 1. There is a bug in os-brick, because the disconnect_volume should not fail, since it is being called with force=True and ignore_errors=True. The issues is that this call [1] is not wrapped in the ExceptionChainer context manager, and it should not even be a flush call, it should be a call to "multipathd remove map $map" instead. 2. The way multipath code is written [2][3], the error we see about "3624a93705842cfae35d7483200015fce is not a multipath device" means 2 different things: it is not a multipath or an error happened. So we don't really know what happened without enabling more verbose multipathd log levels. 3. The "multipath -f" call should not be failing in the first place, because the failure is happening on disconnecting the source volume, which has no data buffered to be written and therefore no reason to fail the flush (unless it's using a friendly name). I don't know if it could be happening that the first flush fails with a timeout (maybe because there is an extend operation happening), but multipathd keeps trying to flush it in the background and when it succeeds it removes the multipath device, which makes following calls fail. If that's the case we would need to change the retry from automatic [4] to manual and check in-between to see if the device has been removed in-between calls. The first issue is definitely a bug, the 2nd one is something that could be changed in the deployment to try to get additional information on the failure, and the 3rd one could be a bug. I'll see if I can find someone who wants to work on the 1st and 3rd points. Cheers, Gorka. [1]: https://github.com/openstack/os-brick/blob/e15edf6c17449899ec8401c37482f7cb… [2]: https://github.com/opensvc/multipath-tools/blob/db4804bc7393f2482448bdd8701… [3]: https://github.com/opensvc/multipath-tools/blob/db4804bc7393f2482448bdd8701… [4]: https://github.com/openstack/os-brick/blob/e15edf6c17449899ec8401c37482f7cb… > > чт, 9 мар. 2023 г. в 15:55, Gorka Eguileor <geguileo(a)redhat.com>: > > > On 06/03, Rishat Azizov wrote: > > > Hi, > > > > > > It works with smaller volumes. > > > > > > multipath.conf attached to thist email. > > > > > > Cinder version - 18.2.0 Wallaby > > > > Hi, > > > > After giving it some thought I think I may know what is going on. > > > > If you have DEBUG logs enabled in cinder-backup when it fails, how many > > calls do you see in the cinder-backup to "multipath -f" from os-brick, > > only one or do you see more? > > > > Cheers, > > Gorka. > > > > > > > > пн, 6 мар. 2023 г. в 17:35, Gorka Eguileor <geguileo(a)redhat.com>: > > > > > > > On 16/02, Rishat Azizov wrote: > > > > > Hello! > > > > > > > > > > We have an error with creating backups from iscsi volume. Usually, > > this > > > > > happens with large backups over 100GB. > > > > > > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > [req-f6619913-6f96-4226-8d75-2da3fca722f1 > > > > 23de1b92e7674cf59486f07ac75b886b > > > > > a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message > > > > handling: > > > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > > > while > > > > > running command. > > > > > Command: multipath -f 3624a93705842cfae35d7483200015ec6 > > > > > Exit code: 1 > > > > > Stdout: '' > > > > > Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a > > > > > multipath device\n' > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > Traceback > > > > > (most recent call last): > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line > > > > 165, > > > > > in _process_incoming > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server res > > = > > > > > self.dispatcher.dispatch(message) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > > line > > > > > 309, in dispatch > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > self._do_dispatch(endpoint, method, ctxt, args) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", > > line > > > > > 229, in _do_dispatch > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > result = > > > > > func(ctxt, **new_args) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in > > wrapper > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > func(self, *args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 410, in > > > > > create_backup > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > volume_utils.update_backup_error(backup, str(err)) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, > > in > > > > > __exit__ > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > self.force_reraise() > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, > > in > > > > > force_reraise > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > raise > > > > > self.value > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 399, in > > > > > create_backup > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > updates > > > > = > > > > > self._run_backup(context, backup, volume) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 493, in > > > > > _run_backup > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > ignore_errors=True) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line > > 1066, > > > > in > > > > > _detach_device > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > force=force, ignore_errors=ignore_errors) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in > > > > > trace_logging_wrapper > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > f(*args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", > > line > > > > 360, > > > > > in inner > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > f(*args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > > > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > > > line 880, in disconnect_volume > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > is_disconnect_call=True) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > > > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > > > line 942, in _cleanup_connection > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > self._linuxscsi.flush_multipath_device(multipath_name) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", > > line > > > > > 382, in flush_multipath_device > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > root_helper=self._root_helper) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in > > > > > _execute > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > result = > > > > > self.__execute(*args, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", > > line > > > > > 172, in execute > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > execute_root(*cmd, **kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line > > > > 247, > > > > > in _wrap > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > return > > > > > self.channel.remote_call(name, args, kwargs) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, > > in > > > > > remote_call > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > raise > > > > > exc_type(*result[2]) > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > > > while > > > > > running command. > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command: > > > > > multipath -f 3624a93705842cfae35d7483200015ec6 > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit > > code: 1 > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: > > '' > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: > > 'Feb > > > > > 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath > > > > device\n' > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > > > > > > Could you please help with this error? > > > > > > > > Hi, > > > > > > > > Does it work for smaller volumes or does it also fail? > > > > > > > > What are your defaults in your /etc/multipath.conf file? > > > > > > > > What Cinder release are you using? > > > > > > > > Cheers, > > > > Gorka. > > > > > > > > > > > > > defaults { > > > user_friendly_names no > > > find_multipaths yes > > > enable_foreign "^$" > > > } > > > > > > blacklist_exceptions { > > > property "(SCSI_IDENT_|ID_WWN)" > > > } > > > > > > blacklist { > > > } > > > > > > devices { > > > device { > > > vendor "PURE" > > > product "FlashArray" > > > fast_io_fail_tmo 10 > > > path_grouping_policy "group_by_prio" > > > failback "immediate" > > > prio "alua" > > > hardware_handler "1 alua" > > > max_sectors_kb 4096 > > > } > > > } > > > > > 2023-03-10 16:42:41.785 2878341 DEBUG cinder.backup.drivers.ceph [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Transferred chunk 1600 of 1600 (84387K/s) _transfer_data /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:426 > 2023-03-10 16:42:42.107 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting metadata type 'volume-base-metadata' _save_vol_base_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:79 > 2023-03-10 16:42:42.139 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Completed fetching metadata type 'volume-base-metadata' _save_vol_base_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:98 > 2023-03-10 16:42:42.139 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting metadata type 'volume-metadata' _save_vol_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:109 > 2023-03-10 16:42:42.147 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] No metadata type 'volume-metadata' available _save_vol_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:123 > 2023-03-10 16:42:42.148 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting metadata type 'volume-glance-metadata' _save_vol_glance_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:132 > 2023-03-10 16:42:42.156 2878341 DEBUG cinder.backup.driver [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Completed fetching metadata type 'volume-glance-metadata' _save_vol_glance_meta /usr/lib/python3.6/site-packages/cinder/backup/driver.py:145 > 2023-03-10 16:42:42.157 2878341 DEBUG cinder.backup.drivers.ceph [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Backing up metadata for volume 7e9beead-01a6-488e-bc2f-47093a8cdbd9. _backup_metadata /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:946 > 2023-03-10 16:42:42.251 2878341 DEBUG cinder.backup.drivers.ceph [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Backup '4f32e959-509d-49d9-9674-acc3776b4b6a' of volume 7e9beead-01a6-488e-bc2f-47093a8cdbd9 finished. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:1011 > 2023-03-10 16:42:42.252 2878341 DEBUG oslo_concurrency.processutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/dm-5 execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:42.613 2878341 DEBUG oslo_concurrency.processutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/dm-5" returned: 0 in 0.361s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:42.614 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] ==> disconnect_volume: call "{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7fe6925c3ac8>, {'target_discovered': False, 'discard': True, 'target_luns': [1, 1, 1, 1], 'target_iqns': ['iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'], 'target_portals': ['10.224.18.46:3260', '10.224.18.47:3260', '10.224.18.48:3260', '10.224.18.49:3260'], 'wwn': '3624a93705842cfae35d7483200015fce', 'qos_specs': {'total_bytes_sec': '524288000', 'read_iops_sec': '12800', 'write_iops_sec': '6400'}, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, {'type': 'block', 'scsi_wwn': '3624a93705842cfae35d7483200015fce', 'path': '/dev/dm-5', 'multipath_id': '3624a93705842cfae35d7483200015fce'}), 'kwargs': {'force': True, 'ignore_errors': True}}" trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:150 > 2023-03-10 16:42:42.616 2878341 DEBUG oslo_concurrency.lockutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: waited 0.001s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > 2023-03-10 16:42:42.616 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Getting connected devices for (ips,iqns,luns)=[('10.224.18.46:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1), ('10.224.18.47:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1), ('10.224.18.48:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1), ('10.224.18.49:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1', 1)] _get_connection_devices /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:821 > 2023-03-10 16:42:42.618 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:42.631 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:42.632 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('10.224.18.46:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n10.224.18.47:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n10.224.18.48:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n10.224.18.49:3260,4294967295 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:42:42.634 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m session execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:42.647 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m session" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:42.647 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('tcp: [234] 10.224.18.46:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\ntcp: [235] 10.224.18.47:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\ntcp: [236] 10.224.18.49:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\ntcp: [237] 10.224.18.48:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash)\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:42:42.649 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('-m', 'session'): stdout=tcp: [234] 10.224.18.46:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [235] 10.224.18.47:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [236] 10.224.18.49:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [237] 10.224.18.48:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > stderr= _run_iscsiadm_bare /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1155 > 2023-03-10 16:42:42.649 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsi session list stdout=tcp: [234] 10.224.18.46:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [235] 10.224.18.47:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [236] 10.224.18.49:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > tcp: [237] 10.224.18.48:3260,4013 iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 (non-flash) > stderr= _run_iscsi_session /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1144 > 2023-03-10 16:42:42.653 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Resulting device map defaultdict(<function ISCSIConnector._get_connection_devices.<locals>.<lambda> at 0x7fe6921bf268>, {('10.224.18.46:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sdb'}, set()), ('10.224.18.47:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sda'}, set()), ('10.224.18.48:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sdc'}, set()), ('10.224.18.49:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'): ({'sdd'}, set())}) _get_connection_devices /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:853 > 2023-03-10 16:42:42.654 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Removing multipathed devices sdc, sdd, sda, sdb remove_connection /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:308 > 2023-03-10 16:42:42.655 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flush multipath device 3624a93705842cfae35d7483200015fce flush_multipath_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:376 > 2023-03-10 16:42:42.657 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:46.675 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 4.019s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:42:46.676 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:42:46.676 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:42:46.685 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 20 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:42:55.994 2878341 DEBUG oslo_service.periodic_task [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Running periodic task BackupManager.publish_service_capabilities run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:211 > 2023-03-10 16:42:55.995 2878341 DEBUG cinder.manager [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python3.6/site-packages/cinder/manager.py:197 > 2023-03-10 16:43:06.707 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 20.031s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:06.707 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:43:06.707 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:06.714 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 40 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:43:46.753 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 40.046s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.754 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:474 > 2023-03-10 16:43:46.754 2880468 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140628271231200]: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:43:06 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' _process_cmd /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:490 > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 485, in _process_cmd > ret = func(*f_args, **f_kwargs) > File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 249, in _wrap > return func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root > return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute > on_completion=on_completion, *cmd, **kwargs) > File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute > cmd=sanitized_cmd) > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:43:06 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:43:46.757 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', ('', 'Mar 10 16:43:06 | 3624a93705842cfae35d7483200015fce is not a multipath device\n', 1, 'multipath -f 3624a93705842cfae35d7483200015fce', None)) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.758 2878341 WARNING os_brick.exception [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flushing 3624a93705842cfae35d7483200015fce failed: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > 2023-03-10 16:43:46.761 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sdc execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.772 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sdc" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.772 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.774 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sdc with /sys/block/sdc/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.774 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdc/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.814 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdc/device/delete" returned: 0 in 0.040s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.815 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.817 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sdd execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.828 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sdd" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.828 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.829 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sdd with /sys/block/sdd/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.831 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdd/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.869 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdd/device/delete" returned: 0 in 0.039s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.870 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.872 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sda execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.883 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sda" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.884 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.885 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sda with /sys/block/sda/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.887 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sda/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.929 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sda/device/delete" returned: 0 in 0.042s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.930 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.931 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipathd del path /dev/sdb execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.942 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipathd del path /dev/sdb" returned: 0 in 0.011s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.943 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('ok\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.944 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Remove SCSI device /dev/sdb with /sys/block/sdb/device/delete remove_scsi_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:75 > 2023-03-10 16:43:46.945 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): tee -a /sys/block/sdb/device/delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.979 2880468 DEBUG oslo_concurrency.processutils [-] CMD "tee -a /sys/block/sdb/device/delete" returned: 0 in 0.034s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.979 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('1', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.980 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Checking to see if SCSI volumes sdc, sdd, sda, sdb have been removed. wait_for_volumes_removal /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:83 > 2023-03-10 16:43:46.981 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] SCSI volumes sdc, sdd, sda, sdb have been removed. wait_for_volumes_removal /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:92 > 2023-03-10 16:43:46.982 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Disconnecting from: [('10.224.18.46:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'), ('10.224.18.47:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'), ('10.224.18.48:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1'), ('10.224.18.49:3260', 'iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1')] _disconnect_connection /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1132 > 2023-03-10 16:43:46.983 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:46.996 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op update -n node.startup -v manual" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:46.996 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:46.997 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:46.998 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.026 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --logout" returned: 0 in 0.028s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.027 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260]\nLogout of [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.028 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260] > Logout of [sid: 234, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.46,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.029 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.041 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.46:3260 --op delete" returned: 0 in 0.012s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.041 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.042 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.043 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.055 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op update -n node.startup -v manual" returned: 0 in 0.012s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.055 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.056 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.057 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.084 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --logout" returned: 0 in 0.026s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.084 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260]\nLogout of [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.086 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260] > Logout of [sid: 235, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.47,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.088 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.101 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.47:3260 --op delete" returned: 0 in 0.013s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.102 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.103 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.104 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.114 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op update -n node.startup -v manual" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.114 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.115 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.116 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.140 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --logout" returned: 0 in 0.024s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.140 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260]\nLogout of [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.141 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260] > Logout of [sid: 237, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.48,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.142 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.153 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.48:3260 --op delete" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.153 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.154 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.155 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op update -n node.startup -v manual execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.165 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op update -n node.startup -v manual" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.165 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.166 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.167 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --logout execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.189 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --logout" returned: 0 in 0.022s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.190 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('Logging out of session [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260]\nLogout of [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260] successful.\n', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.191 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--logout',): stdout=Logging out of session [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260] > Logout of [sid: 236, target: iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1, portal: 10.224.18.49,3260] successful. > stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.192 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op delete execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.202 2880468 DEBUG oslo_concurrency.processutils [-] CMD "iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.55893eb505d1d2a1 -p 10.224.18.49:3260 --op delete" returned: 0 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.202 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (4, ('', '')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:43:47.203 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:1009 > 2023-03-10 16:43:47.204 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flushing again multipath 3624a93705842cfae35d7483200015fce now that we removed the devices. _cleanup_connection /usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py:941 > 2023-03-10 16:43:47.204 2878341 DEBUG os_brick.initiator.linuxscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Flush multipath device 3624a93705842cfae35d7483200015fce flush_multipath_device /usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py:376 > 2023-03-10 16:43:47.205 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.215 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 0.010s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:43:47.216 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:43:47.216 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:43:47.221 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 20 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:43:55.994 2878341 DEBUG oslo_service.periodic_task [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Running periodic task BackupManager.publish_service_capabilities run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:211 > 2023-03-10 16:43:55.995 2878341 DEBUG cinder.manager [req-b7d7b4ae-b092-40bf-9105-97bede752389 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python3.6/site-packages/cinder/manager.py:197 > 2023-03-10 16:44:07.242 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 20.026s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:44:07.242 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:478 > 2023-03-10 16:44:07.242 2880468 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -f 3624a93705842cfae35d7483200015fce execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2023-03-10 16:44:07.249 2880468 DEBUG os_brick.privileged.rootwrap [-] Sleeping for 40 seconds on_execute /usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py:106 > 2023-03-10 16:44:47.287 2880468 DEBUG oslo_concurrency.processutils [-] CMD "multipath -f 3624a93705842cfae35d7483200015fce" returned: 1 in 40.045s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > 2023-03-10 16:44:47.287 2880468 DEBUG oslo_concurrency.processutils [-] 'multipath -f 3624a93705842cfae35d7483200015fce' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:474 > 2023-03-10 16:44:47.288 2880468 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140628271231200]: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' _process_cmd /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:490 > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 485, in _process_cmd > ret = func(*f_args, **f_kwargs) > File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 249, in _wrap > return func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root > return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) > File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute > on_completion=on_completion, *cmd, **kwargs) > File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute > cmd=sanitized_cmd) > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:44:47.288 2880468 DEBUG oslo.privsep.daemon [-] privsep: reply[140628271231200]: (5, 'oslo_concurrency.processutils.ProcessExecutionError', ('', 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n', 1, 'multipath -f 3624a93705842cfae35d7483200015fce', None)) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:511 > 2023-03-10 16:44:47.289 2878341 DEBUG oslo_concurrency.lockutils [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume" :: held 124.674s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > 2023-03-10 16:44:47.289 2878341 DEBUG os_brick.initiator.connectors.iscsi [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] <== disconnect_volume: exception (124674ms) ProcessExecutionError('', 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n', 1, 'multipath -f 3624a93705842cfae35d7483200015fce', None) trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:160 > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server [req-bdebfdef-daf3-4250-8594-9ed91adb7c00 f91779ad06064ebfbeeff54de535a6cd 8a676a415f9541c59705a373a36b0ec4 - - -] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > Command: multipath -f 3624a93705842cfae35d7483200015fce > Exit code: 1 > Stdout: '' > Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Traceback (most recent call last): > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in create_backup > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err)) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server self.force_reraise() > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server raise self.value > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in create_backup > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in _run_backup > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server ignore_errors=True) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066, in _detach_device > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server force=force, ignore_errors=ignore_errors) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 154, in trace_logging_wrapper > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 880, in disconnect_volume > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server is_disconnect_call=True) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 942, in _cleanup_connection > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server self._linuxscsi.flush_multipath_device(multipath_name) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line 382, in flush_multipath_device > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server root_helper=self._root_helper) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in _execute > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Command: multipath -f 3624a93705842cfae35d7483200015fce > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Exit code: 1 > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Stdout: '' > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server Stderr: 'Mar 10 16:44:07 | 3624a93705842cfae35d7483200015fce is not a multipath device\n' > 2023-03-10 16:44:47.314 2878341 ERROR oslo_messaging.rpc.server

3 4

OpenStack 2023.1 Antelope is officially released!
by Herve Beraud 22 Mar '23

22 Mar '23

Hello OpenStack community, I'm excited to announce the final releases for the components of OpenStack 2023.1 Antelope, which conclude the 2023.1 Antelope development cycle. You will find a complete list of all components, their latest versions, and links to individual project release notes documents listed on the new release site. https://releases.openstack.org/antelope/ Congratulations to all of the teams who have contributed to this release! Our next production cycle, 2023.2 Bobcat, has already started. We will meet at the Virtual Project Team Gathering, March 27-31, 2023, to plan the work for the upcoming cycle. I hope to see you there! Thanks, OpenStack Release Management team -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

6 5

OpenInfra Live episode: March 23, 2023, at 9 a.m. CT (14:00 UTC)
by Kristin Barrientos 22 Mar '23

22 Mar '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack community! Episode: OpenStack Antelope: A New Era The OpenStack community released Antelope, the 27th version of the world’s most widely deployed open source cloud infrastructure software, this week. Join us to learn about the latest from community leaders about what was delivered in Antelope and what we can expect in Bobcat, OpenStack's 28th release targeting October 2023. Speakers: Carlos Silva, Rajat Dhasmana, Sylvain Bauza, Jay Faulkner, Kendall Nelson Date and time: March 23, 2023, at 9 a.m. CT (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=YdLTUTyJ1eU LinkedIn: https://www.linkedin.com/events/7042534262494941185/comments/ WeChat: recording will be posted on OpenStack WeChat after the live stream Have an idea for a future episode? Share it now at ideas.openinfra.live. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[ptl][tc][ops][ptg] Operator + Developers interaction (operator-hours) slots in 2023.2 Bobcat PTG
by Ghanshyam Mann 22 Mar '23

22 Mar '23

Hello Everyone/PTL, To improve the interaction/feedback between operators and developers, one of the efforts is to schedule the 'operator-hour' in developers' events. We scheduled the 'operator-hour' in the last vPTG, which had mixed productivity feedback[1]. The TC discussed it and thinks we should continue the 'operator-hour' in March vPTG also. TC will not book the placeholder this time so that slots can be booked in the project room itself, and operators can join developers to have a joint discussion. But at the same time, we need to avoid slot conflict for operators. Every project needs to make sure its 'operator-hour' does not overlap with the related projects (integrated projects which might have common operators, for example. nova, cinder, neutron etc needs to avoid conflict) 'operator-hour'. Guidelines for the project team to book 'operator-hour' --------------------------------------------------------------------------------------- * Request in #openinfra-events IRC channel to register the new track 'operator-hour-<your project name>'. For example, 'operator-hour-nova' * Once the track is registered, find a spot in your project slots where no other project (which you think is related/integrated project and might have common operators) has already booked their operator-hour. Accordingly, book with the newly registered track 'operator-hour-<your project name>'. For example, #operator-hour-nova book essex-WedB1 . * Do not book more than one slot (1 hour) so that other projects will have enough slots open to book. If more discussion is needed on anything, it can be continued in project-specific slots. We request that every project book an 'operator hour' slot for operators to join your PTG session. For any query/conflict, ping TC in #openstack-tc or #openinfra-events IRC channel. [1] https://etherpad.opendev.org/p/Oct2022_PTGFeedback#L32 -gmann

2 3

[largescale-sig] Next meeting: March 22, 15utc
by Thierry Carrez 22 Mar '23

22 Mar '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday, the Antelope release day, in #openstack-operators on OFTC IRC, at 15UTC, our EU+US-friendly time. Since we currently are in DST hell, you should doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230322T15 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[ironic] Meeting cancelled April 10; PTL availabliity
by Jay Faulkner 22 Mar '23

22 Mar '23

After a quick poll of some Ironic contributors, it looks like most of us will be out April 10 for holiday or vacation. I'm cancelling the team meeting as we likely won't have quorum. Along those lines; I will be out the entire week of April 10. If you have anything that will need my attention specifically, please reach out before then. I trust any one of the PTLs-emeritus that are Ironic contributors to handle any event requiring PTL approval while I'm gone. Thanks, Jay Faulkner Ironic PTL

1 0

App Deployment to OpenStack (Alfresco Community)
by Konstantinos Asimakopoulos 22 Mar '23

22 Mar '23

Hello! I am new to OpenStack technology (cloud) and of course willing to dive into this interesting infrastructure. I would like to get information (step by step wiki) on how to deploy open source applications especially Alfresco Community Edition https://www.alfresco.com/ecm-software/alfresco-community-editions to openstack cloud and operate it globally. Is that possible and how? That would help me a lot! Regards Kostas

2 1

[kolla] next weekly meeting cancelled
by Michał Nasiadka 22 Mar '23

22 Mar '23

Hello Koalas, Next weekly meeting (29th March) is cancelled because of PTG on Mon/Tue/Thu - let’s meet there! Michal

1 0

[neutron] Neutron meetings cancelled next week
by Rodolfo Alonso Hernandez 22 Mar '23

22 Mar '23

Hello Neutrinos: The regular Neutron meetings (team, CI and drivers) will be cancelled next week because of the PTG. Join us during the next one! Regards.

1 0

[nova][neutron] Nova-Neutron PTG sessions
by Rodolfo Alonso Hernandez 22 Mar '23

22 Mar '23

Hello all: The Nova-Neutron PTG sessions will take place on Thursday, from 13UTC to 15UTC, in the Juno channel. Please check the agenda [1]. We have 3 topics: * (artom) delete_on_termination for Neutron ports * (dvo-plv) Blueprint: "Add support for Napatech LinkVirt SmartNICs" review * (ralonsoh, artom): https://bugs.launchpad.net/neutron/+bug/1986003 (How to handle the duplicated port binding activate request from Nova, both in Nova and Neutron) Regards. [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg

1 0

[neutron] Neutron PTG sessions schedule
by Rodolfo Alonso Hernandez 22 Mar '23

22 Mar '23

Hello all: Please check the Neutron sessions schedule for the PTG week [1]. All sessions will take place in Juno channel [2]. This is a summary of the sessions and days: * Tuesday (13UTC - 17UTC): retrospective, releases, migrations and project deprecations. * Wednesday (13UTC - 17UTC): new RFEs and operator hour. * Thursday (13UTC - 17UTC): Nova-Neutron sessions, ovn-bgp-agent roadmap and neutron-dynamic-routing RFE. * Friday (13UTC - 17UTC): open hour for core candidates!! If you have any questions, please reply to this email or ping me in IRC (#openstack-neutron, <ralonsoh>). Regards. [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg [2]https://ptg.opendev.org/ptg.html

1 0

[magnum] Secure RBAC implememtion
by Rico Lin 22 Mar '23

22 Mar '23

Hi magnum team I would like to make short report regarding the progress and trigger followup discussion Right now, with patch series https://review.opendev.org/c/openstack/magnum/+/874945 The patchset is a follow of tc goals: Consistent and Secure Default RBAC[1] We have now: * Implementation of Secure RBAC in project member and project reader for most APIs. And also add project scope check for APIs which is not design to run across multiple projects. * Unit test and functional test ready and passed for above features. The change of secure RBAC is currently default to false, so it will not affect on current running environments. And we should enable it in the following cycle. So what it does when not enable those configs are only provided deprecation warning. When enabled, we will requires project_reader role for perform any non-admin GET requests and project_member role for any other non-admin requests(PATCH, DELETE, POST, etc). And will also requires project scope token to allow perform those APIs. One of the patch we can discuss is to explicit set admin authorization to APIs in https://review.opendev.org/c/openstack/magnum/+/875625 This IMO, is an idea change to make sure we don't break admin operations on all APIs to avoid bugs like https://bugs.launchpad.net/neutron/+bug/1997089 , but if there are any other concerns, I would love to learn about it. The patch sets are ready, I think as we already in new developing cycle, would really like if anyone can help to review and landing them. Most of projects are already have these implementation in place, so now would be a good time for magnum to catch up with that goal. [1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… *Rico Lin*

1 0

[all][tc][policy] Canceling policy popup team next week meeting
by Ghanshyam Mann 22 Mar '23

22 Mar '23

Hello Everyone, Due to vPTG weeks, I am cancelling the policy pop-up next meeting scheduled for 28th Mar. https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[openstack][kolla-ansible]
by Nguyễn Hữu Khôi 22 Mar '23

22 Mar '23

Hello guys., I am using Xena(Ubuntu 20.04) and want to upgrade to zed. With my reading from https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html I will do it like that: Upgrade from Xena(20.04 container base) to Yoga(22.04 container base) then upgrade host to 22.04 >> Upgrade from Yoga to Zed.. *** Can you verify with me that we need kolla-ansible upgrade first then upgrade the host? Is that right? I just thought about mariadb, how we can verify schema upgrade and rabbitmq will crash with the new version? What will we do if something crashes like an upgrade database and rabbitmq failed? Thank you. Regards Nguyen Huu Khoi

1 0

PTG March 2023 Registration & Schedule
by Kendall Nelson 21 Mar '23

21 Mar '23

Hello Everyone! The March 2023 Project Teams Gathering is right around the corner (March 27-31) and the schedule is being setup by your team leads! Slots are going fast, so make sure to get your time booked ASAP if you haven't already! You can find the schedule and available slots on the PTGbot website [1]. The PTGbot site is the during-event website to keep track of what's being discussed and any last-minute schedule changes. It is driven via commands in the #openinfra-events IRC channel (on the OFTC network) where the PTGbot listens. If you have questions about the commands that you can give the bot, check out the documentation here[2]. Also, if you haven’t connected to IRC before, here are some docs on how to get setup![3] Lastly, please don't forget to register[4] (it is free after all!). Please let us know if you have any questions via email to ptg(a)openinfra.dev. Thanks! -Kendall (diablo_rojo) [1] PTGbot Site: https://ptg.opendev.org/ptg.html [2] PTGbot Documentation: https://github.com/openstack/ptgbot#open-infrastructure-ptg-bot [3] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [4] PTG Registration: https://openinfra-ptg.eventbrite.com

1 0

[horizon] Cancelling next two weekly meetings
by vishal manchanda 21 Mar '23

21 Mar '23

Hello Team, As agreed, during the last weekly meeting, we are canceling our weekly meeting on 22nd March and 29th March. The next weekly meeting will be on 5th April. See you at the PTG! Also, Please add topics for PTG discussion [1]. Thanks & regards, Vishal Manchanda [1] https://etherpad.opendev.org/p/horizon-bobcat-ptg

1 0

[vptg][ptg][openstack-ansible] Bobcat virtual Project Team Gathering and Operator Hours
by Dmitriy Rabotyagov 21 Mar '23

21 Mar '23

Hi everyone! I'm happy to inform you that the OpenStack-Ansible team is going to have a virtual PTG next Tuesday, on March 28 from 15:00 till 18:00 UTC in Kilo room [1]. Everyone who is interested in participating in further development or regarding project plans for the next releases are warmly welcome to join us. We're also continuing the tradition to have a project Operator Hours. So all operators or folks who are wondering about OpenStack-Ansible concept, designs or just want to share their experience with the project are warmly welcome to join us on Wednesday, March 29 from 17:00 till 18:00 UTC in Havana room [2] So add dates to your calendar and hope seeing/hearing everyone next week! [1] PTG room: https://www.openstack.org/ptg/rooms/kilo [2] Operator hours room: https://www.openstack.org/ptg/rooms/havana

1 0

[ironic] No meeting Monday 3/27/2023
by Jay Faulkner 21 Mar '23

21 Mar '23

Hello, Monday March 27, 2023 is when the vPTG is set to begin. I'm cancelling the Ironic weekly meeting to ensure any Ironic contributors can participate in sessions occurring Monday. Thanks! Jay Faulkner Ironic PTL

1 0

[tc][all] Technical Committee next weekly meeting on March 22 at 1600 UTC
by Nikolla, Kristi 21 Mar '23

21 Mar '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held tomorrow (March 22) at 1600 UTC on #openstack-tc on OFTC IRC. The meeting will be chaired by Kristi Nikolla. A copy of the agenda can be found below. Items can still be proposed by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting * Deciding on meeting time * Gate health check * 2023.2 cycle Leaderless projects ** https://etherpad.opendev.org/p/2023.2-leaderless * Virtual PTG Planning ** March 27-31, 2023, there's the Virtual PTG. ** https://etherpad.opendev.org/p/tc-2023-2-ptg * TC 2023.1 tracker status checks ** https://etherpad.opendev.org/p/tc-2023.1-tracker * Cleanup of PyPI maintainer list for OpenStack Projects ** Etherpad for audit and cleanup of additional PyPi maintainers *** https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup ** ML discussion *** https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031848… * Recurring tasks check ** Bare 'recheck' state *** https://etherpad.opendev.org/p/recheck-weekly-summary * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open There are no noted absences. Thank you, Kristi Nikolla

1 0

[CloudKitty] Virtual PTG March 2023
by Rafael Weingärtner 21 Mar '23

21 Mar '23

Hello everyone, As you probably heard our next PTG will be held virtually in March, 2023. We've marked March 31, at 13:00-15:00 UTC [1]. However, if you guys would like some other dates and/or time, just let me know. The room we selected is called "Bexar". I've also created an etherpad [2] to collect ideas/topics for the PTG sessions. If you have anything to discuss, please don't hesitate to write it there. [1] https://ptg.opendev.org/ptg.html [2] https://etherpad.opendev.org/p/march2023-ptg-cloudkitty -- Rafael Weingärtner

1 0

[Octavia][PTG] Bobcat PTG planning
by Gregory Thiemonge 21 Mar '23

21 Mar '23

Hi Folks, A reminder: the Octavia PTG will be on March 28th (14:00-18:00 UTC), There's a dedicated etherpad for this session: https://etherpad.opendev.org/p/bobcat-ptg-octavia Feel free to add your topic, Greg

1 0

[Glance]Weekly Meeting Cancelled for this week
by Pranali Deore 21 Mar '23

21 Mar '23

Hello, As discussed during last weekly meeting[1], Glance upstream weekly meeting for this week i.e., 23rd March, 2023 has been cancelled. See you all at PTG ! Thanks & Regards, Pranali [1]: https://meetings.opendev.org/irclogs/%23openstack-meeting/%23openstack-meet…

1 0

[neutron]
by Kamil Madac 21 Mar '23

21 Mar '23

Hi All, I'm in the process of planning a small public cloud based on OpenStack. I have quite experience with kolla-ansible deployments which use OVS networking and I have no issues with that. It works stable for my use cases (Vlan provider networks, DVR, tenant networks, floating IPs). For that new deployment I'm looking at OVN deployment which from what I read should be more performant (faster build of instances) and with ability to cover more networking features in OVN instead of needing external software like iptables/dnsmasq. Does anyone use OVN in production and what is your experience (pros/cons)? Is OVN mature enough to replace OVS in the production deployment (are there some basic features from OVS missing)? Thanks in advance for sharing the experience. -- Kamil Madac <https://kmadac.github.io/>

3 4

[ironic][ptg] vPTG scheduling
by Jay Faulkner 20 Mar '23

20 Mar '23

Hey all, The vPTG will be upon us soon, the week of March 27. I booked the following times on behalf of Ironic + BM SIG Operator hour, in accordance with what times worked in Antelope. It's my hope that since we've had little contributor turnover, these times continue to work. I'm completely open to having things moved around if it's more convenient to participants. I've booked the following times, all in Folsom: - Tuesday 1400 UTC - 1700 UTC - Wednesday 1300 UTC Operator hour: baremetal SIG - Wednesday 1400 UTC - 1600 UTC - Wednesday 2200 - 2300 UTC I propose that after the Ironic meeting on March 20, we shortly sync up in the Bobcat PTG etherpad (https://etherpad.opendev.org/p/ironic-bobcat-ptg) to pick topics and assign time. Again, this is all meant to be a suggestion, I'm happy to move things around but didn't want us to miss out on getting things booked. - Jay Faulkner Ironic PTL TC Member

2 6

[tacker] Cancelling next two IRC meetings
by Yasufumi Ogawa 20 Mar '23

20 Mar '23

Hi, I'd like to skip the next two IRC meetings due to a holiday tomorrow for many of us joining from Japan and next week for the bobcat vPTG. I'm looking forward to meet you guys on the vPTG! Cheers, Yasufumi

1 0

[neutron] Bug deputy report for week of March 13th
by Brian Haley 20 Mar '23

20 Mar '23

Hi, I was Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2011573 - [ovn-octavia-provider] Job pep8 failing due to bandit new lint rule - https://review.opendev.org/c/openstack/ovn-octavia-provider/+/877357 * https://bugs.launchpad.net/neutron/+bug/2011590 - Startup times for large OVN dbs is greatly increased by frozen_row() calls - https://review.opendev.org/c/openstack/neutron/+/877383 * https://bugs.launchpad.net/neutron/+bug/2011600 - functional test_get_datapath_id fails with neutron.common.utils.WaitTimeout: Timed out after 5 seconds - needs owner * https://bugs.launchpad.net/neutron/+bug/2011800 - ovn qos extension: update router does not remove no longer present qos rules - https://review.opendev.org/c/openstack/neutron/+/877603 (test only) - Needs code fix still Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/2011377 - test_agent_resync_on_non_existing_bridge failing intermittently sp - https://review.opendev.org/c/openstack/neutron/+/877535 * https://bugs.launchpad.net/neutron/+bug/2011724 - [OVN] Method "create_metadata_port" should pass the "fixed_ips" when creating the port - https://review.opendev.org/c/openstack/neutron/+/877528 * https://bugs.launchpad.net/neutron/+bug/2012104 - Neutron picking incorrect ovn records - Possibly related to https://bugs.launchpad.net/neutron/+bug/1951149 - OVN chassis deleted issue - does user need to manually clean-up? Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/2011687 - O flag is not enabled when ipv6_ra_mode is dhcpv6-stateful - O=1 is actually unnecessary in this case with M=1 based on the RFC, will need to figure out how to update code and/or docs to be in sync - https://review.opendev.org/c/openstack/neutron/+/877601 Misc bugs --------- * https://bugs.launchpad.net/neutron/+bug/2012144 - [OVN] adding/removing floating IPs neutron server errors about binding port - Mech driver shows ovn-bridge-mappings=[], but ovn-sbctl has them - Asked for more information Wishlist bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2011422 - [RFE] The Neutron agents should report the sync process status * https://bugs.launchpad.net/neutron/+bug/2012069 - [OVN] Flooding issue on provider networks with disabled port security - https://review.opendev.org/c/openstack/neutron/+/877675

1 0

[openstack-helm][tacker]
by TAKAHASHI TOSHIAKI(高橋　敏明) 20 Mar '23

20 Mar '23

Hi Openstack-helm team, I’m Toshiaki Takahashi, Tacker’s core developer. To my understanding, OpenStack-helm does not currently provide a Tacker’s helm chart. Recently there have been some requests to deploy Tacker with helm, and we would like to proceed with development of it if possible. Do we need to take any action such as to participate meeting of Openstack-helm project and propose our plan? Or, if OpenStack-helm is planning to have a PTG, I’d like to propose the plan at PTG, but is it planned? (I don't see any schedule at the moment). Regards, Toshiaki

2 3

[interop][ptg] Virtual Bobcat vPTG Planning
by Martin Kopec 20 Mar '23

20 Mar '23

Hello everyone, here is [1] our etherpad for the 2023.2 Bobcat PTG. Please, add your topics there if there is anything you would like to discuss / propose ... You can also vote for time slots for our session(s), so that they fit your schedule, at [2]. If you have any questions, feel free to reach out to me. [1] https://etherpad.opendev.org/p/bobcat-ptg-interop [2] https://framadate.org/2IPXOCvJNNoSGqHu Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[kolla-ansible][yoga][Magnum] Cannot attach cinder volume to pod
by wodel youchi 20 Mar '23

20 Mar '23

Hi, I am trying to attach a cinder volume to my pod, but it does not work. The long story, the default version of kubernetes used in Yoga is 1.23.3 fcore35. When creating a default kubernetes cluster we got : > Image: quay.io/k8scsi/csi-attacher:v2.0.0 > Image: quay.io/k8scsi/csi-provisioner:v1.4.0 > Image: quay.io/k8scsi/csi-snapshotter:v1.2.2 > Image: quay.io/k8scsi/csi-resizer:v0.3.0 > Image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0 > Image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0 > Image: > docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.18.1 > Which 1 - Does not correspond to the documentation of Magnum, the documentation states these defaults for yoga : > Image: 10.0.0.165:4000/csi-attacher:v3.3.0 > Image: 10.0.0.165:4000/csi-provisioner:v3.0.0 > Image: 10.0.0.165:4000/csi-snapshotter:v4.2.1 > Image: 10.0.0.165:4000/csi-resizer:v1.3.0 > Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 > Image: 10.0.0.165:4000/csi-node-driver-registrar:v2.4.0 > Image: 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 > (cinder-csi-plugin:v1.23.0 which does not exists anymore) > 2 - And does not work, csi-cinder-controllerplugin keeps crashing. I tried to use the updates images (using a local registry), but I couldn't attach the cinder-volume, I got : Volumes: html-volume: Type: Cinder (a Persistent Disk resource in OpenStack) VolumeID: f780cb46-ed2a-405d-b901-7201b49c3df1 FSType: ext4 ReadOnly: false SecretRef: nil kube-api-access-slqf4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- * Warning FailedMount 26m (x10 over 135m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[kube-api-access-slqf4 html-volume]: timed out waiting for the condition Warning FailedAttachVolume 3m39s (x40 over 146m) attachdetach-controller AttachVolume.Attach failed for volume "cinder.csi.openstack.org-f780cb46-ed2a-405d-b901-7201b49c3df1" : Attach timeout for volume f780cb46-ed2a-405d-b901-7201b49c3df1 Warning FailedMount 104s (x54 over 146m) kubelet Unable to attach or mount volumes: unmounted volumes=[html-volume], unattached volumes=[html-volume kube-api-access-slqf4]: timed out waiting for the condition* "volume":{"capacity_bytes":5368709120,"volume_id":"7e377933-4ae6-47b7-a685-f484d35153af"}},{"status":{"published_node_ids":["c2531ccf-842e-44d1-85bd-72c811cea199"]},"volume":{"capacity_bytes":1073741824,"volume_id":"f9d5273b-e73d-4b37-8b50-1fcecb910b2a"}}]} I0319 12:36:50.910443 1 connection.go:201] GRPC error: <nil> I0319 12:36:56.925658 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925682 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925687 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925691 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925705 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:56.925828 1 csi_handler.go:312] VA finalizer added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.925836 1 csi_handler.go:326] NodeID annotation added to "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:56.947632 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:56.947646 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"f780cb46-ed2a-405d-b901-7201b49c3df1"} I0319 12:36:58.343821 1 connection.go:200] GRPC response: {"publish_context":{"DevicePath":"/dev/vdc"}} I0319 12:36:58.343834 1 connection.go:201] GRPC error: <nil> I0319 12:36:58.343841 1 csi_handler.go:264] Attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.343848 1 util.go:38] Marking as attached "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" *I0319 12:36:58.348467 1 csi_handler.go:234] Error processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7": failed to mark as attached: volumeattachments.storage.k8s.io <http://volumeattachments.storage.k8s.io> "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" is forbidden: User "system:serviceaccount:kube-system:csi-cinder-controller-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io <http://storage.k8s.io>" at the cluster scope* I0319 12:36:58.348503 1 controller.go:210] Started VA processing "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348509 1 csi_handler.go:224] CSIHandler: processing VA "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348513 1 csi_handler.go:251] Attaching "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348517 1 csi_handler.go:421] Starting attach operation for "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348525 1 csi_handler.go:740] Found NodeID 472bf42d-5ce0-4751-8fec-57bede0024d6 in CSINode k8intcalnewer-56bgom6jntbm-node-0 I0319 12:36:58.348540 1 csi_handler.go:304] VA finalizer is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348552 1 csi_handler.go:318] NodeID annotation is already set on "csi-9f81405424dc2cf210b6465f8b649ef20f85024f169b660fab235c03f64753b7" I0319 12:36:58.348564 1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0319 12:36:58.348567 1 connection.go:194] GRPC request: {"node_id":"472bf42d-5ce0-4751-8fec-57bede0024d6","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext: The I tried even the most updated images : > 10.0.0.165:4000/cinder-csi-plugin:v1.26.2 > 10.0.0.165:4000/csi-provisioner:v3.4.0 > 10.0.0.165:4000/csi-resizer:v1.7.0 > 10.0.0.165:4000/csi-snapshotter:v6.2.1 > 10.0.0.165:4000/csi-attacher:v4.2.0 > 10.0.0.165:4000/csi-node-driver-registrar:v2.7.0 > I had the same problem. Then I tried to use an older version of kubernetes : 1.21.11 with the older images shown above (following this link https://www.roksblog.de/deploy-kubernetes-clusters-in-openstack-within-minu…) and it worked, the cinder volume was successfully mounted inside my nginx pod. - What is the meaning of the error I am having? - Is it magnum related or kubernetes related or both? Regards.

4 6

[Glance][PTG] Glance 2023.2 (Bobcat) vPTG Schedule
by Pranali Deore 20 Mar '23

20 Mar '23

Hello All, The 2023.2 (Bobcat) virtual PTG is going to start next week and we have created our PTG etherpad [1] and also added day wise topics along with timings we are going to discuss. Kindly let me know if you have any concerns with allotted time slots. Friday is reserved for any unplanned discussions. So please feel free to add your topics if you haven't added yet. As a reminder, these are the time slots for our discussion. Tuesday 28 MARCH 2023 1400 UTC to 1700 UTC Wednesday 29 MARCH 2023 1400 UTC to 1700 UTC Thursday 30 MARCH 2023 1400 UTC to 1700 UTC Friday 31 MARCH 2023 1400 UTC to 1700 UTC NOTE: We have scheduled glance operator hours on Thursday at 16:20 UTC(we can extend it if required), let us know your availability for the same. At the moment we don't have any sessions scheduled on Friday, if there are any last moment request(s)/topic(s) we will discuss that on Friday else we will conclude our PTG on Thursday 30th March. We will be using bluejeans for our discussion, kindly try to use it once before the actual discussion. The meeting URL is mentioned in etherpad [1] and will be the same throughout the PTG. [1]: https://etherpad.opendev.org/p/glance-bobcat-ptg Hope to see you there!! Thanks & Regards, Pranali Deore

1 0

[neutron] Extra routes
by Arnaud Morin 20 Mar '23

20 Mar '23

Hey all, When using DVR, is there any way to set extra-routes only on the snat network nodes? I want routes to apply only on north/south communication, not on east/west. I can't find something like this is API. Cheers,

1 0

[openstack][masakari] Ask about Masakari segment
by Nguyễn Hữu Khôi 19 Mar '23

19 Mar '23

Hello guys. I want to ask if I create two Masakari segments then instances will failover on only segment has a group of computes? Because I do test with this scenario, my instance failover on a different segment which has different compute hosts? Do I understand Masakari wrong? Thank you. Regards Nguyen Huu Khoi

2 3

Autoscaling in Kolla Ansible wallaby series
by Uday Dikshit 18 Mar '23

18 Mar '23

Hello Team I am looking forward to have an autoscaling feature on Kolla Ansible wallaby series openstack. I am using Senlin to create cluster, gnocchi for metrics and aodh for alarm. However I am facing issue with aodh alarm state as it gets stuck in the state in which it is. I also found once the load is hiking, the gnocchi metrics get inconsistent. Due to this also the alarm state sticks and do not trigger alarm. To solve this, i relied upon collectd. But collectd service does not push any metric from the Hypervisor to gnocchi. My objective is that, in case of a metric reaching the threshold, alarm should automatically trigger and cluster should scale up or down based on the load. Once the scale up or down task is successfully completed, alarm should return to normal state automatically. Does anybody have any experience with this use case or wanna propose any other solution? Thanks

1 0

[zun] allow zun to get information from blazar database
by James Leong 18 Mar '23

18 Mar '23

Hi all, I am using kolla-ansible for OpenStack deployment in the yoga version. Is It possible to allow zun to retrieve information from the blazar database in zun_api container? I have tried to include the blazar database connection information in the zun.conf file. However, when I try to use the newly added blazar information, I am getting the following error message. oslo_config.cfg.NoSuchOptError: no such option connectionTest in group [database] It seems like I have to set up some option somewhere else in the code. But I was not able to identify them. Thanks for your help James

1 0

[release] Release countdown for week R-0, March 20-24
by Herve Beraud 17 Mar '23

17 Mar '23

Development Focus ----------------- We will be releasing the coordinated OpenStack Antelope 2023.1 release next week, on March 22. Thanks to everyone involved in the Antelope 2023.1 cycle! We are now in pre-release freeze, so no new deliverable will be created until final release, unless a release-critical regression is spotted. Otherwise, teams attending the virtual PTG should start to plan what they will be discussing there, by creating and filling team etherpads. You can access the list of PTG etherpads at: http://ptg.openstack.org/etherpads.html General Information ------------------- On release day, the release team will produce final versions of deliverables following the cycle-with-rc release model, by re-tagging the commit used for the last RC. A patch doing just that will be proposed. PTLs and release liaisons should watch for that final release patch from the release team. While not required, we would appreciate having an ack from each team before we approve it on the 22nd, so that their approval is included in the metadata that goes onto the signed tag. Upcoming Deadlines & Dates -------------------------- Final Antelope 2023.1 release: March 22 Virtual PTG: March 27-31 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[horizon] Bobcat PTG Schedule
by vishal manchanda 17 Mar '23

17 Mar '23

Hello Team, Please Find the Schedule for Horizon Bobcat PTG in the eherpad [1]. Feel Free to add the topics you want to discuss in the PTG. Don't forget to register for PTG, if not done yet [2]. See you at the PTG! Thanks & Regards, Vishal Manchanda (irc: vishalmanchanda) [1] https://etherpad.opendev.org/p/horizon-bobcat-ptg [2] https://openinfra-ptg.eventbrite.com/

1 0

[kolla] Bobcat vPTG slots and topics
by Michał Nasiadka 17 Mar '23

17 Mar '23

Hello, Koalas! Allocated slots for Kolla sessions: 27-30 March 2023: Monday - 13.00 - 17.00 UTC (general, Kolla and Kolla-Ansible) Tuesday - 13.00 - 15.00 UTC (Kolla-Ansible) Tuesday - 15.00 - 17.00 UTC (Operator Hours Kolla) Thursday - 13.00 - 15.00 UTC (Kayobe) Please look at Kolla planning etherpad [1] and fill out topic proposals. Looking forward to meeting you! [1] https://etherpad.opendev.org/p/manila-bobcat-ptg-planning [2] https://ptg.opendev.org/ptg.html Thanks, mnasiadka

1 0

[kolla-ansible] migrate from OVS to OVN
by Nguyễn Hữu Khôi 17 Mar '23

17 Mar '23

Hello guys. Can we use kolla ansible to migrate from OVS to OVN? If then will it have downtime or impacts? Thank you much, Nguyen Huu Khoi

2 4

[magnum][kolla ansible]Ask about use higher magnum image on previous opentack version
by Nguyễn Hữu Khôi 17 Mar '23

17 Mar '23

Hello guys. I use Openstack Xena by using Kolla Ansible tool. due to some reason, I want to use Zed Magnum on my current system(Xena). Can I do this task by rebuilding Magnum images from code then retagging and replacing the Magnum container with new images? Any experience for this? Thank you very much. Nguyen Huu Khoi

1 0

Fwd: [kolla-ansible] Whether the cluster of two control nodes is available
by kaqiu pi 16 Mar '23

16 Mar '23

Hi， I'm a newer in kolla-ansibe. And I could deploy a cluster of two controll nodes by kolla-ansible. But I don't konw whether the cluster is anvailable? I would like to ask, when the number of control nodes is 2, the status of mariadb and rabbitmq clusters, are they safe and available for production? Thanks for any guidance. Good Luck

3 2

[outreachy][cinder] Running test in dev Enviornment
by Toheeb Oyekola 16 Mar '23

16 Mar '23

Hi everyone, I am having some error when i run "tox -e py3", here is a linke to the error https://paste.openstack.org/show/b5SAyZ35FdDuK2sRwTxD/.

2 2

Neutron
by Shrey Chauhan 16 Mar '23

16 Mar '23

2 1

[ironic] [infra] Cleaning up old IPA images from tarballs.o.o
by Dmitry Tantsur 16 Mar '23

16 Mar '23

Hi all, I would like to do a clean-up of old or wrongly created IPA images on the tarballs site. Before I do that, could you please check the proposed list to make sure we don't remove something we expect to be used? The list is https://paste.opendev.org/show/btDps0HFoYG9TKMmv1LB/ Dmitry

3 3

[neutron] PTG poll for meeting slots
by Rodolfo Alonso Hernandez 16 Mar '23

16 Mar '23

Hello Neutrinos: This is the link [1] (that I should have sent yesterday) to vote for the meeting slots during the PTG week. I think that 3 hours per day, from Tuesday to Friday, will be enough to cover the topics we need to discuss. Please feel free to send your vote here. Next week I'll close the poll and schedule the meetings. Thank you! [1]https://doodle.com/meeting/participate/id/eZ83mmvb

1 0

(OpenStack-horizon) unable to open horizon page after installing Open Stack
by Adivya Singh 16 Mar '23

16 Mar '23

Hi Team, I am unable to open Open OpenStack horizon page, after installation When i am opening the link , it says Haproxy service seems up and running, I have tried to Flush IP tables also, Seeing this might be causing the issue Port 443 is also listening. Any thoughts on this [image: image.png]

4 4

[Openstack] Lack of Balance solution such as Watcher.
by Nguyễn Hữu Khôi 16 Mar '23

16 Mar '23

I am building a private cloud.Everything is ok. But I cannot find a way to balance vm when It have heavy load which cause impact other vm on the compute node. I found solutions such as Watcher and Leveller. But they need to be done manually. Watcher is not good because It need cpu metric such as cpu load in Ceilometer which is removed so we cannot use it. Leveller is good, but It is obsolete. Nguyen Huu Khoi

4 7

[cinder] festival of feature reviews 17th March 2023
by Rajat Dhasmana 15 Mar '23

15 Mar '23

Hello Argonauts, We will be having our monthly festival of reviews tomorrow i.e. 17th March (Friday) from 1400-1600 UTC. We are close to the PTG but we still have backlog so good to have a head start in reviews for the next (2023.2) cycle. Following are some additional details: Date: 17th March, 2023 Time: 1400-1600 UTC Meeting link: https://bluejeans.com/556681290 etherpad: https://etherpad.opendev.org/p/cinder-festival-of-reviews See you there! Thanks Rajat Dhasmana

1 0

[cinder] Canceling upstream meeting 22nd March
by Rajat Dhasmana 15 Mar '23

15 Mar '23

Hello Argonauts, As discussed in this week's meeting[1], we will be canceling the Cinder upstream meeting next week i.e. 22nd March, 2023. Since we have RC2 this week, 2023.1 release next week and PTG after that, we don't expect many topics next week, but if you still have any, please add them to the PTG planning etherpad[2]. See you all at the PTG! [1] https://meetings.opendev.org/irclogs/%23openstack-meeting-alt/%23openstack-… [2] https://etherpad.opendev.org/p/bobcat-ptg-cinder-planning

1 0

Re: [manila] create snapshot from share not permitted
by Alvaro Soto 15 Mar '23

15 Mar '23

So you can test and get fresh log outputs, please do the following manila create <PARAMS> manila list manila show <PARAMS> Then go to https://pastebin.com/ and paste the CLI output and errors you see on the manila log files Friendly reminder: please click on 'reply all' so this thread can help more people Cheers! On Tue, Mar 14, 2023 at 1:53 AM garcetto <garcetto(a)gmail.com> wrote: > seems enabled...dont know why is not working... > > $ manila pool-list --detail > > +--------------------------------------+----------------------------------------------+ > | Property | Value > | > > +--------------------------------------+----------------------------------------------+ > | name | ostack-test@generic#GENERIC > | > | share_backend_name | GENERIC > | > | driver_handles_share_servers | True > | > | vendor_name | Open Source > | > | driver_version | 1.0 > | > | storage_protocol | NFS_CIFS > | > | total_capacity_gb | unknown > | > | free_capacity_gb | unknown > | > | reserved_percentage | 0 > | > | reserved_snapshot_percentage | 0 > | > | reserved_share_extend_percentage | 0 > | > | qos | False > | > | pools | None > | > | snapshot_support | True > | > | create_share_from_snapshot_support | True > | > | revert_to_snapshot_support | False > | > | mount_snapshot_support | False > | > | replication_domain | None > | > | filter_function | None > | > | goodness_function | None > | > | security_service_update_support | False > | > | network_allocation_update_support | False > | > | share_server_multiple_subnet_support | False > | > | max_shares_per_share_server | -1 > | > | max_share_server_size | -1 > | > | share_group_stats | {'consistent_snapshot_support': > None} | > | ipv4_support | True > | > | ipv6_support | False > | > | server_pools_mapping | > {'2242964f-be38-4f11-8c90-cdcfcd20c20a': []} | > | timestamp | 2023-03-13T09:13:06.331713 > | > > +--------------------------------------+----------------------------------------------+ > > > On Mon, Mar 13, 2023 at 11:45 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: > >> If you are inside this features support matrix >> >> >> https://docs.openstack.org/manila/latest/admin/share_back_ends_feature_supp… >> >> Examine your configuration as well: >> >> >> - >> >> snapshot_support indicates whether snapshots are supported for shares >> created on the pool/backend. When administrators do not set this capability >> as an extra-spec in a share type, the scheduler can place new shares of >> that type in pools without regard for whether snapshots are supported, and >> those shares will not support snapshots. >> >> >> https://docs.openstack.org/manila/latest/admin/capabilities_and_extra_specs… >> >> Cheers! >> >> On Mon, Mar 13, 2023 at 3:35 AM garcetto <garcetto(a)gmail.com> wrote: >> >>> good morning, >>> i am using manila and generic driver with dhss true, but cannot create >>> snapshot from shares, any help? where can i look at? >>> (cinder backend is a linux nfs server) >>> >>> thank you >>> >>> $ manila snapshot-create share-01 --name Snapshot1 >>> ERROR: Snapshots cannot be created for share >>> '2c8b1b3d-ef82-4372-94df-678539f0d843' since it does not have that >>> capability. (HTTP 422) (Request-ID: >>> req-cab23a46-37dc-4f2b-b26c-d6b21b7453ba) >>> >>> >> >> -- >> >> Alvaro Soto >> >> *Note: My work hours may not be your work hours. Please do not feel the >> need to respond during a time that is not convenient for you.* >> ---------------------------------------------------------- >> Great people talk about ideas, >> ordinary people talk about things, >> small people talk... about other people. >> > -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

2 1

Magnum in yoga release on Ubuntu 22.04
by МАКАРОВ МАКС 15 Mar '23

15 Mar '23

4 5

live resize
by Parsa Aminian 15 Mar '23

15 Mar '23

hello on openstack with ceph backend is it possible to live resize instances ? I want to change flavor without any down time .

5 6

[cinder] Bug Report | 15-03-2023
by Sofia Enriquez 15 Mar '23

15 Mar '23

Hello Argonauts, Medium - Extending SCSI multipath doesn't work if Nova configuration changed. <https://bugs.launchpad.net/os-brick/+bug/2009157> - *Status*: Unassigned. - An Extended multipathed device should not require a reconfigure. <https://bugs.launchpad.net/os-brick/+bug/2009158> - *Status*: Unassigned. - [rbac] Reader user able to delete a user message. <https://bugs.launchpad.net/cinder/+bug/2009818> - *Status*: Unassigned. - Cinder Message API creates failure "'NoneType' object is not subscriptable". <https://bugs.launchpad.net/cinder/+bug/2009483> - *Status*: Marked as duplicate of similar bug <https://bugs.launchpad.net/cinder/+bug/1978729>. Fix proposed to master <https://review.opendev.org/c/openstack/cinder/+/876587>. Low - [HPE] Volume name migration fails with keyerror <https://bugs.launchpad.net/cinder/+bug/2008931>. - *Status*: No fix proposed to master yet. Incomplete - Cannot delete snapshot: invalid backing file. <https://bugs.launchpad.net/cinder/+bug/2009849> Cinder removed: - [nova] size_iops_sec does behave differently than mentioned in docs. <https://bugs.launchpad.net/nova/+bug/2008762> Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[nova][PTG] PTG etherpad, please ideally add your topics before next week
by Sylvain Bauza 15 Mar '23

15 Mar '23

As we discussed on the last nova meeting [1], I'll create an agenda for all the topics we will have for the next vPTG by next Tuesday. For this, it would be nice if most of the topics people would like to discuss are already on the PTG etherpad before Tuesday, so please look at this etherpad and add your own topics rather sooner than later ;-) https://etherpad.opendev.org/p/nova-bobcat-ptg I also add a courtesy ping list item for each of the existing topics. Please add your IRC nick if you can't be around for all of the PTG time, so basically at the beginning of every PTG topic, I'd ping all the folks for it. As a reminder, I booked those time slots for the Diablo room : - *Tuesday* *13:00 UTC - 17:00 UTC* - *Wednesday 13:00 UTC - 17:00 UTC* - *Thursday 13:00 UTC - 17:00 UTC* - *Friday 13:00 UTC - 17:00 UTC* Thanks, -S [1] https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.log.h…

1 0

[publiccloud-sig] Reminder - next meeting March 15th - 0800 UTC
by Felix Kronlage-Dammers 14 Mar '23

14 Mar '23

Hi everyone, better late than not at all ;) Here comes the reminder for the next meeting of the Public Cloud SIG: This is on March 15th (this wednesday) at 0800 UTC. We meet on IRC in #openstack-operators. A preliminary agenda can be found in the pad: https://etherpad.opendev.org/p/publiccloud-sig-meeting See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG read you on wednesday! felix

1 0

Wallaby on Ubuntu 20.04, Neutron 18.6.0 neutron-dhcp-agent RPC unusually slow
by Zakhar Kirpichenko 14 Mar '23

14 Mar '23

Hi! We're running Openstack Wallaby on Ubuntu 20.04, 3 high-performance infra nodes with a RabbitMQ cluster. I updated Neutron components to version 18.6.0, which recently became available in the cloud repository ( http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby main). The exact package versions updated are as follows: Install: libunbound8:amd64 (1.9.4-2ubuntu1.4, automatic), openvswitch-common:amd64 (2.15.2-0ubuntu1~cloud0, automatic) Upgrade: neutron-common:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), python3-werkzeug:amd64 (0.16.1+dfsg1-2, 0.16.1+dfsg1-2ubuntu0.1), neutron-dhcp-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-l3-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), python3-neutron:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-server:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-plugin-ml2:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-metadata-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-linuxbridge-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1) Installed Neutron packages: ii neutron-common 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - common ii neutron-dhcp-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - DHCP agent Firewall-as-a-Service driver for OpenStack Neutron ii neutron-l3-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - l3 agent ii neutron-linuxbridge-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - linuxbridge agent ii neutron-metadata-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - metadata agent ii neutron-plugin-ml2 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - ML2 plugin ii neutron-server 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - server ii python3-neutron 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - Python library ii python3-neutron-lib 2.10.1-0ubuntu1~cloud0 all Neutron shared routines and utilities - Python 3.x ii python3-neutronclient 1:7.2.1-0ubuntu1~cloud0 all client API library for Neutron - Python 3.x Normally this would be an easy update, but this time neutron-dhcp-agent doesn't work properly: 2023-03-14 05:44:27.572 2534501 INFO neutron.agent.dhcp.agent [req-4a362701-cc1f-4b9d-87e6-045b6a388709 - - - - -] Synchronizing state complete 2023-03-14 05:44:38.868 2534501 ERROR neutron_lib.rpc [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Timeout in RPC method dhcp_ready_on_ports. Waiting for 55 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:44:38.871 2534501 WARNING neutron_lib.rpc [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Increasing timeout for dhcp_ready_on_ports calls to 120 seconds. Restart the agent to restore it to the default value.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:45:34.244 2534501 ERROR neutron.agent.dhcp.agent [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Timeout notifying server of ports ready. Retrying...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:47:10.876 2534501 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:47:34.353 2534501 ERROR neutron_lib.rpc [req-607a9252-49b1-4043-aa0d-2457b78dc99e - - - - -] Timeout in RPC method dhcp_ready_on_ports. Waiting for 27 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID f254f735998243c4b0a58ce95c974534 2023-03-14 05:47:34.354 2534501 WARNING neutron_lib.rpc [req-607a9252-49b1-4043-aa0d-2457b78dc99e - - - - -] Increasing timeout for dhcp_ready_on_ports calls to 240 seconds. Restart the agent to restore it to the default value.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID f254f735998243c4b0a58ce95c974534 2023-03-14 05:47:46.681 2534501 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : f254f735998243c4b0a58ce95c974534 2023-03-14 05:48:01.086 2534501 ERROR neutron.agent.dhcp.agent [req-607a9252-49b1-4043-aa0d-2457b78dc99e - - - - -] Timeout notifying server of ports ready. Retrying...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID f254f735998243c4b0a58ce95c974534 2023-03-14 05:49:45.035 2534501 INFO neutron.agent.dhcp.agent [req-5935a0d0-a981-463c-a4ea-23ccbb54c896 - - - - -] DHCP configuration for ports ... (A successful configuration here). While neutron-dhcp-agent is waiting, neutron-server log gets filled up with: neutron-server.log:2023-03-14 05:47:05.761 4171971 INFO neutron.plugins.ml2.plugin [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Attempt 1 to provision port 18cddbb8-f3ed-4b49-9c6f-c0c67b4f7c76 ... neutron-server.log:2023-03-14 05:47:10.727 4171971 INFO neutron.plugins.ml2.plugin [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Attempt 10 to provision port 18cddbb8-f3ed-4b49-9c6f-c0c67b4f7c76 This repeats for each port of each network neutron-dhcp-agent needs to configure. Each subsequent configuration for each network takes about 1-2 minutes, depending on the network size. With earlier Neutron versions the whole process of configuring all networks would finish in under a minute, i.e. DHCP configuration per port (and network) is several orders of magnitude slower than it should be. Once neutron-dhcp-agent finishes synchronization, it seems to work without issues although there aren't that many changes in our cloud to tell whether it's fast or slow, individual port updates seem to happen quickly. All other services are working well, RabbitMQ cluster is working well, infra nodes are not overloaded and there are no apparent issues other than this one with Neutron, thus I am inclined to think that the issue is specific to version 18.6.0 of neutron-dhcp-agent or neutron-server. I would appreciate any advice! Best regards, Zakhar

1 1

[openstack][backup] Experience for instance backup
by Nguyễn Hữu Khôi 14 Mar '23

14 Mar '23

Hello guys. I am looking for instance backup solution. I am using Cinder backup with nfs backup but it looks not too fast. I am using a 10Gbps network. I would like to know experience for best practice for instance backup solutions on Openstack. Thank you. Nguyen Huu Khoi

4 4

[neutron] Bug deputy - report from week of 6th March
by Slawek Kaplonski 14 Mar '23

14 Mar '23

Hi, Sorry for sending it so late but I was off Monday and Tuesday and I forgot about it. Here's list of new bugs reported last week: ## High * https://bugs.launchpad.net/neutron/+bug/2009678 - [OVN] The OVN agent config entry point is incorrect in setup.cfg - assigned to Rodolfo, fix proposed * https://bugs.launchpad.net/neutron/+bug/2009632 - ARP requests from ovnmeta namespaces are sent to physical interfaces of computing nodes - **not assigned yet** * https://bugs.launchpad.net/neutron/+bug/2009703 - [OVN] HW offload event "QoSMinimumBandwidthEvent" fails if the min-bw rule is removed - assigned to Rodolfo, fix proposed ## Medium * https://bugs.launchpad.net/neutron/+bug/2009509 - Large number of FIPs and subnets causes slow sync_routers response - assigned to Adam Oswick * https://bugs.launchpad.net/neutron/+bug/2009804 - [OVN] Method ``get_port_qos`` should always return 2 values - **not assigned yet, low hanging fruit bug** ## Low * https://bugs.launchpad.net/neutron/+bug/2009728 - [OVS] "permitted_ethertypes" should be validated and filtered during the OVS agent initialization - **not assigned yet, low hanging fruit bug** * https://bugs.launchpad.net/neutron/+bug/2009832 - FWaaS docs lack required packages - **not assigned yet, low hanging fruit bug** * https://bugs.launchpad.net/neutron/+bug/2009831 - VPNaaS docs lack packages to install - **not assigned yet, low hanging fruit bug** ## Incomlete * https://bugs.launchpad.net/neutron/+bug/2009807 - Not able to create external physical network - needs some more info but for now it looks more like user's issue rather than bug ## Others * https://bugs.launchpad.net/neutron/+bug/2009705 - [FWaaS ]Openstack Zed - firewall group status doesn't change to ACTIVE. - **unassigned**, it needs someone from FWaaS to look at it and ZhaoHeng is looking into it already. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[tc][all] What's happening in the Technical Committee - 2023 March 14
by Nikolla, Kristi 14 Mar '23

14 Mar '23

Hi all, Please find below a summary for the last 2 weeks of the TC. Meetings ======= - March 1, 2023 A video recoding of the meeting is available https://www.youtube.com/watch?v=HA1owc9qGiE - March 8, 2023 Meeting notes are available https://meetings.opendev.org/meetings/tc/2023/tc.2023-03-08-15.59.html - The next Technical Committee meeting will be tomorrow, March 14, 2023 at 16:00 UTC, chaired by Jay Faulkner. Agenda is available at the link below. Please contact me or Jay if you notice missing items that should be discussed. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee Merged Changes ============== - Appoint Felipe Reyes as OpenStack_Charms PTL (formal-vote) https://review.opendev.org/c/openstack/governance/+/874971 - Appoint Liu as Senlin PTL (formal-vote) https://review.opendev.org/c/openstack/governance/+/874969 - Adding mailto link in upstream opportunities doc (documentation-change) https://review.opendev.org/c/openstack/governance/+/874968 - Move heat-translator and tosca-parser to tacker's governance (project-update) https://review.opendev.org/c/openstack/governance/+/876012 - Fix doc referencing 'admin_api' rule (typo-fix) https://review.opendev.org/c/openstack/governance/+/875860 - Correct CHAIR.rst: reflect vice-chair nom practice https://review.opendev.org/c/openstack/governance/+/875788 - Ironic program adopting x/virtualpdu https://review.opendev.org/c/openstack/governance/+/876208 - Volunteer to serve at 2023.2 TC vice chair https://review.opendev.org/c/openstack/governance/+/875787 - Add chair role to knikolla for 2023.2 https://review.opendev.org/c/openstack/governance/+/875742 Open Changes ============ There are 15 open changes to the Governance repo. https://review.opendev.org/q/project:openstack/governance+status:open Happenings ========== New TC Chair and Vice Chair ---------------------------------------- The TC has a new Chair (Kristi Nikolla) and Vice Chair (Jay Faulkner). Thank you Ghanshyam Mann for your 2 years of service as the TC chair. https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032506.h… TripleO is being deprecated -------------------------------------- The TC voted to continue supporting the Zed release of TripleO, however without a project team. https://review.opendev.org/c/openstack/governance/+/877132 Leaderless projects -------------------------- We're now down to 7 projects without any PTL candidacies (Monasca, Rally, Sahara, Swift, TripleO, Vitrage, and Winstackers). The TC has started reaching out to the teams and former PTLs. If you have any information or interest, please reach out to the TC via the etherpad linked below. https://etherpad.opendev.org/p/2023.2-leaderless Release naming for projects -------------------------------------- The TC continued discussing changes related to version naming for projects. Specifically, there are two proposals about allowing projects to omit the OpenStack version when referring to their project, or flipping the OpenStack and project version order. https://review.opendev.org/c/openstack/governance/+/874484 https://review.opendev.org/c/openstack/governance/+/875942 Virtual PTG ---------------- The TC is considering scheduling the same PTG time slots for itself as the previous PTG. Namely, Thursday and Friday 15:00 - 19:00 UTC. I will be booking the slots later today. https://etherpad.opendev.org/p/tc-2023-2-ptg How to contact the TC: ================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send an email with the tag [tc] on the openstack-discuss mailing list. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 16:00 UTC 3. IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC.

1 0

[ptls][Antelope] OpenInfra Live: OpenStack Antelope
by Kristin Barrientos 14 Mar '23

14 Mar '23

Hi everyone, As we get closer to the OpenStack release, I wanted to reach out to see if any PTL’s were interested in providing their Antelope cycle highlights in an OpenInfra Live[1] episode on Thursday, March 23 at 1500 UTC. Ideally, we would get 4-6 projects represented. Previous examples of OpenStack release episodes can be found here[2] <http://www.youtube.com/watch?v=hwPfjvshxOM>and here[3] <https://www.youtube.com/watch?v=MSbB3L9_MeY>. Please let me know if you’re interested and I can provide next steps. If you would like to provide a project update but that time doesn’t work for you, please share a recording with me and I can get it added to the project navigator. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation [1] https://openinfra.dev/live/ [2] https://www.youtube.com/watch?v=hwPfjvshxOM [3] https://www.youtube.com/watch?v=MSbB3L9_MeY

5 4

Re: [manila] support for encryption
by Alvaro Soto 13 Mar '23

13 Mar '23

I don't think so, and it also doesn't sound like a feature that Manila should implement; try to do that in your backend storage. Cheers! On Sat, Mar 11, 2023 at 7:10 AM garcetto <garcetto(a)gmail.com> wrote: > good afternoon, > data encryption. > thank you > > Il Ven 10 Mar 2023, 21:48 Alvaro Soto <alsotoes(a)gmail.com> ha scritto: > >> You mean data or token encryption? >> >> Cheers! >> >> On Fri, Mar 10, 2023 at 9:08 AM garcetto <garcetto(a)gmail.com> wrote: >> >>> good afternoon, >>> does manila support encryption in some sort ? >>> >>> thank you >>> >> >> >> -- >> >> Alvaro Soto >> >> *Note: My work hours may not be your work hours. Please do not feel the >> need to respond during a time that is not convenient for you.* >> ---------------------------------------------------------- >> Great people talk about ideas, >> ordinary people talk about things, >> small people talk... about other people. >> > -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 0

[manila] create snapshot from share not permitted
by garcetto 13 Mar '23

13 Mar '23

good morning, i am using manila and generic driver with dhss true, but cannot create snapshot from shares, any help? where can i look at? (cinder backend is a linux nfs server) thank you $ manila snapshot-create share-01 --name Snapshot1 ERROR: Snapshots cannot be created for share '2c8b1b3d-ef82-4372-94df-678539f0d843' since it does not have that capability. (HTTP 422) (Request-ID: req-cab23a46-37dc-4f2b-b26c-d6b21b7453ba)

2 1

[Diversity] Diversity and Inclusion WG Meeting reminder
by Amy Marrich 13 Mar '23

13 Mar '23

This is a reminder that the Diversity and Inclusion WG will be meeting tomorrow at 14:00 UTC in the #openinfra-diversity channel on OFTC. We hope members of all OpenInfra projects join us as we look at the Code of Conduct, and continue working on planning for the OpenInfra Summit as well as Foundation-wide diversity surveys. Thanks, Amy (spotz) 0 - https://etherpad.opendev.org/p/diversity-wg-agenda

1 0

*Migration)Red Hat Open Stack Migration Lift and Shift
by Adivya Singh 13 Mar '23

13 Mar '23

Hi Team, I am making a plan for Red hat OpenStack migration(Lift and Shift), The Director is in VM sourced in VMware, which we are migrated in other DC, But the IP and VLAN will change Will it be advisable to change the Provisioning IP and other IP , according to the New VLAN designed in the new DC, Configured in the template , Do a introspection of the new design and Configured the Red hat OpenStack, Any addition on this or guidance would be highly usefull Regards Adivya Singh

2 1

[cinder] proposing Jon Bernard for cinder core
by Rajat Dhasmana 13 Mar '23

13 Mar '23

Hello everyone, I would like to propose Jon Bernard as cinder core. Looking at the review stats for the past 60[1], 90[2], 120[3] days, he has been consistently in the top 5 reviewers with a good +/- ratio and leaving helpful comments indicating good quality of reviews. He has been managing the stable branch releases for the past 2 cycles (Zed and 2023.1) and has helped in releasing security issues as well. Jon has been part of the cinder and OpenStack community for a long time and has shown very active interest in upstream activities, be it release liaison, review contribution, attending cinder meetings and also involving in outreachy activities. He will be a very good addition to our team helping out with the review bandwidth and adding valuable input in our discussions. I will leave this thread open for a week and if there are no objections, I will add Jon Bernard to the cinder core team. [1] https://www.stackalytics.io/report/contribution?module=cinder-group&project… [2] https://www.stackalytics.io/report/contribution?module=cinder-group&project… [3] https://www.stackalytics.io/report/contribution?module=cinder-group&project… Thanks Rajat Dhasmana

5 5

[ovn] safely change bridge_mappings
by Tomáš Bredár 13 Mar '23

13 Mar '23

Hi, I have a running production OpenStack deployment - version Wallaby installed using TripleO. I'm using the default OVN/OVS networking. For provider networks I have two bridges on the compute nodes br-ex and br-ex2. Instances mainly use br-ex for provider networks, but there are some instances which started using a provider network which should be mapped to br-ex2, however I didn't specify "bridge_mappings" on ml2_conf.ini, so the traffic wants to flow through the default datacentre:br-ex. My questions is, what services should I restart on the controller and compute nodes after defining bridge_mappings in [ovs] in ml2_conf.ini. And if this operation is safe and if the instances already using br-ex will lose connectivity? Thanks for your help Tomas

2 3

[outreachy][cinder] Cannot access code repository
by Benjamin Faruna 13 Mar '23

13 Mar '23

Hello, good day, I was selected for the outreachy internship program and I want to contribute to cinders' codebase, but I am having trouble accessing it. Whenever I open the code tab on launchpad I get 2 repositories that haven't been updated in a while but the main page shows a lot of activities on the codebase. Please I want to request help setting up and getting started making contributions. The images of what I see is attached to this email.

3 2

[neutron] Openstack Network Interconnection
by Roberto Bartzen Acosta 13 Mar '23

13 Mar '23

Hey folks. Does anyone have ideas on how to interconnect different Openstack deployments? Consider that we have multiple Datacenters and need to interconnect tenant networks. How could this be done in the context of OpenStack (without using VPN) ? We have some ideas about the usage of OVN-IC (OVN Interconnect). It looks like a great solution to create a network layer between DCs/AZs with the help of the OVN driver. However, Neutron does not support the Transit Switches (OVN-IC design) that are required for this application. We've seen references to abandoned projects like [1] [2] [3]. Does anyone use something similar in production or have an idea about how to do it? Imagine that we need to put workloads on two different AZs that run different Openstack installations, and we want to communicate with the local networks without using a FIP. I believe that the most coherent way to maintain databases consistent in each Openstack would be an integration with Neutron, but I haven't seen any movement on that. Regards, Roberto [1] https://www.youtube.com/watch?v=GizLmSiH1Q0 [2] https://specs.openstack.org/openstack/neutron-specs/specs/stein/neutron-int… [3] https://opendev.org/x/neutron-interconnection -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*

2 3

[keystone] Best-practice for (admin) endpoint config
by Christian Rohmann 13 Mar '23

13 Mar '23

Hello Openstack-discuss, I am wondering what the current recommendations / best-practices are in regards to the endpoints of keystone (and other services). There are three types of endpoints: public, internal and admin: * "public" certainly is for API access done by cloud users - so measures like rate limiting are likely in place. * "internal" is an alternative URL to be used by other services. Sounds reasonable to have an alternate path for the internal communication of OpenStack services, like having a management network Also reading https://docs.openstack.org/security-guide/api-endpoints/api-endpoint-config… this seems to be the recommendation. * "admin" - This is the one that gives me a little headache. According to commits 1) https://opendev.org/openstack/keystone/commit/4ec69218454d9f8be7150e2cee50c… 2) https://github.com/openstack/keystone/commit/ecf721a3c176daf67d00536c48e80e… there should actually be no admin endpoint for Keystone anymore. Or should there? But looking at openstack-ansible doing the endpoint config (https://opendev.org/openstack/openstack-ansible-os_keystone/src/commit/a020…) all tree types are still configured? Backwards compatibility for services still expecting this endpoint? 1) Ceilometer - https://bugs.launchpad.net/ceilometer/+bug/1981207 2) Heat - https://review.opendev.org/c/openstack/openstacksdk/+/777343 Apart from Keystone also other services have "admin" endpoints which can be configured and placed as such into the service catalog. What is the reasoning behind that? Thanks and with kind regards, Christian

1 0

[cinder] Openstack-ansible and cinder GPFS backend
by Nicolas Melot 13 Mar '23

13 Mar '23

Hi, I can find doc on using various backends for cinder (https://docs.openstack.org/openstack-ansible-os_cinder/zed/configure-cinder…) and some documentation to configure a GPFS backend for cinder (https://docs.openstack.org/cinder/zed/configuration/block-storage/drivers/i…) but I cannot find any documentation to deploy cinder with GPFS backend using openstack-ansible. Does this exist at all? Is there any documentation? /Nicolas

3 2

[outreachy][cinder]
by Desire Barine 13 Mar '23

13 Mar '23

Hello Sofia Enriquez, I'm Desire Barine, an Outreachy applicant. I would love to work on Extend automated validation of API reference request/response samples project. I would like to get started with the contribution. I am currently going over the instructions on contributions given. This is my first time contributing on an open source project but I'm really excited to get started. I'm proficient in python, bash and have worked on Rest api creation before. I would love to hear from you. Desire.

2 1

[cinder][PTG] Cinder 2023.2 (Bobcat) PTG Planning
by Rajat Dhasmana 13 Mar '23

13 Mar '23

Hello All, The 2023.2 (Bobcat) virtual PTG is approaching and will be held between 27-31 March, 2023. I've created a planning etherpad[1] and a PTG etherpad[2] to gather topics for the PTG. Note that you only need to add topics in the planning etherpad and those will be arranged in the PTG etherpad later. Dates: Tuesday (28th March) to Friday (31st March) 2023 Time: 1300 to 1700 UTC Etherpad: https://etherpad.opendev.org/p/bobcat-ptg-cinder-planning Please add the topics as early as possible as finalizing and arranging topics would require some buffer time. [1] https://etherpad.opendev.org/p/bobcat-ptg-cinder-planning [2] https://etherpad.opendev.org/p/bobcat-ptg-cinder Thanks Rajat Dhasmana

1 1

[Outreachy] Setting up development envirenment
by Khadija 12 Mar '23

12 Mar '23

Hi Sofia! I have made myself familiar with the launchpad, storyboard and gerrit. I have also experimented in the Sandbox projects. I was trying to set up my development environment following https://docs.openstack.org/cinder/latest/contributor/development.environmen… but I was unable to run unit tests, on running command 'tox -e py3' I get error saying AttributeError: module 'py' has no attribute 'io' Kindly help me with this so that I can start working on my first issue :) Thank you!

4 6

Allow user access calendar
by vincent lee 12 Mar '23

12 Mar '23

Hi all, I am using kolla-ansible for openstack deployment in yoga version. Is it possible to allow normal user (member role) to view the calendar graph under reservation (lease)? Best Vincent

1 0

Member access calendar
by vincent lee 12 Mar '23

12 Mar '23

Hi all, I am using kolla-ansible for openstack deployment in yoga version. Is it possible to allow normal user (member role) to view the calendar graph under reservation (lease)? Best Vincent

1 0

bare metal provisioner replacement for Fuel
by Frank Smith 11 Mar '23

11 Mar '23

Hey all, quick question: Previously I would use Fuel to deploy Openstack. It was a good tool, as it built a deployment server where I would see new, un-provisioned servers show up in the list, and I could choose the roles, including stacking roles, onto the servers and then provision the whole cluster. There were no complex yaml or JSON files to deal with and it would retry the deployment on an error state. When done, I had a nice OpenStack cluster, a Ceph backend, and life was good. Now it seems Fuel has been retired. What is comparable to the abilities of Fuel today? What are people using for such deployments now? I get that devstack, microstack and packstack are great for all-in-one installs, but Fuel was great for deploying a whole rack of servers from bare metal. I am not asking for any free training here, but instead asking for info on a similar product, if there is one, so I can learn that and be OpenStack productive once more. Thank you for any help, --Francis Smith

3 2

Re: bare metal provisioner replacement for Fuel
by Thomas Goirand 11 Mar '23

11 Mar '23

1 0

[manila] support for encryption
by garcetto 10 Mar '23

10 Mar '23

good afternoon, does manila support encryption in some sort ? thank you

2 1

cryptography min version (non-rust) through 2024.1
by Corey Bryant 10 Mar '23

10 Mar '23

Hi All, As you probably know, recent versions of cryptography have hard dependencies on rust. Are there any community plans to continue supporting a minimum (non-rust) version of cryptography until a specific release? The concern I have downstream in Ubuntu is that we need to continue being compatible with cryptography 3.4.8 through openstack 2024.1. This is because all releases through 2024.1 will be backported to the ubuntu 22.04 cloud archives which will use cryptography 3.4.8. Once we get to 2024.2, we will be backporting to 24.04 cloud archives, which will have the new rust-based versions of cryptography. The current upper-constraint for cryptography is 38.0.2, but the various requirements.txt min versions are much lower (e.g. keystone has cryptography>=2.7). This is likely to lead to patches landing with features that are only in 38.0.2, so it will likely be difficult to enforce min version support. But perhaps a stance toward maintaining compatibility could be established. Thoughts? Corey

4 7

[neutron] bonding sriov nic inside VMs
by Satish Patel 10 Mar '23

10 Mar '23

Folks, As you know, SR-IOV doesn't support bonding so the only solution is to implement LACP bonding inside the VM. I did some tests in the lab to create two physnet and map them with two physical nic and create VF and attach them to VM. So far all good but one problem I am seeing is each neutron port I create has an IP address associated and I can use only one IP on bond but that is just a waste of IP in the Public IP pool. Are there any way to create sriov port but without IP address?

2 6

[security][ironic] Ironic + the VMT
by Jay Faulkner 10 Mar '23

10 Mar '23

Hey all, A recent topic at Ironic meetings the last couple of weeks have been around vulnerability management. Ironic has not been using the OpenStack VMT traditionally; for reasons that AFAICT are lost to time. Is there any reason Ironic should not be vulnerability-managed? Is the security team willing to have us? The only potential complication is that Ironic may receive reports for vendor libraries used by Ironic but not maintained by Ironic -- I was hoping there might already be some historical precedent for how we handle those; it can't be that unique to Ironic. What do folks think? Thanks, Jay Faulkner Ironic PTL TC Member

2 2

[release] Release countdown for week R-1, March 13-17
by Elõd Illés 10 Mar '23

10 Mar '23

Development Focus ----------------- We are on the final mile of the 2023.1 Antelope development cycle! Remember that the 2023.1 Antelope final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. March 17th, 2023 is the deadline for final 2023.1 Antelope release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on March 22nd, 2023. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the 2023.2 Bobcat development cycle, including discussing PTG sessions content, in preparation of the 2023.2 Bobcat Virtual PTG (March 27-31, 2023). Actions ------- Watch for any translation patches coming through on the stable/2023.1 branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/2023.1 branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final 2023.1 Antelope release: March 22nd, 2023 2023.2 Bobcat Virtual PTG: March 27-31, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[openstack-dev][PCI passthrough] How to use PCI passthrough feature correctly? And is this BUG in update_devices_from_hypervisor_resources?
by Simon Jones 10 Mar '23

10 Mar '23

Hi all, I'm working on openstack Yoga's PCI passthrough feature, follow this link: https://docs.openstack.org/nova/latest/admin/pci-passthrough.html I configure exactly as the link said, but when I create server use this command, I found ERROR: ``` openstack server create --flavor cirros-os-dpu-test-1 --image cirros \ --nic net-id=066c8dc2-c98b-4fb8-a541-8b367e8f6e69 \ --security-group default --key-name mykey provider-instance | fault | {'code': 500, 'created': '2023-02-23T06:13:43Z', 'message': 'No valid host was found. There are not enough hosts available.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1548, in schedule_and_build_instances\n host_lists = self._schedule_instances(context, request_specs[0],\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 908, in _schedule_instances\n host_lists = self.query_client.select_destinations(\n File "/usr/lib/python3/dist-packages/nova/scheduler/client/query.py", line 41, in select_destinations\n return self.scheduler_rpcapi.select_destinations(context, spec_obj,\n File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call\n result = self.transport._send(\n File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.\nTraceback (most recent call last):\n\n File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations\n selections = self._select_destinations(\n\n File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations\n selections = self._schedule(\n\n File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule\n self._ensure_sufficient_hosts(\n\n File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts\n raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. There are not enough hosts available.\n\n'} | // this is what I configured: gyw@c1:~$ openstack flavor show cirros-os-dpu-test-1 +----------------------------+------------------------------+ | Field | Value | +----------------------------+------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 1 | | id | 0 | | name | cirros-os-dpu-test-1 | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='a1:1' | | ram | 64 | | rxtx_factor | 1.0 | | swap | | | vcpus | 1 | +----------------------------+------------------------------+ // in controller node /etc/nova/nova.conf [filter_scheduler] enabled_filters = PciPassthroughFilter available_filters = nova.scheduler.filters.all_filters [pci] passthrough_whitelist = {"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"} alias = { "vendor_id":"15b3", "product_id":"101e", "device_type":"type-VF", "name":"a1" } // in compute node /etc/nova/nova.conf [pci] passthrough_whitelist = {"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"} alias = { "vendor_id":"15b3", "product_id":"101e", "device_type":"type-VF", "name":"a1" } ``` The detail ERROR I found is: - The reason why "There are not enough hosts available" is, nova-scheduler's log shows "There are 0 hosts available but 1 instances requested to build", which means no hosts support PCI passthough feature. This is nova-schduler's log ``` 2023-02-28 06:11:58.329 1942637 DEBUG nova.scheduler.manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Starting to schedule for instances: ['8ddfbe2c-f929-4b62-8b73-67902df8fb60'] select_destinations /usr/lib/python3/dist-packages/nova/scheduler/manager.py:141 2023-02-28 06:11:58.330 1942637 DEBUG nova.scheduler.request_filter [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] compute_status_filter request filter added forbidden trait COMPUTE_STATUS_DISABLED compute_status_filter /usr/lib/python3/dist-packages/nova/scheduler/request_filter.py:254 2023-02-28 06:11:58.330 1942637 DEBUG nova.scheduler.request_filter [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Request filter 'compute_status_filter' took 0.0 seconds wrapper /usr/lib/python3/dist-packages/nova/scheduler/request_filter.py:46 2023-02-28 06:11:58.331 1942637 DEBUG nova.scheduler.request_filter [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Request filter 'accelerators_filter' took 0.0 seconds wrapper /usr/lib/python3/dist-packages/nova/scheduler/request_filter.py:46 2023-02-28 06:11:58.332 1942637 DEBUG nova.scheduler.request_filter [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Request filter 'remote_managed_ports_filter' took 0.0 seconds wrapper /usr/lib/python3/dist-packages/nova/scheduler/request_filter.py:46 2023-02-28 06:11:58.485 1942637 DEBUG oslo_concurrency.lockutils [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Lock "567eb2f1-7173-4eee-b9e7-66932ed70fea" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:386 2023-02-28 06:11:58.488 1942637 DEBUG oslo_concurrency.lockutils [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Lock "567eb2f1-7173-4eee-b9e7-66932ed70fea" "released" by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.003s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:400 2023-02-28 06:11:58.494 1942637 DEBUG oslo_db.sqlalchemy.engines [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py:314 2023-02-28 06:11:58.520 1942637 INFO nova.scheduler.host_manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Host mapping not found for host c1c2. Not tracking instance info for this host. 2023-02-28 06:11:58.520 1942637 DEBUG oslo_concurrency.lockutils [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Lock "('c1c2', 'c1c2')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:386 2023-02-28 06:11:58.521 1942637 DEBUG nova.scheduler.host_manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=16.0,cpu_info='{"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["sse4.2", "mds-no", "stibp", "pdpe1gb", "xsaveopt", "ht", "intel-pt", "mtrr", "abm", "tm", "lm", "umip", "mca", "pku", "ds_cpl", "rdrand", "adx", "rdseed", "lahf_lm", "xgetbv1", "nx", "invpcid", "rdtscp", "tsc", "xsavec", "pcid", "arch-capabilities", "pclmuldq", "spec-ctrl", "fsgsbase", "avx2", "md-clear", "vmx", "syscall", "mmx", "ds", "ssse3", "avx", "dtes64", "fxsr", "msr", "acpi", "vpclmulqdq", "smap", "erms", "pge", "cmov", "sha-ni", "fsrm", "x2apic", "xsaves", "cx8", "pse", "pse36", "clflushopt", "vaes", "pni", "ssbd", "movdiri", "movbe", "clwb", "xtpr", "de", "invtsc", "fpu", "tsc-deadline", "pae", "clflush", "ibrs-all", "waitpkg", "sse", "sse2", "bmi1", "3dnowprefetch", "cx16", "popcnt", "rdctl-no", "fma", "tsc_adjust", "xsave", "ss", "skip-l1dfl-vmentry", "sse4.1", "rdpid", "monitor", "vme", "tm2", "pat", "pschange-mc-no", "movdir64b", "gfni", "mce", "smep", "sep", "apic", "arat", "f16c", "bmi2", "aes", "pbe", "est", "pdcm"]}',created_at=2023-02-14T03:19:40Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=415,free_disk_gb=456,free_ram_mb=31378,host='c1c2',host_ip=192.168.28.21,hypervisor_hostname='c1c2',hypervisor_type='QEMU',hypervisor_version=4002001,id=8,local_gb=456,local_gb_used=0,mapped=0,memory_mb=31890,memory_mb_used=512,metrics='[]',numa_topology='{" nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{" nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163962, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["cpuset", "memory_usage", "cpu_usage", "id", "pinned_cpus", "pcpuset", "socket", "network_metadata", "siblings", "mempages", "memory"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.5,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2023-02-28T06:01:33Z,uuid=c360cc82-f0fd-4662-bccd-e1f02b27af51,vcpus=12,vcpus_used=0) _locked_update /usr/lib/python3/dist-packages/nova/scheduler/host_manager.py:167 2023-02-28 06:11:58.524 1942637 DEBUG nova.scheduler.host_manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Update host state with aggregates: [] _locked_update /usr/lib/python3/dist-packages/nova/scheduler/host_manager.py:170 2023-02-28 06:11:58.524 1942637 DEBUG nova.scheduler.host_manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Update host state with service dict: {'id': 17, 'uuid': '6d0921a6-427d-4a82-a7d2-41dfa003125a', 'host': 'c1c2', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 121959, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2023, 2, 28, 6, 11, 49, tzinfo=datetime.timezone.utc), 'forced_down': False, 'version': 61, 'created_at': datetime.datetime(2023, 2, 14, 3, 19, 40, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 2, 28, 6, 11, 49, tzinfo=datetime.timezone.utc), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3/dist-packages/nova/scheduler/host_manager.py:173 2023-02-28 06:11:58.524 1942637 DEBUG nova.scheduler.host_manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Update host state with instances: [] _locked_update /usr/lib/python3/dist-packages/nova/scheduler/host_manager.py:176 2023-02-28 06:11:58.525 1942637 DEBUG oslo_concurrency.lockutils [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Lock "('c1c2', 'c1c2')" "released" by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.004s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:400 2023-02-28 06:11:58.525 1942637 DEBUG nova.filters [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Starting with 1 host(s) get_filtered_objects /usr/lib/python3/dist-packages/nova/filters.py:70 2023-02-28 06:11:58.526 1942637 DEBUG nova.pci.stats [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] ---- before ---- _filter_pools /usr/lib/python3/dist-packages/nova/pci/stats.py:542 2023-02-28 06:11:58.526 1942637 DEBUG nova.pci.stats [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] [] _filter_pools /usr/lib/python3/dist-packages/nova/pci/stats.py:543 2023-02-28 06:11:58.526 1942637 DEBUG nova.pci.stats [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] ---- after ---- _filter_pools /usr/lib/python3/dist-packages/nova/pci/stats.py:545 2023-02-28 06:11:58.527 1942637 DEBUG nova.pci.stats [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] [] _filter_pools /usr/lib/python3/dist-packages/nova/pci/stats.py:546 2023-02-28 06:11:58.527 1942637 DEBUG nova.pci.stats [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Not enough PCI devices left to satisfy request _filter_pools /usr/lib/python3/dist-packages/nova/pci/stats.py:556 2023-02-28 06:11:58.527 1942637 DEBUG nova.scheduler.filters.pci_passthrough_filter [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] (c1c2, c1c2) ram: 31378MB disk: 424960MB io_ops: 0 instances: 0 doesn't have the required PCI devices (InstancePCIRequests(instance_uuid=<?>,requests=[InstancePCIRequest])) host_passes /usr/lib/python3/dist-packages/nova/scheduler/filters/pci_passthrough_filter.py:52 2023-02-28 06:11:58.528 1942637 INFO nova.filters [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Filter PciPassthroughFilter returned 0 hosts 2023-02-28 06:11:58.528 1942637 DEBUG nova.filters [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Filtering removed all hosts for the request with instance ID '8ddfbe2c-f929-4b62-8b73-67902df8fb60'. Filter results: [('PciPassthroughFilter', None)] get_filtered_objects /usr/lib/python3/dist-packages/nova/filters.py:114 2023-02-28 06:11:58.528 1942637 INFO nova.filters [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Filtering removed all hosts for the request with instance ID '8ddfbe2c-f929-4b62-8b73-67902df8fb60'. Filter results: ['PciPassthroughFilter: (start: 1, end: 0)'] 2023-02-28 06:11:58.529 1942637 DEBUG nova.scheduler.manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] Filtered [] _get_sorted_hosts /usr/lib/python3/dist-packages/nova/scheduler/manager.py:610 2023-02-28 06:11:58.529 1942637 DEBUG nova.scheduler.manager [req-13b1baee-e02d-40fc-926d-d497e70ca0dc ff627ad39ed94479b9c5033bc462cf78 512866f9994f4ad8916d8539a7cdeec9 - default default] There are 0 hosts available but 1 instances requested to build. _ensure_sufficient_hosts /usr/lib/python3/dist-packages/nova/scheduler/manager.py:450 ``` Then I search database, I found PCI configure of compute node is not upload: ``` gyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 PCI_DEVICE No inventory of class PCI_DEVICE for c360cc82-f0fd-4662-bccd-e1f02b27af51 (HTTP 404) gyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 PCI_DEVICE No inventory of class PCI_DEVICE for c360cc82-f0fd-4662-bccd-e1f02b27af51 (HTTP 404) gyw@c1:~$ openstack resource class show PCI_DEVICE +-------+------------+ | Field | Value | +-------+------------+ | name | PCI_DEVICE | +-------+------------+ gyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 MEMORY_MB +------------------+-------+ | Field | Value | +------------------+-------+ | allocation_ratio | 1.5 | | min_unit | 1 | | max_unit | 31890 | | reserved | 512 | | step_size | 1 | | total | 31890 | | used | 0 | +------------------+-------+ 这个 31890 能跟上面compute node resource tracker上报的日志对上。 gyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 VCPU 、^Cgyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 VCPU +------------------+-------+ | Field | Value | +------------------+-------+ | allocation_ratio | 16.0 | | min_unit | 1 | | max_unit | 12 | | reserved | 0 | | step_size | 1 | | total | 12 | | used | 0 | +------------------+-------+ gyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 SRIOV_NET_VF No inventory of class SRIOV_NET_VF for c360cc82-f0fd-4662-bccd-e1f02b27af51 (HTTP 404) gyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 DISK_GB +------------------+-------+ | Field | Value | +------------------+-------+ | allocation_ratio | 1.0 | | min_unit | 1 | | max_unit | 456 | | reserved | 0 | | step_size | 1 | | total | 456 | | used | 0 | +------------------+-------+ gyw@c1:~$ openstack resource provider inventory show c360cc82-f0fd-4662-bccd-e1f02b27af51 IPV4_ADDRESS No inventory of class IPV4_ADDRESS for c360cc82-f0fd-4662-bccd-e1f02b27af51 (HTTP 404) MariaDB [nova]> select * from compute_nodes; +---------------------+---------------------+---------------------+----+------------+-------+-----------+----------+------------+----------------+---------------+-----------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+--------------+------------------+-------------+---------------------+---------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-----------------+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------+----------------------+--------------------------------------+-----------------------+--------+ | created_at | updated_at | deleted_at | id | service_id | vcpus | memory_mb | local_gb | vcpus_used | memory_mb_used | local_gb_used | hypervisor_type | hypervisor_version | cpu_info | disk_available_least | free_ram_mb | free_disk_gb | current_workload | running_vms | hypervisor_hostname | deleted | host_ip | supported_instances | pci_stats | metrics | extra_resources | stats | numa_topology | host | ram_allocation_ratio | cpu_allocation_ratio | uuid | disk_allocation_ratio | mapped | +---------------------+---------------------+---------------------+----+------------+-------+-----------+----------+------------+----------------+---------------+-----------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+--------------+------------------+-------------+---------------------+---------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-----------------+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------+----------------------+--------------------------------------+-----------------------+--------+ | 2023-01-04 01:55:44 | 2023-01-04 03:02:28 | 2023-02-13 08:34:08 | 1 | NULL | 4 | 3931 | 60 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 4, "cores": 1, "threads": 1}, "features": ["pat", "cmov", "ibrs-all", "pge", "sse4.2", "sse", "mmx", "ibrs", "avx2", "syscall", "fpu", "mtrr", "xsaves", "mce", "invpcid", "tsc_adjust", "ssbd", "pku", "ibpb", "xsave", "xsaveopt", "pae", "lm", "pdcm", "bmi1", "avx512vnni", "stibp", "x2apic", "avx512dq", "pcid", "nx", "bmi2", "erms", "3dnowprefetch", "de", "avx512bw", "arch-capabilities", "pni", "fma", "rdctl-no", "sse4.1", "rdseed", "arat", "avx512vl", "avx512f", "pclmuldq", "msr", "fxsr", "sse2", "amd-stibp", "hypervisor", "tsx-ctrl", "clflushopt", "cx16", "clwb", "xgetbv1", "xsavec", "adx", "rdtscp", "mds-no", "cx8", "aes", "tsc-deadline", "pse36", "fsgsbase", "umip", "spec-ctrl", "lahf_lm", "md-clear", "avx512cd", "amd-ssbd", "vmx", "apic", "f16c", "pse", "tsc", "movbe", "smep", "ss", "pschange-mc-no", "ssse3", "popcnt", "avx", "vme", "smap", "pdpe1gb", "mca", "skip-l1dfl-vmentry", "abm", "sep", "clflush", "rdrand"]} | 49 | 3419 | 60 | 0 | 0 | gyw | 1 | 192.168.2.99 | [["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"]] | {"nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3], "pcpuset": [0, 1, 2, 3], "memory": 3931, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0], [1], [2], [3]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 1006396, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": null}, "nova_object.changes": ["cpuset", "pinned_cpus", "mempages", "network_metadata", "cpu_usage", "pcpuset", "memory", "id", "socket", "siblings", "memory_usage"]}]}, "nova_object.changes": ["cells"]} | gyw | 1.5 | 16 | b1bf35bd-a9ad-4f0c-9033-776a5c6d1c9b | 1 | 1 | | 2023-01-04 03:12:17 | 2023-01-31 06:36:36 | 2023-02-23 08:50:29 | 2 | NULL | 4 | 3931 | 60 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 4, "cores": 1, "threads": 1}, "features": ["pclmuldq", "fsgsbase", "f16c", "fxsr", "ibpb", "adx", "movbe", "aes", "x2apic", "abm", "mtrr", "arat", "sse4.2", "bmi1", "stibp", "sse4.1", "pae", "vme", "msr", "skip-l1dfl-vmentry", "fma", "pcid", "avx2", "de", "ibrs-all", "ssse3", "apic", "umip", "xsavec", "3dnowprefetch", "amd-ssbd", "sse", "nx", "fpu", "pse", "smap", "smep", "lahf_lm", "pni", "spec-ctrl", "xsave", "xsaves", "rdtscp", "vmx", "avx512f", "cmov", "invpcid", "hypervisor", "erms", "rdctl-no", "cx16", "cx8", "tsc", "pge", "pdcm", "rdrand", "avx", "amd-stibp", "avx512vl", "xsaveopt", "mds-no", "popcnt", "clflushopt", "sse2", "xgetbv1", "rdseed", "pdpe1gb", "pschange-mc-no", "clwb", "avx512vnni", "mca", "tsx-ctrl", "tsc_adjust", "syscall", "pse36", "mmx", "avx512cd", "avx512bw", "pku", "tsc-deadline", "arch-capabilities", "avx512dq", "ssbd", "clflush", "mce", "ss", "pat", "bmi2", "lm", "ibrs", "sep", "md-clear"]} | 49 | 3419 | 60 | 0 | 0 | c1c1 | 2 | 192.168.2.99 | [["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"]] | {"nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3], "pcpuset": [0, 1, 2, 3], "memory": 3931, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0], [1], [2], [3]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 1006393, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "total", "size_kb", "reserved"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "total", "size_kb", "reserved"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "total", "size_kb", "reserved"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["tunneled", "physnets"]}, "socket": null}, "nova_object.changes": ["memory_usage", "socket", "cpuset", "siblings", "id", "mempages", "pinned_cpus", "memory", "pcpuset", "network_metadata", "cpu_usage"]}]}, "nova_object.changes": ["cells"]} | c1c1 | 1.5 | 16 | 1eac1c8d-d96a-4eeb-9868-5a341a80c6df | 1 | 0 | | 2023-02-07 08:25:27 | 2023-02-07 08:25:27 | 2023-02-13 08:34:22 | 3 | NULL | 12 | 31890 | 456 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["sha-ni", "intel-pt", "pat", "monitor", "movbe", "nx", "msr", "avx2", "md-clear", "popcnt", "rdseed", "pse36", "mds-no", "ds", "sse", "fsrm", "rdctl-no", "pse", "dtes64", "ds_cpl", "xgetbv1", "lahf_lm", "smep", "waitpkg", "smap", "fsgsbase", "sep", "tsc_adjust", "cmov", "ibrs-all", "mtrr", "cx16", "f16c", "arch-capabilities", "pclmuldq", "clflush", "erms", "umip", "xsaves", "xsavec", "ssse3", "acpi", "tsc", "movdir64b", "vpclmulqdq", "skip-l1dfl-vmentry", "xsave", "arat", "mmx", "rdpid", "sse2", "ssbd", "pdpe1gb", "spec-ctrl", "adx", "pcid", "de", "pku", "est", "pae", "tsc-deadline", "pdcm", "clwb", "vme", "rdtscp", "fxsr", "3dnowprefetch", "invpcid", "x2apic", "tm", "lm", "fma", "bmi1", "sse4.1", "abm", "xsaveopt", "pschange-mc-no", "syscall", "clflushopt", "pbe", "avx", "cx8", "vmx", "gfni", "fpu", "mce", "tm2", "movdiri", "invtsc", "apic", "bmi2", "mca", "pge", "rdrand", "xtpr", "sse4.2", "stibp", "ht", "ss", "pni", "vaes", "aes"]} | 416 | 31378 | 456 | 0 | 0 | c-MS-7D42 | 3 | 192.168.2.99 | [["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["aarch64", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppc64le", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]] | {" nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163866, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "reserved", "used", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "reserved", "used", "size_kb"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "reserved", "used", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["network_metadata", "cpuset", "mempages", "id", "socket", "cpu_usage", "memory", "pinned_cpus", "pcpuset", "siblings", "memory_usage"]}]}, "nova_object.changes": ["cells"]} | c-MS-7D42 | 1.5 | 16 | f115a1c2-fda3-42c6-945a-8b54fef40daf | 1 | 0 | | 2023-02-07 09:53:12 | 2023-02-13 08:38:04 | 2023-02-13 08:39:33 | 4 | NULL | 12 | 31890 | 456 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["rdctl-no", "acpi", "umip", "invpcid", "bmi1", "clflushopt", "pclmuldq", "movdir64b", "ssbd", "apic", "rdpid", "ht", "fsrm", "pni", "pse", "xsaves", "cx16", "nx", "f16c", "arat", "popcnt", "mtrr", "vpclmulqdq", "intel-pt", "spec-ctrl", "syscall", "3dnowprefetch", "ds", "mce", "bmi2", "tm2", "md-clear", "fpu", "monitor", "pae", "erms", "dtes64", "tsc", "fsgsbase", "xgetbv1", "est", "mds-no", "tm", "x2apic", "xsavec", "cx8", "stibp", "clflush", "ssse3", "pge", "movdiri", "pdpe1gb", "vaes", "gfni", "mmx", "clwb", "waitpkg", "xsaveopt", "pse36", "aes", "pschange-mc-no", "sse2", "abm", "ss", "pcid", "sep", "rdseed", "mca", "skip-l1dfl-vmentry", "pat", "smap", "sse", "lahf_lm", "avx", "cmov", "sse4.1", "sse4.2", "ibrs-all", "smep", "vme", "tsc_adjust", "arch-capabilities", "fma", "movbe", "adx", "avx2", "xtpr", "pku", "pbe", "rdrand", "tsc-deadline", "pdcm", "ds_cpl", "de", "invtsc", "xsave", "msr", "fxsr", "lm", "vmx", "sha-ni", "rdtscp"]} | 416 | 31378 | 456 | 0 | 0 | c-MS-7D42 | 4 | 192.168.28.21 | [["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["aarch64", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppc64le", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]] | {" nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163866, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "total", "used", "reserved"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "total", "used", "reserved"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "total", "used", "reserved"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["siblings", "cpuset", "mempages", "socket", "pcpuset", "memory", "memory_usage", "id", "network_metadata", "cpu_usage", "pinned_cpus"]}]}, "nova_object.changes": ["cells"]} | c1c2 | 1.5 | 16 | 10ea8254-ad84-4db9-9acd-5c783cb8600e | 1 | 0 | | 2023-02-13 08:41:21 | 2023-02-13 08:41:22 | 2023-02-13 09:56:50 | 5 | NULL | 12 | 31890 | 456 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["bmi2", "ht", "pae", "pku", "monitor", "avx2", "sha-ni", "acpi", "ssbd", "syscall", "mca", "mmx", "mds-no", "erms", "fsrm", "arat", "xsaves", "movbe", "movdir64b", "fpu", "clflush", "nx", "mce", "pse", "cx8", "aes", "avx", "xsavec", "invpcid", "est", "xgetbv1", "fxsr", "rdrand", "vaes", "cmov", "intel-pt", "smep", "dtes64", "f16c", "adx", "sse2", "stibp", "rdseed", "xsave", "skip-l1dfl-vmentry", "sse4.1", "rdpid", "ds", "umip", "pni", "rdctl-no", "clwb", "md-clear", "pschange-mc-no", "msr", "popcnt", "sse4.2", "pge", "tm2", "pat", "xtpr", "fma", "gfni", "sep", "ibrs-all", "tsc", "ds_cpl", "tm", "clflushopt", "pcid", "de", "rdtscp", "vme", "cx16", "lahf_lm", "ss", "pdcm", "x2apic", "pbe", "movdiri", "tsc-deadline", "invtsc", "apic", "fsgsbase", "mtrr", "vpclmulqdq", "ssse3", "3dnowprefetch", "abm", "xsaveopt", "tsc_adjust", "pse36", "pclmuldq", "bmi1", "smap", "arch-capabilities", "lm", "vmx", "sse", "pdpe1gb", "spec-ctrl", "waitpkg"]} | 416 | 31378 | 456 | 0 | 0 | c-MS-7D42 | 5 | 192.168.28.21 | [["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["aarch64", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppc64le", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]] | {"nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163866, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "size_kb", "total", "reserved"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "size_kb", "total", "reserved"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "size_kb", "total", "reserved"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["tunneled", "physnets"]}, "socket": 0}, "nova_object.changes": ["pinned_cpus", "cpuset", "memory_usage", "id", "cpu_usage", "network_metadata", "siblings", "mempages", "socket", "memory", "pcpuset"]}]}, "nova_object.changes": ["cells"]} | c1c2 | 1.5 | 16 | 8efa100f-ab14-45fd-8c39-644b49772883 | 1 | 0 | | 2023-02-13 09:57:30 | 2023-02-13 09:57:31 | 2023-02-13 13:52:57 | 6 | NULL | 12 | 31890 | 456 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["rdpid", "intel-pt", "fxsr", "pclmuldq", "xsaveopt", "pae", "xsave", "movdiri", "syscall", "ibrs-all", "mmx", "tsc_adjust", "abm", "ssbd", "sse", "mce", "clwb", "vmx", "dtes64", "ssse3", "fsrm", "est", "bmi1", "mtrr", "avx2", "pse36", "pat", "gfni", "mds-no", "clflushopt", "cmov", "fma", "sep", "mca", "ss", "umip", "popcnt", "skip-l1dfl-vmentry", "ht", "sha-ni", "pdcm", "pdpe1gb", "rdrand", "pge", "lahf_lm", "aes", "xsavec", "pni", "smep", "md-clear", "waitpkg", "tm", "xgetbv1", "stibp", "apic", "vaes", "fpu", "ds_cpl", "ds", "sse4.2", "3dnowprefetch", "smap", "x2apic", "vpclmulqdq", "acpi", "avx", "de", "pbe", "sse2", "xsaves", "monitor", "clflush", "tm2", "pschange-mc-no", "bmi2", "movbe", "pku", "pcid", "xtpr", "erms", "movdir64b", "cx8", "nx", "rdctl-no", "invpcid", "spec-ctrl", "tsc", "adx", "invtsc", "f16c", "rdtscp", "vme", "pse", "lm", "cx16", "fsgsbase", "rdseed", "msr", "sse4.1", "arch-capabilities", "arat", "tsc-deadline"]} | 416 | 31378 | 456 | 0 | 0 | c-MS-7D42 | 6 | 192.168.28.21 | [["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["aarch64", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppc64le", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]] | {" nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163866, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "total", "reserved"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "total", "reserved"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "total", "reserved"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["memory_usage", "id", "mempages", "pinned_cpus", "network_metadata", "pcpuset", "cpuset", "siblings", "socket", "cpu_usage", "memory"]}]}, "nova_object.changes": ["cells"]} | c1c2 | 1.5 | 16 | 8f5b58c5-d5d7-452c-9ec7-cff24baf6c94 | 1 | 0 | | 2023-02-14 01:35:43 | 2023-02-14 01:35:43 | 2023-02-14 03:16:51 | 7 | NULL | 12 | 31890 | 456 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["pcid", "pse36", "movdir64b", "apic", "nx", "vpclmulqdq", "mtrr", "popcnt", "pdcm", "fsgsbase", "lahf_lm", "sse2", "pae", "aes", "movdiri", "xsaves", "erms", "invtsc", "waitpkg", "pbe", "ht", "pni", "avx2", "rdpid", "fxsr", "tm2", "pku", "x2apic", "fma", "pge", "rdseed", "pdpe1gb", "mmx", "sse4.1", "sha-ni", "xtpr", "tsc_adjust", "cx16", "xsave", "cx8", "mce", "md-clear", "gfni", "clwb", "msr", "abm", "f16c", "ss", "xsaveopt", "ds_cpl", "pse", "syscall", "cmov", "3dnowprefetch", "ssse3", "pclmuldq", "arch-capabilities", "ibrs-all", "arat", "ds", "pat", "invpcid", "vaes", "xsavec", "mds-no", "tm", "smep", "acpi", "fsrm", "movbe", "fpu", "sse4.2", "umip", "rdtscp", "tsc-deadline", "skip-l1dfl-vmentry", "est", "rdctl-no", "clflush", "spec-ctrl", "tsc", "lm", "avx", "vmx", "clflushopt", "rdrand", "dtes64", "smap", "ssbd", "sse", "xgetbv1", "stibp", "mca", "adx", "vme", "bmi1", "pschange-mc-no", "intel-pt", "de", "monitor", "bmi2", "sep"]} | 416 | 31378 | 456 | 0 | 0 | c-MS-7D42 | 7 | 192.168.28.21 | [["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["aarch64", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppc64le", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]] | {"nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163866, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "total", "used", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "total", "used", "size_kb"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "total", "used", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["tunneled", "physnets"]}, "socket": 0}, "nova_object.changes": ["pcpuset", "mempages", "socket", "cpu_usage", "memory_usage", "siblings", "memory", "pinned_cpus", "network_metadata", "cpuset", "id"]}]}, "nova_object.changes": ["cells"]} | c1c2 | 1.5 | 16 | 9f81894d-028d-4bb0-ada0-116159b13ed8 | 1 | 0 | | 2023-02-14 03:19:40 | 2023-02-28 07:30:14 | NULL | 8 | NULL | 12 | 31890 | 456 | 0 | 512 | 0 | QEMU | 4002001 | {"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["ds_cpl", "vpclmulqdq", "pbe", "lahf_lm", "clflush", "xsaveopt", "bmi2", "clwb", "skip-l1dfl-vmentry", "de", "cmov", "pku", "acpi", "xsavec", "3dnowprefetch", "mmx", "mtrr", "vmx", "movbe", "ssse3", "adx", "syscall", "spec-ctrl", "abm", "umip", "fxsr", "pge", "vme", "xsave", "rdctl-no", "rdrand", "sse", "cx8", "gfni", "dtes64", "invpcid", "md-clear", "fsrm", "fma", "arat", "rdtscp", "tsc", "sse4.1", "xsaves", "est", "tm", "ibrs-all", "stibp", "fsgsbase", "cx16", "arch-capabilities", "ssbd", "pse", "avx", "pse36", "pdpe1gb", "tm2", "nx", "pdcm", "sse4.2", "erms", "sep", "aes", "mca", "invtsc", "bmi1", "rdseed", "ss", "ht", "x2apic", "movdir64b", "rdpid", "lm", "tsc-deadline", "msr", "mce", "smep", "avx2", "sse2", "pschange-mc-no", "intel-pt", "tsc_adjust", "ds", "monitor", "popcnt", "f16c", "pae", "movdiri", "xgetbv1", "pcid", "pni", "pclmuldq", "xtpr", "apic", "mds-no", "sha-ni", "smap", "vaes", "waitpkg", "pat", "fpu", "clflushopt"]} | 415 | 31378 | 456 | 0 | 0 | c1c2 | 0 | 192.168.28.21 | [["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["aarch64", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppc64le", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]] | {"nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": []}, "nova_object.changes": ["objects"]} | [] | NULL | {"failed_builds": "0"} | {"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163962, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "reserved", "total", "used"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "reserved", "total", "used"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "reserved", "total", "used"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["tunneled", "physnets"]}, "socket": 0}, "nova_object.changes": ["cpu_usage", "siblings", "id", "pcpuset", "cpuset", "network_metadata", "memory", "pinned_cpus", "memory_usage", "socket", "mempages"]}]}, "nova_object.changes": ["cells"]} | c1c2 | 1.5 | 16 | c360cc82-f0fd-4662-bccd-e1f02b27af51 | 1 | 0 | +---------------------+---------------------+---------------------+----+------------+-------+-----------+----------+------------+----------------+---------------+-----------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------+--------------+------------------+-------------+---------------------+---------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-----------------+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------+----------------------+--------------------------------------+-----------------------+--------+ 8 rows in set (0.001 sec) ``` Then I search code, I found these infomation is upload in compute node's nova-compute, in the init process in nova/compute/resource_tracker.py:_init_compute_node Then I add some debug log, I found in _init_compute_node, there are PCI configure which read from /etc/nova/nova.conf, but in _setup_pci_tracker pop these infomation, this is code: ``` def _setup_pci_tracker(self, context, compute_node, resources): if not self.pci_tracker: LOG.debug("_setup_pci_tracker 11111") LOG.debug(resources) # 2023-02-28 14:01:33.782 443293 DEBUG nova.compute.resource_tracker [req-52743217-f0b6-4525-b9fd-371246b6c929 - - - - -] _setup_pci_tracker 11111 _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:772 # 2023-02-28 14:01:33.782 443293 DEBUG nova.compute.resource_tracker [req-52743217-f0b6-4525-b9fd-371246b6c929 - - - - -] {'supported_instances': [('alpha', 'qemu', 'hvm'), ('armv7l', 'qemu', 'hvm'), ('aarch64', 'qemu', 'hvm'), ('cris', 'qemu', 'hvm'), ('i686', 'qemu', 'hvm'), ('i686', 'kvm', 'hvm'), ('lm32', 'qemu', 'hvm'), ('m68k', 'qemu', 'hvm'), ('microblaze', 'qemu', 'hvm'), ('microblazeel', 'qemu', 'hvm'), ('mips', 'qemu', 'hvm'), ('mipsel', 'qemu', 'hvm'), ('mips64', 'qemu', 'hvm'), ('mips64el', 'qemu', 'hvm'), ('ppc', 'qemu', 'hvm'), ('ppc64', 'qemu', 'hvm'), ('ppc64le', 'qemu', 'hvm'), ('s390x', 'qemu', 'hvm'), ('sh4', 'qemu', 'hvm'), ('sh4eb', 'qemu', 'hvm'), ('sparc', 'qemu', 'hvm'), ('sparc64', 'qemu', 'hvm'), ('unicore32', 'qemu', 'hvm'), ('x86_64', 'qemu', 'hvm'), ('x86_64', 'kvm', 'hvm'), ('xtensa', 'qemu', 'hvm'), ('xtensaeb', 'qemu', 'hvm')], 'vcpus': 12, 'memory_mb': 31890, 'local_gb': 456.8854446411133, 'vcpus_used': 0, 'memory_mb_used': 3856, 'local_gb_used': 18.015335083007812, 'hypervisor_type': 'QEMU', 'hypervisor_version': 4002001, 'hypervisor_hostname': 'c1c2', 'cpu_info': '{"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["sse4.2", "mds-no", "stibp", "pdpe1gb", "xsaveopt", "ht", "intel-pt", "mtrr", "abm", "tm", "lm", "umip", "mca", "pku", "ds_cpl", "rdrand", "adx", "rdseed", "lahf_lm", "xgetbv1", "nx", "invpcid", "rdtscp", "tsc", "xsavec", "pcid", "arch-capabilities", "pclmuldq", "spec-ctrl", "fsgsbase", "avx2", "md-clear", "vmx", "syscall", "mmx", "ds", "ssse3", "avx", "dtes64", "fxsr", "msr", "acpi", "vpclmulqdq", "smap", "erms", "pge", "cmov", "sha-ni", "fsrm", "x2apic", "xsaves", "cx8", "pse", "pse36", "clflushopt", "vaes", "pni", "ssbd", "movdiri", "movbe", "clwb", "xtpr", "de", "invtsc", "fpu", "tsc-deadline", "pae", "clflush", "ibrs-all", "waitpkg", "sse", "sse2", "bmi1", "3dnowprefetch", "cx16", "popcnt", "rdctl-no", "fma", "tsc_adjust", "xsave", "ss", "skip-l1dfl-vmentry", "sse4.1", "rdpid", "monitor", "vme", "tm2", "pat", "pschange-mc-no", "movdir64b", "gfni", "mce", "smep", "sep", "apic", "arat", "f16c", "bmi2", "aes", "pbe", "est", "pdcm"]}', 'disk_available_least': 415.5914993286133, 'pci_passthrough_devices': '[{"dev_id": "pci_0000_02_00_0", "address": "0000:02:00.0", "product_id": "a809", "vendor_id": "144d", "numa_node": 0, "label": "label_144d_a809", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_01_00_5", "address": "0000:01:00.5", "product_id": "101e", "vendor_id": "15b3", "numa_node": null, "label": "label_15b3_101e", "dev_type": "type-VF", "parent_addr": "0000:01:00.0", "parent_ifname": "enp1s0f0np0", "capabilities": {"network": ["rx", "tx", "sg", "tso", "gso", "gro", "rxvlan", "txvlan", "rxhash", "rdma", "txudptnl"]}}, {"dev_id": "pci_0000_00_1c_0", "address": "0000:00:1c.0", "product_id": "7ab8", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7ab8", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_08_0", "address": "0000:00:08.0", "product_id": "464f", "vendor_id": "8086", "numa_node": null, "label": "label_8086_464f", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_1f_4", "address": "0000:00:1f.4", "product_id": "7aa3", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7aa3", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_01_00_3", "address": "0000:01:00.3", "product_id": "101e", "vendor_id": "15b3", "numa_node": null, "label": "label_15b3_101e", "dev_type": "type-VF", "parent_addr": "0000:01:00.0", "parent_ifname": "enp1s0f0np0", "capabilities": {"network": ["rx", "tx", "sg", "tso", "gso", "gro", "rxvlan", "txvlan", "rxhash", "rdma", "txudptnl"]}}, {"dev_id": "pci_0000_00_06_0", "address": "0000:00:06.0", "product_id": "464d", "vendor_id": "8086", "numa_node": null, "label": "label_8086_464d", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_17_0", "address": "0000:00:17.0", "product_id": "7ae2", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7ae2", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_1f_5", "address": "0000:00:1f.5", "product_id": "7aa4", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7aa4", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_01_00_6", "address": "0000:01:00.6", "product_id": "101e", "vendor_id": "15b3", "numa_node": null, "label": "label_15b3_101e", "dev_type": "type-VF", "parent_addr": "0000:01:00.0", "parent_ifname": "enp1s0f0np0", "capabilities": {"network": ["rx", "tx", "sg", "tso", "gso", "gro", "rxvlan", "txvlan", "rxhash", "rdma", "txudptnl"]}}, {"dev_id": "pci_0000_00_1f_0", "address": "0000:00:1f.0", "product_id": "7a86", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7a86", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_02_0", "address": "0000:00:02.0", "product_id": "4692", "vendor_id": "8086", "numa_node": null, "label": "label_8086_4692", "dev_type": "type-PF"}, {"dev_id": "pci_0000_00_16_0", "address": "0000:00:16.0", "product_id": "7ae8", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7ae8", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_1c_1", "address": "0000:00:1c.1", "product_id": "7ab9", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7ab9", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_1c_3", "address": "0000:00:1c.3", "product_id": "7abb", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7abb", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_04_00_0", "address": "0000:04:00.0", "product_id": "8125", "vendor_id": "10ec", "numa_node": null, "label": "label_10ec_8125", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_00_0", "address": "0000:00:00.0", "product_id": "4650", "vendor_id": "8086", "numa_node": null, "label": "label_8086_4650", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_01_0", "address": "0000:00:01.0", "product_id": "460d", "vendor_id": "8086", "numa_node": null, "label": "label_8086_460d", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_01_00_0", "address": "0000:01:00.0", "product_id": "a2d6", "vendor_id": "15b3", "numa_node": null, "label": "label_15b3_a2d6", "dev_type": "type-PF", "mac_address": "08:c0:eb:8e:bd:f4"}, {"dev_id": "pci_0000_01_00_2", "address": "0000:01:00.2", "product_id": "c2d3", "vendor_id": "15b3", "numa_node": null, "label": "label_15b3_c2d3", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_14_2", "address": "0000:00:14.2", "product_id": "7aa7", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7aa7", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_14_3", "address": "0000:00:14.3", "product_id": "7af0", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7af0", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_00_1f_3", "address": "0000:00:1f.3", "product_id": "7ad0", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7ad0", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_01_00_4", "address": "0000:01:00.4", "product_id": "101e", "vendor_id": "15b3", "numa_node": null, "label": "label_15b3_101e", "dev_type": "type-VF", "parent_addr": "0000:01:00.0", "parent_ifname": "enp1s0f0np0", "capabilities": {"network": ["rx", "tx", "sg", "tso", "gso", "gro", "rxvlan", "txvlan", "rxhash", "rdma", "txudptnl"]}}, {"dev_id": "pci_0000_05_00_0", "address": "0000:05:00.0", "product_id": "0612", "vendor_id": "1b21", "numa_node": null, "label": "label_1b21_0612", "dev_type": "type-PCI"}, {"dev_id": "pci_0000_01_00_1", "address": "0000:01:00.1", "product_id": "a2d6", "vendor_id": "15b3", "numa_node": null, "label": "label_15b3_a2d6", "dev_type": "type-PF", "mac_address": "08:c0:eb:8e:bd:f5"}, {"dev_id": "pci_0000_00_14_0", "address": "0000:00:14.0", "product_id": "7ae0", "vendor_id": "8086", "numa_node": null, "label": "label_8086_7ae0", "dev_type": "type-PCI"}]', 'numa_topology': '{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163962, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["cpuset", "memory_usage", "cpu_usage", "id", "pinned_cpus", "pcpuset", "socket", "network_metadata", "siblings", "mempages", "memory"]}]}, "nova_object.changes": ["cells"]}', 'host_ip': '192.168.28.21'} _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:773 self.pci_tracker = pci_manager.PciDevTracker(context, compute_node) if 'pci_passthrough_devices' in resources: dev_json = resources.pop('pci_passthrough_devices') self.pci_tracker.update_devices_from_hypervisor_resources( dev_json) LOG.debug("_setup_pci_tracker 22222") LOG.debug(resources) # 2023-02-28 14:01:33.798 443293 DEBUG nova.compute.resource_tracker [req-52743217-f0b6-4525-b9fd-371246b6c929 - - - - -] _setup_pci_tracker 22222 _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:781 # 2023-02-28 14:01:33.799 443293 DEBUG nova.compute.resource_tracker [req-52743217-f0b6-4525-b9fd-371246b6c929 - - - - -] {'supported_instances': [('alpha', 'qemu', 'hvm'), ('armv7l', 'qemu', 'hvm'), ('aarch64', 'qemu', 'hvm'), ('cris', 'qemu', 'hvm'), ('i686', 'qemu', 'hvm'), ('i686', 'kvm', 'hvm'), ('lm32', 'qemu', 'hvm'), ('m68k', 'qemu', 'hvm'), ('microblaze', 'qemu', 'hvm'), ('microblazeel', 'qemu', 'hvm'), ('mips', 'qemu', 'hvm'), ('mipsel', 'qemu', 'hvm'), ('mips64', 'qemu', 'hvm'), ('mips64el', 'qemu', 'hvm'), ('ppc', 'qemu', 'hvm'), ('ppc64', 'qemu', 'hvm'), ('ppc64le', 'qemu', 'hvm'), ('s390x', 'qemu', 'hvm'), ('sh4', 'qemu', 'hvm'), ('sh4eb', 'qemu', 'hvm'), ('sparc', 'qemu', 'hvm'), ('sparc64', 'qemu', 'hvm'), ('unicore32', 'qemu', 'hvm'), ('x86_64', 'qemu', 'hvm'), ('x86_64', 'kvm', 'hvm'), ('xtensa', 'qemu', 'hvm'), ('xtensaeb', 'qemu', 'hvm')], 'vcpus': 12, 'memory_mb': 31890, 'local_gb': 456.8854446411133, 'vcpus_used': 0, 'memory_mb_used': 3856, 'local_gb_used': 18.015335083007812, 'hypervisor_type': 'QEMU', 'hypervisor_version': 4002001, 'hypervisor_hostname': 'c1c2', 'cpu_info': '{"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["sse4.2", "mds-no", "stibp", "pdpe1gb", "xsaveopt", "ht", "intel-pt", "mtrr", "abm", "tm", "lm", "umip", "mca", "pku", "ds_cpl", "rdrand", "adx", "rdseed", "lahf_lm", "xgetbv1", "nx", "invpcid", "rdtscp", "tsc", "xsavec", "pcid", "arch-capabilities", "pclmuldq", "spec-ctrl", "fsgsbase", "avx2", "md-clear", "vmx", "syscall", "mmx", "ds", "ssse3", "avx", "dtes64", "fxsr", "msr", "acpi", "vpclmulqdq", "smap", "erms", "pge", "cmov", "sha-ni", "fsrm", "x2apic", "xsaves", "cx8", "pse", "pse36", "clflushopt", "vaes", "pni", "ssbd", "movdiri", "movbe", "clwb", "xtpr", "de", "invtsc", "fpu", "tsc-deadline", "pae", "clflush", "ibrs-all", "waitpkg", "sse", "sse2", "bmi1", "3dnowprefetch", "cx16", "popcnt", "rdctl-no", "fma", "tsc_adjust", "xsave", "ss", "skip-l1dfl-vmentry", "sse4.1", "rdpid", "monitor", "vme", "tm2", "pat", "pschange-mc-no", "movdir64b", "gfni", "mce", "smep", "sep", "apic", "arat", "f16c", "bmi2", "aes", "pbe", "est", "pdcm"]}', 'disk_available_least': 415.5914993286133, 'numa_topology': '{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163962, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["cpuset", "memory_usage", "cpu_usage", "id", "pinned_cpus", "pcpuset", "socket", "network_metadata", "siblings", "mempages", "memory"]}]}, "nova_object.changes": ["cells"]}', 'host_ip': '192.168.28.21'} _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:782 dev_pools_obj = self.pci_tracker.stats.to_device_pools_obj() LOG.debug("_setup_pci_tracker 44444") LOG.debug(dev_pools_obj) compute_node.pci_device_pools = dev_pools_obj LOG.debug("_setup_pci_tracker 33333") LOG.debug(resources) LOG.debug(compute_node) # 2023-02-28 14:01:33.799 443293 DEBUG nova.compute.resource_tracker [req-52743217-f0b6-4525-b9fd-371246b6c929 - - - - -] _setup_pci_tracker 33333 _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:786 # 2023-02-28 14:01:33.800 443293 DEBUG nova.compute.resource_tracker [req-52743217-f0b6-4525-b9fd-371246b6c929 - - - - -] {'supported_instances': [('alpha', 'qemu', 'hvm'), ('armv7l', 'qemu', 'hvm'), ('aarch64', 'qemu', 'hvm'), ('cris', 'qemu', 'hvm'), ('i686', 'qemu', 'hvm'), ('i686', 'kvm', 'hvm'), ('lm32', 'qemu', 'hvm'), ('m68k', 'qemu', 'hvm'), ('microblaze', 'qemu', 'hvm'), ('microblazeel', 'qemu', 'hvm'), ('mips', 'qemu', 'hvm'), ('mipsel', 'qemu', 'hvm'), ('mips64', 'qemu', 'hvm'), ('mips64el', 'qemu', 'hvm'), ('ppc', 'qemu', 'hvm'), ('ppc64', 'qemu', 'hvm'), ('ppc64le', 'qemu', 'hvm'), ('s390x', 'qemu', 'hvm'), ('sh4', 'qemu', 'hvm'), ('sh4eb', 'qemu', 'hvm'), ('sparc', 'qemu', 'hvm'), ('sparc64', 'qemu', 'hvm'), ('unicore32', 'qemu', 'hvm'), ('x86_64', 'qemu', 'hvm'), ('x86_64', 'kvm', 'hvm'), ('xtensa', 'qemu', 'hvm'), ('xtensaeb', 'qemu', 'hvm')], 'vcpus': 12, 'memory_mb': 31890, 'local_gb': 456.8854446411133, 'vcpus_used': 0, 'memory_mb_used': 3856, 'local_gb_used': 18.015335083007812, 'hypervisor_type': 'QEMU', 'hypervisor_version': 4002001, 'hypervisor_hostname': 'c1c2', 'cpu_info': '{"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["sse4.2", "mds-no", "stibp", "pdpe1gb", "xsaveopt", "ht", "intel-pt", "mtrr", "abm", "tm", "lm", "umip", "mca", "pku", "ds_cpl", "rdrand", "adx", "rdseed", "lahf_lm", "xgetbv1", "nx", "invpcid", "rdtscp", "tsc", "xsavec", "pcid", "arch-capabilities", "pclmuldq", "spec-ctrl", "fsgsbase", "avx2", "md-clear", "vmx", "syscall", "mmx", "ds", "ssse3", "avx", "dtes64", "fxsr", "msr", "acpi", "vpclmulqdq", "smap", "erms", "pge", "cmov", "sha-ni", "fsrm", "x2apic", "xsaves", "cx8", "pse", "pse36", "clflushopt", "vaes", "pni", "ssbd", "movdiri", "movbe", "clwb", "xtpr", "de", "invtsc", "fpu", "tsc-deadline", "pae", "clflush", "ibrs-all", "waitpkg", "sse", "sse2", "bmi1", "3dnowprefetch", "cx16", "popcnt", "rdctl-no", "fma", "tsc_adjust", "xsave", "ss", "skip-l1dfl-vmentry", "sse4.1", "rdpid", "monitor", "vme", "tm2", "pat", "pschange-mc-no", "movdir64b", "gfni", "mce", "smep", "sep", "apic", "arat", "f16c", "bmi2", "aes", "pbe", "est", "pdcm"]}', 'disk_available_least': 415.5914993286133, 'numa_topology': '{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163962, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["cpuset", "memory_usage", "cpu_usage", "id", "pinned_cpus", "pcpuset", "socket", "network_metadata", "siblings", "mempages", "memory"]}]}, "nova_object.changes": ["cells"]}', 'host_ip': '192.168.28.21'} _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:787 # 2023-02-28 14:01:33.800 443293 DEBUG nova.compute.resource_tracker [req-52743217-f0b6-4525-b9fd-371246b6c929 - - - - -] ComputeNode(cpu_allocation_ratio=16.0,cpu_info='{"arch": "x86_64", "model": "Broadwell-noTSX-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 6, "threads": 2}, "features": ["sse4.2", "mds-no", "stibp", "pdpe1gb", "xsaveopt", "ht", "intel-pt", "mtrr", "abm", "tm", "lm", "umip", "mca", "pku", "ds_cpl", "rdrand", "adx", "rdseed", "lahf_lm", "xgetbv1", "nx", "invpcid", "rdtscp", "tsc", "xsavec", "pcid", "arch-capabilities", "pclmuldq", "spec-ctrl", "fsgsbase", "avx2", "md-clear", "vmx", "syscall", "mmx", "ds", "ssse3", "avx", "dtes64", "fxsr", "msr", "acpi", "vpclmulqdq", "smap", "erms", "pge", "cmov", "sha-ni", "fsrm", "x2apic", "xsaves", "cx8", "pse", "pse36", "clflushopt", "vaes", "pni", "ssbd", "movdiri", "movbe", "clwb", "xtpr", "de", "invtsc", "fpu", "tsc-deadline", "pae", "clflush", "ibrs-all", "waitpkg", "sse", "sse2", "bmi1", "3dnowprefetch", "cx16", "popcnt", "rdctl-no", "fma", "tsc_adjust", "xsave", "ss", "skip-l1dfl-vmentry", "sse4.1", "rdpid", "monitor", "vme", "tm2", "pat", "pschange-mc-no", "movdir64b", "gfni", "mce", "smep", "sep", "apic", "arat", "f16c", "bmi2", "aes", "pbe", "est", "pdcm"]}',created_at=2023-02-14T03:19:40Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=415,free_disk_gb=456,free_ram_mb=31378,host='c1c2',host_ip=192.168.28.21,hypervisor_hostname='c1c2',hypervisor_type='QEMU',hypervisor_version=4002001,id=8,local_gb=456,local_gb_used=18,mapped=0,memory_mb=31890,memory_mb_used=3856,metrics='[]',numa_topology='{" nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{" nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.5", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], "memory": 31890, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[0, 1], [10, 11], [2, 3], [6, 7], [4, 5], [8, 9]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 8163962, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {" nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["size_kb", "used", "reserved", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}, "socket": 0}, "nova_object.changes": ["cpuset", "memory_usage", "cpu_usage", "id", "pinned_cpus", "pcpuset", "socket", "network_metadata", "siblings", "mempages", "memory"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.5,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2023-02-28T06:00:15Z,uuid=c360cc82-f0fd-4662-bccd-e1f02b27af51,vcpus=12,vcpus_used=0) _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:788 LOG.debug(compute_node.pci_device_pools.fields) # 2023-02-28 15:27:57.213 447003 DEBUG nova.compute.resource_tracker [req-7f09d102-95de-4b6f-a0f7-d6ee2fc26e6d - - - - -] {'objects': List(default=<class 'oslo_versionedobjects.fields.UnspecifiedDefault'>,nullable=False)} _setup_pci_tracker /usr/lib/python3/dist-packages/nova/compute/resource_tracker.py:789 ``` As above, this code "dev_json = resources.pop('pci_passthrough_devices')" pop these infomation. So my question is 1) is this BUG ? 2) what update_devices_from_hypervisor_resources do ? 3) how to use PCI passthough in openstack Yoga ? ---- Simon Jones

4 13

[neutron] Drivers meeting cancelled
by Rodolfo Alonso Hernandez 10 Mar '23

10 Mar '23

Hello Neutrinos: Due to the lack of agenda, today's drivers meeting is cancelled. Have a nice weekend!

1 0

[barbican] Canceling weekly meeting (March 14th) + PTG discussion topics
by Grzegorz Grasza 10 Mar '23

10 Mar '23

Hi all, I'm on PTO next week, so I'm canceling the weekly meeting. We have just one last meeting before the Virtual PTG, so I booked a time slot for us on Tuesday, March 28th at 13:00 UTC. Please add topics you would like to discuss to the etherpad: https://etherpad.opendev.org/p/march2023-ptg-barbican If there are more things to discuss, I'll book another time slot later in the week. Thanks, / Greg

1 0

[all] broken pepe8 jobs caused by bandit 1.7.5
by Takashi Kajinami 10 Mar '23

10 Mar '23

fyi; It seems the new release of bandit (1.7.5) just came out and this introduces a new lint rule to require defining the timeout parameter for all "requests" calls. https://github.com/PyCQA/bandit/commit/5ff73ff8ff956df7d63fde49c3bd671db8e8… This is currently affecting heat and quick search shows some of the other projects contain some code not compliant with this rule(barbican, ceilometer, cinder, glance, manila, nova, ...). Also, it seems we do not pin bandit by u-c for some reason this likely affects all stable branches. Actually I first noticed this when I tried to backport one fix to 2023.1 branch of heat... Thank you, Takashi

2 2

[neutron][ovn] domain-search dhcp option per network/subnet level
by Михаил 09 Mar '23

09 Mar '23

Greetings to all! In my openstack setup (vanilla ubuntu 22.04 Zed release + OVN networking ) I try to find a way to setup a scheme where my instances should get dhcp search domain option on network/subnet level. For example - when I create network/subnet, I want to tell neutron, that all created ports in that network/subnet must receive my dhcp search-domain option Any help advices will appreciated. Thank you! -- Best regards, Mikhail.

2 2

[manila] Share configuration with cinder as a backend
by Jan Wasilewski 09 Mar '23

09 Mar '23

Hi, I am looking for instructions on how to configure a Manila service with Cinder as a backend. I have gone through the https://github.com/openstack/manila/blob/master/doc/source/configuration/sh… page, and I am wondering if someone has a link to a preconfigured "golden image" that can be used as a service_image. Additionally, I am wondering if this configuration can be used with driver_handles_share_servers=True (generally, I see that both options are supported here), but are there any specific limitations? I have already configured other backends (standalone ZFS, NFS, and Huawei driver), but I would like to test share snapshot retrieval using Cinder as a backend. Unfortunately, as I see, Cinder is not as straightforward as other backends(or no one is using it), which is why I am asking for some hints. Thanks in advance /Jan

2 1

[barbican] Canceling weekly meeting (March 14th) + PTG discussion topics
by Grzegorz Grasza 09 Mar '23

09 Mar '23

Hi all, I'm on PTO next week, so I'm canceling the weekly meeting. We have just one last meeting before the Virtual PTG, so I booked a time slot for us on Tuesday, March 28th at 13:00 UTC. Please add topics you would like to discuss to the etherpad: https://etherpad.opendev.org/p/march2023-ptg-barbican If there are more things to discuss, I'll book another time slot later in the week. Thanks, / Greg

1 0

[neutron][largescale-sig] agent_down_time and report_interval
by Arnaud Morin 09 Mar '23

09 Mar '23

Hello neutron and large-scalers, Is there any recommendation on tuning the neutron report_interval (agent) and agent_down_time (server) to "optimize" the communication between agents and servers without putting to much heavy duty on both rabbit and database? We are currently facing some scaling issue regarding this, and we found out that, at least, CERN did some tweak about this ([1] and [2]) Is there anyone else with specific configuration on that part? I have the feeling that this could be increased (so report will happen less often). One obvious side effect of this is the fact that the server will take more time to see a down agent, is there any other side effect that could happen? Is it something we can eventually add in our large-scale documentation ([3])? Cheers, [1] https://youtu.be/5WL47L1P5kE?t=1173 [2] https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c0… [3] https://docs.openstack.org/large-scale/journey/configure/index.html

2 2

Re: [cinder] Error when creating backups from iscsi volume
by Gorka Eguileor 09 Mar '23

09 Mar '23

On 06/03, Rishat Azizov wrote: > Hi, > > It works with smaller volumes. > > multipath.conf attached to thist email. > > Cinder version - 18.2.0 Wallaby Hi, After giving it some thought I think I may know what is going on. If you have DEBUG logs enabled in cinder-backup when it fails, how many calls do you see in the cinder-backup to "multipath -f" from os-brick, only one or do you see more? Cheers, Gorka. > > пн, 6 мар. 2023 г. в 17:35, Gorka Eguileor <geguileo(a)redhat.com>: > > > On 16/02, Rishat Azizov wrote: > > > Hello! > > > > > > We have an error with creating backups from iscsi volume. Usually, this > > > happens with large backups over 100GB. > > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > [req-f6619913-6f96-4226-8d75-2da3fca722f1 > > 23de1b92e7674cf59486f07ac75b886b > > > a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message > > handling: > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > while > > > running command. > > > Command: multipath -f 3624a93705842cfae35d7483200015ec6 > > > Exit code: 1 > > > Stdout: '' > > > Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a > > > multipath device\n' > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Traceback > > > (most recent call last): > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line > > 165, > > > in _process_incoming > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server res = > > > self.dispatcher.dispatch(message) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line > > > 309, in dispatch > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > self._do_dispatch(endpoint, method, ctxt, args) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line > > > 229, in _do_dispatch > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server result = > > > func(ctxt, **new_args) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > func(self, *args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in > > > create_backup > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > volume_utils.update_backup_error(backup, str(err)) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in > > > __exit__ > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > self.force_reraise() > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in > > > force_reraise > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server raise > > > self.value > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in > > > create_backup > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server updates > > = > > > self._run_backup(context, backup, volume) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in > > > _run_backup > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > ignore_errors=True) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066, > > in > > > _detach_device > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > force=force, ignore_errors=ignore_errors) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in > > > trace_logging_wrapper > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > f(*args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line > > 360, > > > in inner > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > f(*args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > line 880, in disconnect_volume > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > is_disconnect_call=True) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > line 942, in _cleanup_connection > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > self._linuxscsi.flush_multipath_device(multipath_name) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line > > > 382, in flush_multipath_device > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > root_helper=self._root_helper) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in > > > _execute > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server result = > > > self.__execute(*args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line > > > 172, in execute > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > execute_root(*cmd, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line > > 247, > > > in _wrap > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > self.channel.remote_call(name, args, kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in > > > remote_call > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server raise > > > exc_type(*result[2]) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > while > > > running command. > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command: > > > multipath -f 3624a93705842cfae35d7483200015ec6 > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit code: 1 > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: '' > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: 'Feb > > > 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath > > device\n' > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > > Could you please help with this error? > > > > Hi, > > > > Does it work for smaller volumes or does it also fail? > > > > What are your defaults in your /etc/multipath.conf file? > > > > What Cinder release are you using? > > > > Cheers, > > Gorka. > > > > > defaults { > user_friendly_names no > find_multipaths yes > enable_foreign "^$" > } > > blacklist_exceptions { > property "(SCSI_IDENT_|ID_WWN)" > } > > blacklist { > } > > devices { > device { > vendor "PURE" > product "FlashArray" > fast_io_fail_tmo 10 > path_grouping_policy "group_by_prio" > failback "immediate" > prio "alua" > hardware_handler "1 alua" > max_sectors_kb 4096 > } > }

1 0

OpenInfra Live March 9, 2023, at 9 a.m. CT
by Kristin Barrientos 08 Mar '23

08 Mar '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by OpenMetal Episode: Examination of Cost Differences b/w Private Cloud and Hyperscalers Todd Robinson, president of OpenMetal, will be discussing the examination of cost differences between private cloud and hyperscalers. Date and time: March 9, 2023, at 9 a.m. CT (15:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=6GCGhuRpPqM LinkedIn: https://www.linkedin.com/events/7038909988492242944/comments/ WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Todd Robinson Have an idea for a future episode? Share it now at ideas.openinfra.live. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[Magnum] Magnum service status in Openstack Antelope
by A Tom 08 Mar '23

08 Mar '23

Hi, Will magnum be supported in Antelope and further releases of Openstack? I was looking for some informations about this and I’m quite confused, because here I don’t see Magnum service: https://releases.openstack.org/antelope/index.html and here Magnum is mentioned: https://docs.openstack.org/2023.1.antelope/projects.html Which version of kubernetes is supported in Zed, and which (if) will be in Antelope? Because there’s no such information in compatibility matrix: https://wiki.openstack.org/wiki/Magnum Best regards, Adam Tomas

3 2

[magnum] security groups for magnum nodes
by Vivian Rook 08 Mar '23

08 Mar '23

Is there an option for adding security groups to a given magnum template, and thus the nodes that such a template would create? I have an NFS server, and it is setup to only allow connections from nodes with the "nfs" security group. A few pods in my cluster mount the NFS server, and are blocked as a result. Is it possible to setup magnum so that it adds the "nfs" security group to the worker nodes (it would be alright if it has to be worker and control nodes)? Thank you! -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

2 2

[neutron] New Neutron releases for Xena, Yoga and Zed
by Rodolfo Alonso Hernandez 08 Mar '23

08 Mar '23

Hello Neutrinos: Please check the patches for the new stable releases for the Neutron projects: * Xena: https://review.opendev.org/c/openstack/releases/+/876827 * Yoga: https://review.opendev.org/c/openstack/releases/+/876828 * Zed: https://review.opendev.org/c/openstack/releases/+/876835 Feel free to comment on the patch if you found something wrong or a missing pending patch. Regards.

1 0

[largescale-sig] Next meeting: March 8, 9utc
by Thierry Carrez 08 Mar '23

08 Mar '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC, our APAC+EU-friendly time. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230308T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[horizon] Cancelling Today's Weekly meeting
by vishal manchanda 08 Mar '23

08 Mar '23

Hello Team, As it is a holiday for me, I will not be able to host today's horizon weekly meeting. So let's cancel today's weekly meeting. If anything urgent, please reach out to the horizon core team. Thanks & regards, Vishal Manchanda

1 0

[cinder] Could I use system lvm volume group for cinder-volume
by 韩光宇 08 Mar '23

08 Mar '23

Hello, I have a physical node and the system uses lvm partitions. With the service already running above, is it possible for me to deploy cinder-volume that uses the lvm backend without affecting the existing service? The physical disk space is basically allocated to the /dev/sda4 physical volume, and the pv wad added to the `uniontechos` volume group. All the space of `uniontechos` vg is allocated to the system logical volume (/dev/uniontechos/root), there is 5T space (a lot of free space). I want to try to reduce the system logical volume——/dev/uniontechos/root, and let cinder-volume use the uniontechos volume group at the same time , I don't know if it is feasible. Could I get some advices? Thank for any help! ```shell # pvdisplay ... --- Physical volume --- PV Name /dev/sda4 VG Name uniontechos PV Size <5.43 TiB / not usable 2.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 1422313 Free PE 0 Allocated PE 1422313 PV UUID 8LF270-LYD1-kuP1-iWZb-BZEQ-LdA8-UwEa58 # lvdisplay --- Logical volume --- LV Path /dev/uniontechos/root LV Name root VG Name uniontechos LV UUID cEypcY-xcbC-JFoO-d3MS-uMPU-ntQa-WEeE1c LV Write Access read/write LV Creation host, time compute2, 2022-04-28 15:21:27 +0800 LV Status available # open 1 LV Size 5.42 TiB Current LE 1421289 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0 ``` Best wishes, Han

1 0

[magnum] certificate authority key location
by Vivian Rook 07 Mar '23

07 Mar '23

If I want to create a credential for a user to access a magnum cluster I can do so as described in https://docs.openstack.org/magnum/latest/user/#id4 Namely by running: openstack coe ca sign secure-k8s-cluster client.csr > cert.pem I would like to do this without calling the openstack cli. Where does magnum store its ca key file? I could not find it on the control node under /etc/kubernetes/certs (the ca.crt is there, though no ca.key) Thank you! -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

[nova][ptg] Strawman proposal for vPTG timeslots
by Sylvain Bauza 07 Mar '23

07 Mar '23

Hi team, We recently discussed this in the Nova meeting [1] but I'd like to reemphasize here and now that I proposed to book 4 timeslots of one hour during 4 days for the next vPTG. https://ptg.opendev.org/ptg.html As you see, the proposed timeline will be : Tuesday, Wednesday, Thursday, Friday between 13:00UTC and 17:00UTC. May you have concerns with this proposal, please express them by replying to this thread. As a reminder, please add the topics you'd like to cover during the vPTG in the PTG etherpad : https://etherpad.opendev.org/p/nova-bobcat-ptg Thanks, -Sylvain [1] https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-07-16.00.log.h…

1 0