January 2024 - openstack-discuss - lists.openstack.org

sqlite vs NULL
by Pete Zaitcev 01 Feb '24

01 Feb '24

Greetings: In short: Should it be possible to select by key that is NULL in our DBs? This is needed for Cinder, review 657543[1]. Tests run on sqlite. Long explanation: The review tinkers with SQL Alchemy's query, so it may be doing it wrong. However, I looked at the queries and this seems to happen. A query for everything finds the NULL entry. The query looks like this: SELECT backups.created_at AS backups_created_at, (... very many AS ...) FROM backups LEFT OUTER JOIN backup_metadata AS backup_metadata_1 ON backup_metadata_1.backup_id = backups.id AND backup_metadata_1.deleted = false WHERE backups.deleted = false AND backups.host = %(host_1)s So, the entry is in the database. However, this one returns everything that matches the arguments, but not the one with key of NULL: SELECT backups.created_at AS backups_created_at, (... very many AS ...) FROM backups LEFT OUTER JOIN backup_metadata AS backup_metadata_1 ON backup_metadata_1.backup_id = backups.id AND backup_metadata_1.deleted = false WHERE backups.deleted = false AND backups.status IN (%(status_1)s, %(status_2)s, %(status_3)s, %(status_4)s, %(status_5)s, %(status_6)s, NULL) AND backups.host = %(host_1)s ORDER BY backups.created_at DESC, backups.id DESC The backup with a NULL status is in the DB, but I cannot get it with objects.BackupList.get_all(ctxt, filters={'status': (None,)}) One other thing: Is there an easy way to make negative query through SQLalchemy? My real task is to find all the backups that are not stable (complete and error), so that I can restart it. Thanks for any suggestions, -- Pete [1] https://review.opendev.org/c/openstack/cinder/+/657543

3 3

Cinder Bug Report 2024-01-31
by Jon Bernard 01 Feb '24

01 Feb '24

Hello Argonauts, Apologies for the hiatus. Between holidays, snow days, and sick days, January has been quite a month. Here are current bugs for this week, we will resume the bug meetings after the weekly upstream meeting next week. Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - Support for the "bring your own keys" approach for Cinder <https://bugs.launchpad.net/cinder/+bug/2051108> - Status: New - Documentation of Ceph auth caps for RBD clients used by Cinder / Glance / Nova is missing or inconsistent <https://bugs.launchpad.net/cinder/+bug/2051244> - Status: New - Dell PowerMax driver issues restapi version 92 for Unisphere 10.1 <https://bugs.launchpad.net/cinder/+bug/2051828> - Status: In Progress - Dell PowerMax: Add timeout when PowerMax does not return RESTAPI call <https://bugs.launchpad.net/cinder/+bug/2051830> - Status: In Progress - Fibre channel not scanning all targets <https://bugs.launchpad.net/os-brick/+bug/2051237> - Status: In Progress Thanks, -- Jon

1 0

What are the correct auth caps for Ceph RBD clients of Cinder / Glance / Nova
by Christian Rohmann 31 Jan '24

31 Jan '24

Hey openstack-discuss, I am a little confused about correct and required Ceph authx permissions for the RBD clients in Cinder, Glance and also Nova: When Glance is requested to delete an image it will check if this image has depended children, see https://opendev.org/openstack/glance_store/src/commit/6f5011d1f05c99894fb8b…. The children of Glance images usually are (Cinder) volumes, which therefore live in a different RBD pool "volumes". But if such children do exist a 500 error is thrown by Glance API. There also is an bug about this issue on Launchpad [3]. Manually using the RBD client shows the same error: > # rbd -n client.glance -k /etc/ceph/ceph.client.glance.keyring -p > images children $IMAGE_ID > > 2023-12-13T16:51:48.131+0000 7f198cf4e640 -1 > librbd::image::OpenRequest: failed to retrieve name: (1) Operation not > permitted > 2023-12-13T16:51:48.131+0000 7f198d74f640 -1 librbd::ImageState: > 0x5639fdd5af60 failed to open image: (1) Operation not permitted > rbd: listing children failed: (1) Operation not permitted > 2023-12-13T16:51:48.131+0000 7f1990c474c0 -1 librbd::api::Image: > list_descendants: failed to open descendant b7078ed7ace50d from pool > instances:(1) Operation not permitted So it's a permission error. Following either the documentation of Glance [1] or Ceph [2] on configuring the ceph auth caps there is no mention of granting anything towards the volume pool to Glance. So this is what I currently have configured: > client.cinder > key: REACTED > caps: [mgr] profile rbd pool=volumes, profile rbd-read-only > pool=images > caps: [mon] profile rbd > caps: [osd] profile rbd pool=volumes, profile rbd-read-only > pool=images > > client.glance > key: REACTED > caps: [mgr] profile rbd pool=images > caps: [mon] profile rbd > caps: [osd] profile rbd pool=images > > client.nova > key: REACTED > caps: [mgr] profile rbd pool=instances, profile rbd pool=images > caps: [mon] profile rbd > caps: [osd] profile rbd pool=instances, profile rbd pool=images > When granting the glance client e.g. "rbd-read-only" to the volumes pool via: > > # ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd > pool=images, profile rbd-read-only pool=volumes' mgr 'profile rbd > pool=images, profile rbd-read-only pool=volumes' > the error is gone. I am wondering through if this is really just a documentation bug (at OpenStack AND Ceph equally) and if Glance really needs read-only on the whole volumes pool or if there is some other capability that covers asking for child images. All in all I am simply wondering what the correct and least-privilege ceph auth caps for the RBD clients in Cinder, Glance and Nova would look like. Thanks Christian [1] https://docs.openstack.org/glance/latest/configuration/configuring.html#con… [2] https://docs.ceph.com/en/latest/rbd/rbd-openstack/#setup-ceph-client-authen… [3] https://bugs.launchpad.net/glance/+bug/2045158

6 15

[cinder] This week's meeting will be in video+IRC
by Rajat Dhasmana 31 Jan '24

31 Jan '24

Hello Argonauts, As we keep last meeting of the month in video + IRC mode, this week's meeting (today) will be held in video + IRC mode with details as follows: Date: 31st January, 2024 Time: 1400 UTC Meeting link: https://meet.google.com/mmb-fhtd-sjm IRC Channel: #openstack-meeting-alt Make sure you're connected to both the gmeet meeting and IRC. Thanks Rajat Dhasmana

1 0

[all][oslo][ironic][tc] Eventlet version bumps (topic:bump-eventlet in gerrit)
by Jay Faulkner 31 Jan '24

31 Jan '24

Hi all, https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged. If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained. I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so. From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic. Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL 0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.

2 6

[cinder] qos don t keep modifications
by garcetto 31 Jan '24

31 Jan '24

good evening, i was trying qos on cinder volumes, and it is working fine, BUT i found that if i modify the qos limits, or update or even remove them the actual vols that already have the previous iops limits do NOT thake new values...the newly created vols yes, the old ones not...is it a bug or is it normal? here i create a 100 iops limit, set on volumes type, THEN modify it, even reboot istance, BUT old value persisist... [image: image.png] thank you.

3 3

Openstack-Ansible External Networking OVN (Br-ex)
by jjjamesg 31 Jan '24

31 Jan '24

I can't for the life of me get external connectivity working, openstack itself works i just cant get external connectivity working, I can see ovs has created br-ex with bond1 attached. ovs-vsctl get open . external_ids:ovn-bridge-mapping shows: "physnst1:br-ex" I have tried creating the network with both: openstack network create --share --external --provider-physical-network physnet1 --provider-network-type vlan --provider-segment 233 public openstack network create --share --external --provider-physical-network physnet1 --provider-network-type flat publicnet-flat Only errors i can see is in the neutron container when i try and either create an instance on said network or attach a fip is is: (similar errors from a vlan type network this is just the error i pulled at the time) ERROR neutron.plugins.ml2.managers [req-de7f1cb8-ac8e-4dd6-ad2a-e14afdb152b5 req-c6a2aaff-ccf9-4ef3-9ef 7-1a8e4c7a1e41 9c27de6bdc4449abbbbcd5e5c4951bb9 2fced1ff77f5428eb2b63879bdd608cc - - default default] Failed to bind port 03cc1708-d4a2-4b11-bca2-40ebe0acff60 on host 02 for vnic_type normal using segments [{'id': '2ef64682-b5ff-472b-935d-a3c2b615d70f', 'network_type': 'flat', 'physical_network': 'physnet1', 'segmentation_id': None, 'network_id': '52b4f4e9-47f3-431b-8c45-ec8cc17f561d'}] WARNING neutron.scheduler.dhcp_agent_scheduler [None req-aafa81d9-479c-4f56-be6f-54d4ab3b42aa 1b29a 2297a224cdcaaba84e0eac30205 bafabcf2414d4adf98cab165c3f3de12 - - default default] No more DHCP agents WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [None req-aafa81d9-479c-4f56-be6f-54d4ab3b42aa 1b29a2297a224cdcaaba84e0eac30205 bafabcf2414d4adf98cab165c3f3de12 - - default default] Unable to schedule network 5fa049f5-0ed4-4287-9c63-dd753c853c35: no agents available; will retry on subsequent port and subnet creation events. I just cant figure out WHY this isn't working.. Below is my config: ############# ### /etc/network/interfaces: ############# auto bond1 iface bond1 inet manual bond-slaves eno2 eno4 bond-mode 802.3ad bond-miimon 100 bond-downdelay 200 bond-updelay 200 bond-lacp-rate 1 mtu 9000 auto bond1.30 iface bond1.30 inet manual vlan-raw-device bond1 ## (br-pubv is for my public vip) auto bond1.232 iface bond1.232 inet manual vlan-raw-device bond1 auto br-overlay iface br-overlay inet static bridge_stp off bridge_waitport 0 bridge_fd 0 bridge_ports bond1.30 address auto br-pubv iface br-pubv inet static address gateway bridge_stp off bridge_waitport 0 bridge_fd 0 bridge_ports bond1.232 ############# ### openstack_user_config.yml: ############# - network: container_bridge: "br-ex" network_interface: "bond1" type: "vlan" range: "232:332" net_name: "physnet1" group_binds: - neutron_ovn_controller ############# ### user_variables.yml ############# neutron_plugin_type: ml2.ovn neutron_plugin_base: - ovn-router - qos - neutron.services.ovn_l3.plugin.OVNL3RouterPlugin - neutron_dynamic_routing.services.bgp.bgp_plugin.BgpPlugin - vpnaas - metering - firewall_v2 neutron_ml2_drivers_type: "vlan,local,geneve,flat" neutron_provider_networks: network_types: "geneve" network_geneve_ranges: "1:1000" network_vlan_ranges: "physnet1" network_mappings: "physnet1:br-ex" network_interface_mappings: "br-ex:bond1" Bond1 is a trunk port (no native vlan), that has access to both overlay vlan and the 232:332 vlan range which both work when setting either network as a bridge on bond1 so connectivity is there. I have network-northd/gateway_hosts defined as well as: neutron_neutron_conf_overrides: ovn: enable_distributed_floating_ip: True openstack_host_specific_kernel_modules: - name: "openvswitch" pattern: "CONFIG_OPENVSWITCH" This should by all accounts just work, but for some reason for me it's not, what steps have i missed?

4 8

Incorrectly implemented service tokens in kolla-ansible
by Franciszek Przewoźny 31 Jan '24

31 Jan '24

Hi all, Can someone from kolla-ansible development have a look into that: https://bugs.launchpad.net/kolla-ansible/+bug/2049762 ? As for now, the provided configuration is totally different to what's described in manual. Thanks! Franciszek Przewoźny

2 2

[masakari][masakari-monitors] Feature to monitor hosts by kubernetes
by dogyun kim 31 Jan '24

31 Jan '24

Hi team masakari! I'm using openstack on kubernetes and also masakari for HA. I have developed a function monitoring by using kubernetes api internally to masakari-monitors for k8s-native openstack and want to contribute. blueprint link : https://blueprints.launchpad.net/masakari/+spec/host-monitors-by-kubernetes

5 5

[cinder] Inactive 3rd party CIs
by Takashi Kajinami 31 Jan '24

31 Jan '24

Hi, While I was trying to get a fresh restult from DellEMC VNX CI in [1], I noticed that some of the 3rd part CIs are inactive and haven't been posted any results for a long period. # See [2] and the list below. Note that the list may contain CIs for retired drivers. I'm wondering if someone from the cinder team can review the status and reach the vendors maintaining these CIs. If any of these are no longer working and there is no plan to recover these then I believe we have to mark the associated volume drivers unsupported and then remove these in a future release. Thank you, Takashi irc: tkajinam [1] https://review.opendev.org/c/openstack/cinder/+/834805 [2] http://cinderstats.ivehearditbothways.com/cireport.txt $ grep -B 2 -E 'last seen: 20(1|2[0123])' cireport.txt Checking name: Brocade Openstack CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Brocade_Openstack_CI first seen: 2019-03-07 02:43:59 (1781 days, 23:21:11 old) https://review.opendev.org/641539 last seen: 2019-06-17 20:36:40 (1679 days, 5:28:30 old) https://review.opendev.org/562053 -- Checking name: Cloudbase Cinder SMB3 CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Cloudbase_Cinder_SMB3_CI first seen: 2022-08-24 06:02:19 (515 days, 20:03:06 old) https://review.opendev.org/854268 last seen: 2022-11-02 13:32:21 (445 days, 12:33:04 old) https://review.opendev.org/863276 -- Checking name: DataCore CI - https://wiki.openstack.org/wiki/ThirdPartySystems/DataCore_CI first seen: 2018-10-11 20:55:57 (1928 days, 5:09:33 old) https://review.opendev.org/593191 last seen: 2020-02-13 14:16:20 (1438 days, 11:49:10 old) https://review.opendev.org/707643 -- Checking name: DellEMC PowerMAX CI - https://wiki.openstack.org/wiki/ThirdPartySystems/DellEMC_PowerMAX_CI first seen: 2022-06-17 12:14:23 (583 days, 13:51:32 old) https://review.opendev.org/821745 last seen: 2023-11-29 12:13:36 (53 days, 13:52:19 old) https://review.opendev.org/902106 -- Checking name: DellEMC SC CI - https://wiki.openstack.org/wiki/ThirdPartySystems/DellEMC_SC_CI first seen: 2022-05-24 00:06:06 (608 days, 2:00:17 old) https://review.opendev.org/839257 last seen: 2023-03-26 19:21:14 (301 days, 6:45:09 old) https://review.opendev.org/877583 -- Checking name: DellEMC VNX CI - https://wiki.openstack.org/wiki/ThirdPartySystems/DellEMC_VNX_CI first seen: 2021-09-11 03:27:04 (862 days, 22:39:38 old) https://review.opendev.org/760199 last seen: 2022-05-26 21:42:33 (605 days, 4:24:09 old) https://review.opendev.org/842131 -- Checking name: DellEMC XtremIO CI - https://wiki.openstack.org/wiki/ThirdPartySystems/DellEMC_XtremIO_CI first seen: 2021-11-17 06:06:53 (795 days, 19:59:58 old) https://review.opendev.org/818037 last seen: 2022-04-11 17:06:22 (650 days, 9:00:29 old) https://review.opendev.org/836211 -- Checking name: Fungible Storage CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Fungible_Storage_CI first seen: 2023-02-21 10:45:34 (334 days, 15:21:31 old) https://review.opendev.org/843493 last seen: 2023-05-25 11:17:36 (241 days, 14:49:29 old) https://review.opendev.org/880275 -- Checking name: Hedvig CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Hedvig_CI first seen: 2020-11-04 02:16:19 (1173 days, 23:50:53 old) https://review.opendev.org/761160 last seen: 2020-11-20 13:10:00 (1157 days, 12:57:12 old) https://review.opendev.org/762884 -- Checking name: HPE Nimble Storage CI - https://wiki.openstack.org/wiki/ThirdPartySystems/HPE_Nimble_Storage_CI first seen: 2023-04-20 19:24:38 (276 days, 6:42:42 old) https://review.opendev.org/877755 last seen: 2023-07-31 10:12:19 (174 days, 15:55:01 old) https://review.opendev.org/886456 -- Checking name: HPE Storage CI - https://wiki.openstack.org/wiki/ThirdPartySystems/HPE_Storage_CI first seen: 2016-07-21 05:02:38 (2740 days, 21:04:54 old) https://review.opendev.org/344766 last seen: 2016-08-09 22:53:51 (2721 days, 3:13:41 old) https://review.opendev.org/318737 -- Checking name: HPMSA CI - https://wiki.openstack.org/wiki/ThirdPartySystems/HPMSA_CI first seen: 2023-06-07 03:02:16 (228 days, 23:05:25 old) https://review.opendev.org/850002 last seen: 2023-12-05 15:15:09 (47 days, 10:52:32 old) https://review.opendev.org/900513 -- Checking name: Huawei FusionStorage CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Huawei_FusionStorage_CI first seen: 2019-02-21 15:07:45 (1795 days, 11:00:03 old) https://review.opendev.org/635203 last seen: 2020-07-23 09:23:28 (1277 days, 16:44:20 old) https://review.opendev.org/742136 -- Checking name: Huawei Volume CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Huawei_Volume_CI first seen: 2018-02-09 08:03:20 (2172 days, 18:04:47 old) https://review.opendev.org/542700 last seen: 2018-06-12 10:12:22 (2049 days, 15:55:45 old) https://review.opendev.org/574508 -- Checking name: IBM GPFS CI - https://wiki.openstack.org/wiki/ThirdPartySystems/IBM_GPFS_CI first seen: 2017-08-30 23:39:19 (2335 days, 2:28:56 old) https://review.opendev.org/499271 last seen: 2022-01-25 14:29:37 (726 days, 11:38:38 old) https://review.opendev.org/819011 -- Checking name: IBM Storage CI - https://wiki.openstack.org/wiki/ThirdPartySystems/IBM_Storage_CI first seen: 2021-10-25 15:30:42 (818 days, 10:37:42 old) https://review.opendev.org/809518 last seen: 2023-07-11 09:37:45 (194 days, 16:30:39 old) https://review.opendev.org/880518 -- Checking name: Infortrend CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Infortrend_CI first seen: 2020-01-10 07:03:58 (1472 days, 19:04:35 old) https://review.opendev.org/694379 last seen: 2020-05-01 16:25:58 (1360 days, 9:42:35 old) https://review.opendev.org/722806 -- Checking name: INSPUR CI - https://wiki.openstack.org/wiki/ThirdPartySystems/INSPUR_CI first seen: 2019-08-13 20:31:35 (1622 days, 5:37:05 old) https://review.opendev.org/676274 last seen: 2020-06-09 02:34:51 (1321 days, 23:33:49 old) https://review.opendev.org/734395 -- Checking name: Intel RSD CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Intel_RSD_CI first seen: 2020-08-22 20:40:00 (1247 days, 5:28:49 old) https://review.opendev.org/747540 last seen: 2020-09-30 20:50:30 (1208 days, 5:18:19 old) https://review.opendev.org/746267 -- Checking name: Kaminario K2 CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Kaminario_K2_CI first seen: 2019-09-03 21:12:51 (1601 days, 4:56:06 old) https://review.opendev.org/679785 last seen: 2020-05-02 20:21:51 (1359 days, 5:47:06 old) https://review.opendev.org/724548 -- Checking name: openstack lightbits - https://wiki.openstack.org/wiki/ThirdPartySystems/openstack_lightbits first seen: 2023-01-18 14:02:54 (368 days, 12:06:19 old) https://review.opendev.org/865429 last seen: 2023-10-30 15:55:42 (83 days, 10:13:31 old) https://review.opendev.org/792209 -- Checking name: LINBIT LINSTOR CI - https://wiki.openstack.org/wiki/ThirdPartySystems/LINBIT_LINSTOR_CI first seen: 2019-12-23 08:12:07 (1490 days, 17:57:14 old) https://review.opendev.org/700296 last seen: 2020-04-04 21:33:05 (1387 days, 4:36:16 old) https://review.opendev.org/717383 -- Checking name: MacroSAN Volume CI - https://wiki.openstack.org/wiki/ThirdPartySystems/MacroSAN_Volume_CI first seen: 2019-06-21 02:47:54 (1675 days, 23:21:27 old) https://review.opendev.org/666477 last seen: 2019-08-21 12:59:52 (1614 days, 13:09:29 old) https://review.opendev.org/674156 -- Checking name: Mellanox CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Mellanox_CI first seen: 2021-12-02 22:17:29 (780 days, 3:51:59 old) https://review.opendev.org/733622 last seen: 2022-01-27 23:20:17 (724 days, 2:49:11 old) https://review.opendev.org/818830 -- Checking name: NetApp SolidFire CI - https://wiki.openstack.org/wiki/ThirdPartySystems/NetApp_SolidFire_CI first seen: 2022-01-07 03:00:03 (744 days, 23:09:43 old) https://review.opendev.org/796114 last seen: 2022-12-27 10:41:51 (390 days, 15:27:54 old) https://review.opendev.org/853126 -- Checking name: Nexenta CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Nexenta_CI first seen: 2023-05-18 19:46:37 (248 days, 6:23:16 old) https://review.opendev.org/881470 last seen: 2023-12-18 12:42:24 (34 days, 13:27:29 old) https://review.opendev.org/903883 -- Checking name: ProphetStor CI - https://wiki.openstack.org/wiki/ThirdPartySystems/ProphetStor_CI first seen: 2018-04-05 22:54:14 (2117 days, 3:15:54 old) https://review.opendev.org/559172 last seen: 2018-11-27 18:08:32 (1881 days, 8:01:36 old) https://review.opendev.org/603283 -- Checking name: qnap-ci - https://wiki.openstack.org/wiki/ThirdPartySystems/qnap-ci first seen: 2016-12-01 08:58:50 (2607 days, 17:11:33 old) https://review.opendev.org/313140 last seen: 2019-03-27 21:38:37 (1761 days, 4:31:46 old) https://review.opendev.org/648174 -- Checking name: Quobyte CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Quobyte_CI first seen: 2020-08-05 14:17:31 (1264 days, 11:53:01 old) https://review.opendev.org/742491 last seen: 2020-10-26 22:12:30 (1182 days, 3:58:02 old) https://review.opendev.org/754728 -- Checking name: SandStone Storage CI - https://wiki.openstack.org/wiki/ThirdPartySystems/SandStone_Storage_CI first seen: 2019-09-04 11:39:43 (1600 days, 14:30:51 old) https://review.opendev.org/679970 last seen: 2020-04-19 05:55:57 (1372 days, 20:14:37 old) https://review.opendev.org/720678 -- Checking name: Seagate CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Seagate_CI first seen: 2023-01-27 08:03:49 (359 days, 18:06:55 old) https://review.opendev.org/851640 last seen: 2023-05-25 22:32:18 (241 days, 3:38:26 old) https://review.opendev.org/884253 -- Checking name: Synology DSM CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Synology_DSM_CI first seen: 2020-09-04 18:45:57 (1234 days, 7:25:05 old) https://review.opendev.org/749996 last seen: 2020-11-20 08:17:25 (1157 days, 17:53:37 old) https://review.opendev.org/757082 -- Checking name: TOYOU TYDS CI - https://wiki.openstack.org/wiki/ThirdPartySystems/TOYOU_TYDS_CI first seen: 2021-11-20 16:22:53 (792 days, 9:48:15 old) https://review.opendev.org/818098 last seen: 2023-10-18 01:25:34 (96 days, 0:45:34 old) https://review.opendev.org/898600 -- Checking name: Veritas Access CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Veritas_Access_CI first seen: 2018-12-11 02:38:26 (1867 days, 23:32:48 old) https://review.opendev.org/624193 last seen: 2019-03-15 10:56:07 (1773 days, 15:15:07 old) https://review.opendev.org/643555 -- Checking name: Virtuozzo Storage CI - https://wiki.openstack.org/wiki/ThirdPartySystems/Virtuozzo_Storage_CI first seen: 2019-08-09 21:32:36 (1626 days, 4:38:47 old) https://review.opendev.org/664860 last seen: 2019-09-25 02:45:11 (1579 days, 23:26:12 old) https://review.opendev.org/639180

4 4