June 2023 - openstack-discuss - lists.openstack.org

[kolla] Weekly meetings in July 2023
by Michał Nasiadka 26 Jun '23

26 Jun '23

Hello Koalas, We’ve decided last week that due to my absence from 1st July - we’re cancelling weekly meetings for the first three weeks of July - i.e. we will meet 26th of July for the only July meeting. NOTE: This Wednesday meeting is not cancelled In case of queries of additional topics - please reach out on #openstack-kolla Best regards, Michal

1 0

[kolla-ansible][yoga] How can I modify the logrotate configuration for kolla to save some space
by wodel youchi 26 Jun '23

26 Jun '23

Hi, Is there a way to tweak kolla's logrotate configuration, to save up some space on /var/log/kolla? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

2 2

Kolla iser (RDMA) support
by Tomas Leypold 26 Jun '23

26 Jun '23

Hi, I have a kolla-ansible deployment, and I am trying to switch from the LVM iSCSI protocol to iSER (RDMA). On the host machine, I have installed rdma-core, which enabled kernel module loading for the NIC card we are using, the "BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller". I have verified the function of RDMA with ib_write_bw, which worked fine. The first problem I encountered with Kolla is the absence of necessary packages - ibverbs-providers, which includes userspace drivers for RDMA. Without it, cinder_volume will not attach a volume. I did a image rebuild with package ibverbs-providers included. I will fill an issue for this later. Then I just changed target_protocol = iser in cinder_volume config file. The current issue I am facing is that the instance is stuck at the "Spawning" phase. In the nova_compute log, I see the following lines: 2023-06-26 10:57:36.204 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f4dd45adc30>, 'connection_properties': {'target_discovered': False, 'target_portal': '10.47.1.25:3260', 'target_iqn': 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26', 'target_lun': 1, 'volume_id': 'bc382b4c-dac4-4ca6-a260-0d450d6c4e26', 'auth_method': 'CHAP', 'auth_username': '8n7WyZ2wZFNd7B33rfYw', 'auth_password': '***', 'encrypted': False, 'qos_specs': None, 'access_mode': 'rw': False}}" trace_logging_wrapper /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py:174 2023-06-26 10:57:36.208 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Acquiring lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:68 2023-06-26 10:57:36.208 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:73 2023-06-26 10:57:36.209 7 INFO os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Trying to connect to iSCSI portal 10.47.1.25:3260 2023-06-26 10:57:36.219 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm (): stdout= stderr=iscsiadm: No records found _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.227 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--interface', 'iser', '--op', 'new'): stdout=New iSCSI node [iser:[hw=,ip=,net_if=,iscsi_if=iser] 10.47.1.25,3260,-1 iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26] added stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.234 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.241 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.247 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', '8n7WyZ2wZFNd7B33rfYw'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.254 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:57:36.260 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. _run_iscsiadm_bare /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1182 2023-06-26 10:57:36.261 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. _run_iscsi_session /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1171 2023-06-26 10:57:36.261 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. 2 min later it timeouts: 2023-06-26 10:59:38.377 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Failed to login iSCSI target iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26 on portal 10.47.1.25:3260 (exit code 8).: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2023-06-26 10:59:38.378 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" "released" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: held 122.170s inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:87 2023-06-26 10:59:38.378 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Failed to connect to iSCSI portal 10.47.1.25:3260. 2023-06-26 10:59:38.378 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Getting connected devices for (ips,iqns,luns)=[('10.47.1.25:3260', 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26', 1)] _get_connection_devices /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:819 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. _run_iscsiadm_bare /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1182 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. _run_iscsi_session /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1171 2023-06-26 10:59:38.394 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Resulting device map defaultdict(<function ISCSIConnector._get_connection_devices.<locals>.<lambda> at 0x7f4dd43230a0>, {('10.47.1.25:3260', 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26'): (set(), set())}) _get_connection_devices /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:852 2023-06-26 10:59:38.394 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Disconnecting from: [('10.47.1.25:3260', 'iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26')] _disconnect_connection /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1160 2023-06-26 10:59:38.401 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:38.407 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--logout',): stdout= stderr=iscsiadm: No matching sessions found _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:38.413 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'delete'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:38.414 7 DEBUG os_brick.utils [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Finished call to 'os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_single_volume' after 122.207(s), this was the 1st time calling it. log_it /var/lib/kolla/venv/lib/python3.10/site-packages/tenacity/after.py:30 2023-06-26 10:59:38.415 7 DEBUG os_brick.utils [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Retrying os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_single_volume in 1.0 seconds as it raised VolumeDeviceNotFound: Volume device not found at .. log_it /var/lib/kolla/venv/lib/python3.10/site-packages/tenacity/before_sleep.py:40 2023-06-26 10:59:39.393 7 DEBUG oslo_service.periodic_task [None req-a325bda7-8910-43e3-90a9-6ae0dc8ae6a7 - - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py:210 2023-06-26 10:59:39.416 7 WARNING os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Service needs to call os_brick.setup() before connecting volumes, if it doesn't it will break on the next release 2023-06-26 10:59:39.416 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Acquiring lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:68 2023-06-26 10:59:39.417 7 DEBUG os_brick.initiator.connectors.base [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Lock "connect_to_iscsi_portal-10.47.1.25:3260-iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26" acquired by "os_brick.initiator.connectors.iscsi.ISCSIConnector._connect_to_iscsi_portal_unsafe" :: waited 0.001s inner /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py:73 2023-06-26 10:59:39.418 7 INFO os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] Trying to connect to iSCSI portal 10.47.1.25:3260 2023-06-26 10:59:39.426 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm (): stdout= stderr=iscsiadm: No records found _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.435 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--interface', 'iser', '--op', 'new'): stdout=New iSCSI node [iser:[hw=,ip=,net_if=,iscsi_if=iser] 10.47.1.25,3260,-1 iqn.2010-10.org.openstack:volume-bc382b4c-dac4-4ca6-a260-0d450d6c4e26] added stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.441 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.scan', '-v', 'manual'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.447 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.authmethod', '-v', 'CHAP'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.454 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.username', '-v', '8n7WyZ2wZFNd7B33rfYw'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.454 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 20 __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:263 2023-06-26 10:59:39.461 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('--op', 'update', '-n', 'node.session.auth.password', '-v', '***'): stdout= stderr= _run_iscsiadm /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1026 2023-06-26 10:59:39.467 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions. _run_iscsiadm_bare /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1182 2023-06-26 10:59:39.467 7 DEBUG os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions. _run_iscsi_session /var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py:1171 2023-06-26 10:59:39.468 7 WARNING os_brick.initiator.connectors.iscsi [None req-67fc8494-95ce-45ca-972b-63d0833da0b2 c10f2f77bb9d43a0832dcc9262d63943 49fccac069b847a98fe3aae84ecf9aa4 - - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions. In this log, the problem is the line "Failed to login to iSCSI target", but I am not sure why this is happening. Does anyone have any ideas about what could be wrong? Thanks! --- Best regards, Tomas Leypold

1 0

Neutron BGP agent advertisement and l3/openvswitch-agent problems (zed)
by Bryan Huang 25 Jun '23

25 Jun '23

Dear folks, Recently, we met some neutron networking problems in our envrionment, openstack version is zed, and kolla-ansible as the deployment tool. 1. Neutron BGP agent doesn't advertise the floating IPs to the BGP peer, in case of the floating IPs were served for port forwarding, but the floating IPs attached to VM/Container were advertised correctly. so the question is this scenario supported by BGP agent, if not when will it be supported, is it in the plan? 2. iptable rules restoring error in l3-agent and openvswitch-agent (A bug was reported in launchpad: https://bugs.launchpad.net/neutron/+bug/2024976) Bug #2024976 “iptable rules restoring error in l3-agent and open...” : Bugs : neutron<https://bugs.launchpad.net/neutron/+bug/2024976> Openstack version: zed/stable OS version: Ubuntu 22.04.2 LTS Kernel version: 5.15.0-75-generic #82-Ubuntu Deployment: kolla-ansible iptable rules restoring error in l3-agent and openvswitch-agent: openvswitch-agnet log: 2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: [... bugs.launchpad.net openvswitch-agnet log: 2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager *filter :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :neutron-filter-top - [0:0] :neutron-openvswi-FORWARD - [0:0] :neutron-openvswi-INPUT - [0:0] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-local - [0:0] :neutron-openvswi-sg-chain - [0:0] :neutron-openvswi-sg-fallback - [0:0] -I FORWARD 1 -j neutron-filter-top -I FORWARD 2 -j neutron-openvswi-FORWARD -I INPUT 1 -j neutron-openvswi-INPUT -I OUTPUT 1 -j neutron-filter-top -I OUTPUT 2 -j neutron-openvswi-OUTPUT -I neutron-filter-top 1 -j neutron-openvswi-local -I neutron-openvswi-FORWARD 1 -m physdev --physdev-out tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 2 -m physdev --physdev-in tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 3 -m physdev --physdev-out tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-FORWARD 4 -m physdev --physdev-in tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -I neutron-openvswi-sg-chain 1 -j ACCEPT -I neutron-openvswi-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP COMMIT # Completed by iptables_manager # Generated by iptables_manager *raw :OUTPUT - [0:0] :PREROUTING - [0:0] :neutron-openvswi-OUTPUT - [0:0] :neutron-openvswi-PREROUTING - [0:0] -I OUTPUT 1 -j neutron-openvswi-OUTPUT -I PREROUTING 1 -j neutron-openvswi-PREROUTING COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `physdev':No such file or directory Error occurred at line: 19 Try `iptables-restore -h' or 'iptables-restore --help' for more information. l3-agent log: 2023-06-23 16:15:49.545 33 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['ip', 'netns', 'exec', 'qrouter-0f0e60d0-bf51-4361-901b-4b998201b44b', 'iptables-restore', '-n']; Stdin: # Generated by iptables_manager *filter :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] :neutron-l3-agent-scope - [0:0] -I FORWARD 1 -j neutron-filter-top -I FORWARD 2 -j neutron-l3-agent-FORWARD -I INPUT 1 -j neutron-l3-agent-INPUT -I OUTPUT 1 -j neutron-filter-top -I OUTPUT 2 -j neutron-l3-agent-OUTPUT -I neutron-filter-top 1 -j neutron-l3-agent-local -I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope -I neutron-l3-agent-scope 1 -m mark --mark 0x1/0xffff -j DROP COMMIT # Completed by iptables_manager # Generated by iptables_manager *mangle :FORWARD - [0:0] :INPUT - [0:0] :OUTPUT - [0:0] :POSTROUTING - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-POSTROUTING - [0:0] :neutron-l3-agent-PREROUTING - [0:0] :neutron-l3-agent-float-snat - [0:0] :neutron-l3-agent-floatingip - [0:0] :neutron-l3-agent-mark - [0:0] :neutron-l3-agent-scope - [0:0] -I FORWARD 1 -j neutron-l3-agent-FORWARD -I INPUT 1 -j neutron-l3-agent-INPUT -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING -I PREROUTING 1 -j neutron-l3-agent-PREROUTING -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-mark -I neutron-l3-agent-PREROUTING 2 -j neutron-l3-agent-scope -I neutron-l3-agent-PREROUTING 3 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000 -I neutron-l3-agent-PREROUTING 4 -j neutron-l3-agent-floatingip -I neutron-l3-agent-PREROUTING 5 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff -I neutron-l3-agent-float-snat 1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000 COMMIT # Completed by iptables_manager # Generated by iptables_manager *nat :OUTPUT - [0:0] :POSTROUTING - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-POSTROUTING - [0:0] :neutron-l3-agent-PREROUTING - [0:0] :neutron-l3-agent-float-snat - [0:0] :neutron-l3-agent-snat - [0:0] :neutron-postrouting-bottom - [0:0] -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING -I POSTROUTING 2 -j neutron-postrouting-bottom -I PREROUTING 1 -j neutron-l3-agent-PREROUTING -I neutron-l3-agent-POSTROUTING 1 ! -o rfp-0f0e60d0-b -m conntrack ! --ctstate DNAT -j ACCEPT -I neutron-l3-agent-PREROUTING 1 -d 137.175.31.207/32 -i rfp-0f0e60d0-b -j DNAT --to-destination 10.10.0.246 -I neutron-l3-agent-float-snat 1 -s 10.10.0.246/32 -j SNAT --to-source 137.175.31.207 --random-fully -I neutron-l3-agent-snat 1 -j neutron-l3-agent-float-snat -I neutron-postrouting-bottom 1 -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat COMMIT # Completed by iptables_manager # Generated by iptables_manager *raw :OUTPUT - [0:0] :PREROUTING - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-PREROUTING - [0:0] -I OUTPUT 1 -j neutron-l3-agent-OUTPUT -I PREROUTING 1 -j neutron-l3-agent-PREROUTING COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `mark':No such file or directory Error occurred at line: 19 And we check the system the x_tables kernel module were loaded: # lsmod | grep x_tables x_tables 53248 12 xt_conntrack,nft_compat,xt_tcpudp,xt_physdev,xt_nat,xt_comment,ip6_tables,xt_connmark,xt_CT,ip_tables,xt_REDIRECT,xt_mark (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*mark.so" ./lib/x86_64-linux-gnu/xtables/libxt_connmark.so ./lib/x86_64-linux-gnu/xtables/libxt_mark.so ./lib/x86_64-linux-gnu/xtables/libebt_mark.so (neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*physdev.so" ./lib/x86_64-linux-gnu/xtables/libxt_physdev.so Does someone have ever met the problems what is the solution the resovle them. Thanks in advance Sincerely， Bryan

1 0

the nova errors (vresion queens) errors
by 无数的星球 25 Jun '23

25 Jun '23

I reported an error on the compute node after building nova。 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager [req-a85e1436-e5a4-4395-ba5c-899d3b21b7bc - - - - -] Error updating resources for node compute.: KeyError: 'allocations' 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager Traceback (most recent call last): 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7578, in update_available_resource_for_node 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager rt.update_available_resource(context, nodename) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 720, in update_available_resource 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager self._update_available_resource(context, resources) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager return f(*args, **kwargs) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 771, in _update_available_resource 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager context, self.compute_nodes[nodename], migrations) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1277, in _remove_deleted_instances_allocations 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager context, cn.uuid) or {} 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 64, in wrapper 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager return f(self, *a, **k) 2023-06-23 10:04:25.334 15207 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1785, in get_allocations_for_resource_provider After this error occurs, after building the neutron and creating the virtual machine, it will be unable to create the virtual machine, which will be reported in the controller NoValidHost: No valid host was found. 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager Traceback (most recent call last): 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1165, in schedule_and_build_instances 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager instance_uuids, return_alternates=True) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 760, in _schedule_instances 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return_alternates=return_alternates) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 793, in wrapped 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return func(*args, **kwargs) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/init.py", line 53, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/init.py", line 37, in __run_method 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager retry=self.retry) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager timeout=timeout, retry=retry) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 625, in send 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager retry=retry) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 616, in _send 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager raise result 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager Traceback (most recent call last): 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager return func(*args, **kwargs) 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 139, in select_destinations 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager raise exception.NoValidHost(reason="") 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager NoValidHost: No valid host was found. 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager 2023-06-22 10:57:15.649 3732 ERROR nova.conductor.manager Note: As the logs were obtained at different times, please ignore the timing issue. The following will list my nova configuration files. Please review them. Thank you system is centos 7 2009 When a computer error is found, Error updating resources for node compute.: KeyError: 'allocations' the controller will report it 2023-06-24 10:45:51.133 96992 INFO nova.osapi_compute.wsgi.server [-] 192.168.80.20 "GET /resource_providers/6304d158-7a38-4a61-b6f6-f16c22e27b53/allocations HTTP/1.1" status: 300 len: 720 time: 0.0006239 the placement is Fix bugs Listen 8778 <VirtualHost *:8778>   WSGIProcessGroup nova-placement-api   WSGIApplicationGroup %{GLOBAL}   WSGIPassAuthorization On   WSGIDaemonProcess nova-placement-api processes=3 threads=1 user=nova group=nova   WSGIScriptAlias / /usr/bin/nova-placement-api   <IfVersion >= 2.4>     ErrorLogFormat "%M"   </IfVersion>   ErrorLog /var/log/nova/nova-placement-api.log   #SSLEngine On   #SSLCertificateFile ...   #SSLCertificateKeyFile ... </VirtualHost> Alias /nova-placement-api /usr/bin/nova-placement-api <Location /nova-placement-api>   SetHandler wsgi-script   Options +ExecCGI   WSGIProcessGroup nova-placement-api   WSGIApplicationGroup %{GLOBAL}   WSGIPassAuthorization On </Location> <Directory /usr/bin>   <IfVersion >= 2.4>         Require all granted   </IfVersion>   <IfVersion < 2.4>         Order allow,deny         Allow from all   </IfVersion> </Directory>

1 0

[kolla-ansible][zun] Deleting container through api
by James Leong 24 Jun '23

24 Jun '23

Hi all, I am currently playing around with the Zun component. I have deployed the yoga version of OpenStack using kolla-ansible. Is there an api to delete a container based on the ID without any user context? Best, Jame

1 0

Next minimum libvirt / QEMU versions for "C" release (2024)
by Kashyap Chamarthy 24 Jun '23

24 Jun '23

Hi, folks! The last time we incremented versions for libvirt and QEMU was for "Wallaby" release[1]. It'Although we advertized NEXT_MIN_{LIBVIRT,QEMU} versions to be libvirt 7, 0, 0 and QEMU 5.2.0, we actually didn't bump. So, for "Bobcat", we'll bump the MIN libvirt and QEMU versions to: MIN_LIBVIRT_VERSION = (7, 0, 0) MIN_QEMU_VERSION = (5, 2, 0) For the upcoming "C" release (2024), we annouce the NEXT_MIN_{LIBVIRT,QEMU} to be the following: NEXT_MIN_LIBVIRT_VERSION = (8, 0, 0) NEXT_MIN_QEMU_VERSION = (6, 2, 0) Hope that sounds fine. If anyone has concerns or further comments, please raise them on this thread or add 'em in the patch below: https://review.opendev.org/c/openstack/nova/+/886825 Pick next min libvirt / QEMU versions for "C" (2024) release Rationale --------- We picked the above NEXT_MIN version based on Ubuntu "Jammy" (22.04). It is the lowest common denominotor among Debian 11 ("Bullseye"), Ubuntu 22.04 ("Jammy"), and CentOS 9 Stream. - Ubuntu 22.04 (Jammy): - libvirt-daemon: 8.0.0-1ubuntu7.5 - qemu-system-x86: 6.2+dfsg-2ubuntu6.11 - Debian 11 (Bullseye): - libvirt: 7.0.0-3+deb11u2 - qemu: 5.2+dfsg-11+deb11u2 - CentOS Stream 9: - libvirt-daemon-kvm-9.3.0-2.el9.x86_64.rpm - qemu-kvm-8.0.0-4.el9.x86_64.rpm [1] https://opendev.org/openstack/nova/commit/95724bbaef6 -- libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION, 2020-09-28) -- /kashyap

4 4

[kolla] does cinder volumes works on ceph rbd pools erasure coded?
by garcetto 23 Jun '23

23 Jun '23

good evening, trying to make it working, (ubuntu 22, kolla all-in-one, external ceph wirh both rdb replica AND erasure code pools). with replica cinder-volumes works. no way with ec pools? cephx and access already check all ok... any idea? is it supported? thank you

1 0

[kolla] User Forum Vancouver
by Michał Nasiadka 23 Jun '23

23 Jun '23

Hello Koalas, Thank you for attending User Forum on Vancouver 2023 Summit. We’ve had a fruitful conversation around Kolla-Ansible performance and additional actions that can be taken to improve day 2 operations and stability of production environments. Link to etherpad: https://etherpad.opendev.org/p/YVR23-kolla-user-forum See you next time - and I once again encourage everybody to attend Kolla Weekly Meetings [1] or write messages to the Mailing List. [1]: https://meetings.opendev.org/#Kolla_Team_Meeting Michal

1 0

[Glance] Weekly Meeting Cancelled
by Pranali Deore 23 Jun '23

23 Jun '23

Hello, Glance weekly meeting for next week, Thursday 29th June, has been cancelled and instead we are going to have review party on 28th June. Kindly please refer the meeting etherpad[1] for more details, [1]: https://etherpad.opendev.org/p/glance-team-meeting-agenda Thanks and regards, ~Pranali

1 0