July 2024 - openstack-discuss - lists.openstack.org

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[horizon][ceph] object storage container storage policy missing
by Satish Patel 25 Nov '24

25 Nov '24

Folks, Openstack Release: Zed Recently I have integrated ceph RGW service with openstack/keystone and testing with command line tool "openstack container create foo1" and it works but when I go to Horizon UI I am seeing it doesn't have storage policy dropdown list and because of that I am not able to create container or upload file from Horizon GUI https://ibb.co/KhYW6Rd It works with commands like but not horizon. $ openstack container create foo1 +---------------------------------------+-----------+------------------------------------------------+ | account | container | x-trans-id | +---------------------------------------+-----------+------------------------------------------------+ | AUTH_d4d82849885e4126aef402c408ad2c5c | foo1 | tx00000c057454841b13497-0065b558b4-42ff2eb-ahm | +---------------------------------------+-----------+------------------------------------------------+ $ openstack container list +------+ | Name | +------+ | foo1 | +------+

2 2

[ironic] Having issue with deploying ISO image to node
by James Leong 18 Nov '24

18 Nov '24

Hi all, I am trying to play around with ironic, and I am having an issue with deploying an Ubuntu 20.04 ISO image onto my bare metal node. Since I am quite new to bare metal, I am not entirely sure what went wrong as I couldn't find any error messages shown on my log file (ironic-conductor, ironic-api, ironic_neutron_agent, and nova-compute-ironic). This might be my guess; the ironic inspection is hitting timeout, which is causing the deployment to timeout. However, I have no idea why the inspection is failing at timeout. Any help would be appreciated. Best, James

6 7

[largescale-sig]scaling story
by Alex Song (宋文平) 25 Oct '24

25 Oct '24

Hi everyone, we want to share our scaling story. Please refer the link: https://etherpad.opendev.org/p/large-scale-inspur Thanks, Alex Song

6 8

[nova][libvirtd] internal error: unable to execute QEMU command 'query-named-block-nodes'
by Satish Patel 22 Oct '24

22 Oct '24

Folks, I have a ceph cluster and recently I configured rbd-mirror to replicate all data to remove ceph cluster for disaster recovery. Yesterday for POC we did a hard cutover on ceph and point openstack to the new cluster. All other vms came back up fine but 2 vms stuck in this error state in libvirt logs 2024-01-31 22:44:37.591+0000: 474597: error : qemuMonitorJSONCheckErrorFull:399 : internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied If it's a wider issue than it should impact all the vms but why only two VM stuck and not starting and libvirt giving me this error in logs nova-compute logs also showing same error 2024-01-31 22:44:40.925 7 INFO nova.compute.manager [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] [instance: 2de0f880-77c7-4d2c-9e01-898c57ad3693] Successfully reverted task state from powering-on on failure for instance. 2024-01-31 22:44:40.944 7 ERROR oslo_messaging.rpc.server [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied

3 8

[kuryr][tc] Final call for Kuryr PTL/DPL liaisons
by Ghanshyam Mann 15 Oct '24

15 Oct '24

Hello Everyone, Kuryr project ends up being a leaderless project for the 2024.2 Dalmatian cycle[1]. There is no volunteer to serve as PTL or DPL liaisons yet. As the next step, I have proposed to mark this project Inactive: - https://review.opendev.org/c/openstack/governance/+/916945 If you would like to serve as a leader and help maintain this project, this is the right time to step up by replying to this email or commenting in gerrit change 916945. [1] https://etherpad.opendev.org/p/2024.2-leaderless#L99 Ref: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… -gmann

4 10

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

[all][tc][ptls] Eventlet: Python 3.13 - Attach your seatbelt
by Herve Beraud 06 Sep '24

06 Sep '24

Hi all, As Python 3.13 is close to become a reality, new concerning incompatibilities between this new CPython version and Eventlet also become a reality: - https://github.com/eventlet/eventlet/issues/964 Fedora Python maintainers proposed a patch, but for now the problem remains unsolved. Also, many people reported bugs related to incompatibility between python3.12/dnspython/eventlet/pip-system-cert. There is an example who summarize well the problem https://github.com/eventlet/eventlet/issues/957 Do you have some visibility concerning the adoption of Python 3.13 into Openstack? Turbulences ahead... -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

4 5

[Kolla] [LetsEncrypt] Internal/External/Backend Certificates
by jjjamesg＠proton.me 05 Sep '24

05 Sep '24

Hi all, another one here that can't seem to get kolla running with letsencrypt certs, just curious as to what settings are required for kolla to pull internal/external/backend certs from letsencrypt? I've looked at the docs and have tried a few settings but keep getting errors, sure i had this working last week before i tore it down ready to deploy for production and now i just cant get it to complete the deployment because of certs, only thing i can think of is i need to run kolla-ansible -i multinode certificates but i thought that was just for self made certs? Also, what are best practices for backend tls certs, self signed or using the internal cert etc?? Is it possible to use letsencrypt at all for backend? Cheers all

3 6

[ops][kolla][kolla-ansible] Lets Encrypt Deployment on 2024.1/v18
by Forrest Fuqua 03 Sep '24

03 Sep '24

The documentation seems a little incomplete when it comes to deploying a stack with let's encrypt The attached config is after trying a few settings, but the lets encrypt container errors out every time during deployment causing haproxy to not get a proper ssl certificate the services.d entries are showing its ignoring kolla_internal_fqdn_cert and still using kolla_internal.pem in most services causing the following deployment error TASK [service-ks-register : keystone | Creating services] **************************************************************************************************************************************************** FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (5 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (4 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (3 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (2 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (1 retries left). failed: [100.70.0.1] (item=keystone (identity)) => {"action": "os_keystone_service", "ansible_loop_var": "item", "attempts": 5, "changed": false, "item": {"description": "Openstack Identity Service", "endpoints": [{"interface": "internal", "url": "https://openstack.cyberrange.rit.edu:5000"}, {"interface": "public", "url": "https://openstack.cyberrange.rit.edu:5000"}], "name": "keystone", "type": "identity"}, "module_stderr": "Failed to discover available identity versions when contacting https://openstack.cyberrange.rit.edu:5000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n httplib_response = self._make_request(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 404, in _make_request\n self._validate_conn(conn)\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 1058, in _validate_conn\n conn.connect()\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connection.py\", line 419, in connect\n self.sock = ssl_wrap_socket(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/ssl_.py\", line 449, in ssl_wrap_socket\n ssl_sock = _ssl_wrap_socket_impl(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/ssl_.py\", line 493, in _ssl_wrap_socket_impl\n return ssl_context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib/python3.10/ssl.py\", line 513, in wrap_socket\n return self.sslsocket_class._create(\n File \"/usr/lib/python3.10/ssl.py\", line 1100, in _create\n self.do_handshake()\n File \"/usr/lib/python3.10/ssl.py\", line 1371, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/requests/adapters.py\", line 486, in send\n resp = conn.urlopen(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 799, in urlopen\n retries = retries.increment(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/retry.py\", line 592, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1021, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/adapters.py\", line 517, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 133, in _do_create_plugin\n disc = self.get_discovery(session,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 605, in get_discovery\n return discover.get_discovery(session=session, url=url,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 1459, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 539, in __init__\n self._data = get_version_data(session, url,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 106, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1154, in get\n return self.request(url, 'GET', **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 930, in request\n resp = send(**kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1025, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://openstack.cyberrange.rit.edu:5000: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 107, in <module>\n _ansiballz_main()\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.openstack.cloud.plugins.modules.catalog_service', init_globals=dict(_module_fqn='ansible_collections.openstack.cloud.plugins.modules.catalog_service', _modlib_path=modlib_path),\n File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 211, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 207, in main\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 415, in __call__\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 113, in run\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 175, in _find\n File \"/opt/ansible/lib/python3.10/site-packages/openstack/service_description.py\", line 89, in __get__\n proxy = self._make_proxy(instance)\n File \"/opt/ansible/lib/python3.10/site-packages/openstack/service_description.py\", line 289, in _make_proxy\n found_version = temp_adapter.get_api_major_version()\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/adapter.py\", line 352, in get_api_major_version\n return self.session.get_api_major_version(auth or self.auth, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1289, in get_api_major_version\n return auth.get_api_major_version(self, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 497, in get_api_major_version\n data = get_endpoint_data(discover_versions=discover_versions)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 268, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 131, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 203, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 155, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://openstack.cyberrange.rit.edu:5000: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} #config excerpts from globals.yml kolla_base_distro: "ubuntu" kolla_internal_vip_address: "100.70.0.20" # /etc/hosts updated on all hosts to have openstack.cyberrange.rit.edu point to the internal IP kolla_internal_fqdn: "openstack.cyberrange.rit.edu" kolla_external_vip_address: "129.21.246.130" kolla_external_fqdn: "openstack.cyberrange.rit.edu" kolla_enable_tls_internal: "yes" kolla_enable_tls_external: "yes" kolla_certificates_dir: "/etc/kolla/certificates" kolla_external_fqdn_cert: "{{ kolla_certificates_dir }}/haproxy.pem" kolla_internal_fqdn_cert: "{{ kolla_certificates_dir }}/haproxy.pem" #kolla_admin_openrc_cacert: "" kolla_copy_ca_into_containers: "yes" haproxy_backend_cacert: "{{ 'ca-certificates.crt' if kolla_base_distro in ['debian', 'ubuntu'] else 'ca-bundle.trust.crt' }}" haproxy_backend_cacert_dir: "/etc/ssl/certs" letsencrypt_email: "fffics(a)rit.edu" enable_letsencrypt: yes letsencrypt_cert_server: "https://acme-v02.api.letsencrypt.org/directory" # attempt to renew Let's Encrypt certificate every 12 hours letsencrypt_cron_renew_schedule: "0 */12 * * *"

3 2