July 2024 - openstack-discuss - lists.openstack.org

[largescale-sig]scaling story
by Alex Song (宋文平) 25 Oct '24

25 Oct '24

Hi everyone, we want to share our scaling story. Please refer the link: https://etherpad.opendev.org/p/large-scale-inspur Thanks, Alex Song

6 8

[nova][libvirtd] internal error: unable to execute QEMU command 'query-named-block-nodes'
by Satish Patel 22 Oct '24

22 Oct '24

Folks, I have a ceph cluster and recently I configured rbd-mirror to replicate all data to remove ceph cluster for disaster recovery. Yesterday for POC we did a hard cutover on ceph and point openstack to the new cluster. All other vms came back up fine but 2 vms stuck in this error state in libvirt logs 2024-01-31 22:44:37.591+0000: 474597: error : qemuMonitorJSONCheckErrorFull:399 : internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied If it's a wider issue than it should impact all the vms but why only two VM stuck and not starting and libvirt giving me this error in logs nova-compute logs also showing same error 2024-01-31 22:44:40.925 7 INFO nova.compute.manager [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] [instance: 2de0f880-77c7-4d2c-9e01-898c57ad3693] Successfully reverted task state from powering-on on failure for instance. 2024-01-31 22:44:40.944 7 ERROR oslo_messaging.rpc.server [None req-b33485b5-8740-48ae-8b5b-a440de3f11a4 c48fcfb9347f413f92fcece065644b00 ca5c652478c7429e964257990800e9cb - - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'query-named-block-nodes': cannot read image start for probe: Permission denied

3 8

[kuryr][tc] Final call for Kuryr PTL/DPL liaisons
by Ghanshyam Mann 15 Oct '24

15 Oct '24

Hello Everyone, Kuryr project ends up being a leaderless project for the 2024.2 Dalmatian cycle[1]. There is no volunteer to serve as PTL or DPL liaisons yet. As the next step, I have proposed to mark this project Inactive: - https://review.opendev.org/c/openstack/governance/+/916945 If you would like to serve as a leader and help maintain this project, this is the right time to step up by replying to this email or commenting in gerrit change 916945. [1] https://etherpad.opendev.org/p/2024.2-leaderless#L99 Ref: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… -gmann

4 10

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

[all][tc][ptls] Eventlet: Python 3.13 - Attach your seatbelt
by Herve Beraud 06 Sep '24

06 Sep '24

Hi all, As Python 3.13 is close to become a reality, new concerning incompatibilities between this new CPython version and Eventlet also become a reality: - https://github.com/eventlet/eventlet/issues/964 Fedora Python maintainers proposed a patch, but for now the problem remains unsolved. Also, many people reported bugs related to incompatibility between python3.12/dnspython/eventlet/pip-system-cert. There is an example who summarize well the problem https://github.com/eventlet/eventlet/issues/957 Do you have some visibility concerning the adoption of Python 3.13 into Openstack? Turbulences ahead... -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

4 5

[Kolla] [LetsEncrypt] Internal/External/Backend Certificates
by jjjamesg＠proton.me 05 Sep '24

05 Sep '24

Hi all, another one here that can't seem to get kolla running with letsencrypt certs, just curious as to what settings are required for kolla to pull internal/external/backend certs from letsencrypt? I've looked at the docs and have tried a few settings but keep getting errors, sure i had this working last week before i tore it down ready to deploy for production and now i just cant get it to complete the deployment because of certs, only thing i can think of is i need to run kolla-ansible -i multinode certificates but i thought that was just for self made certs? Also, what are best practices for backend tls certs, self signed or using the internal cert etc?? Is it possible to use letsencrypt at all for backend? Cheers all

3 6

[ops][kolla][kolla-ansible] Lets Encrypt Deployment on 2024.1/v18
by Forrest Fuqua 03 Sep '24

03 Sep '24

The documentation seems a little incomplete when it comes to deploying a stack with let's encrypt The attached config is after trying a few settings, but the lets encrypt container errors out every time during deployment causing haproxy to not get a proper ssl certificate the services.d entries are showing its ignoring kolla_internal_fqdn_cert and still using kolla_internal.pem in most services causing the following deployment error TASK [service-ks-register : keystone | Creating services] **************************************************************************************************************************************************** FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (5 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (4 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (3 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (2 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (1 retries left). failed: [100.70.0.1] (item=keystone (identity)) => {"action": "os_keystone_service", "ansible_loop_var": "item", "attempts": 5, "changed": false, "item": {"description": "Openstack Identity Service", "endpoints": [{"interface": "internal", "url": "https://openstack.cyberrange.rit.edu:5000"}, {"interface": "public", "url": "https://openstack.cyberrange.rit.edu:5000"}], "name": "keystone", "type": "identity"}, "module_stderr": "Failed to discover available identity versions when contacting https://openstack.cyberrange.rit.edu:5000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n httplib_response = self._make_request(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 404, in _make_request\n self._validate_conn(conn)\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 1058, in _validate_conn\n conn.connect()\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connection.py\", line 419, in connect\n self.sock = ssl_wrap_socket(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/ssl_.py\", line 449, in ssl_wrap_socket\n ssl_sock = _ssl_wrap_socket_impl(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/ssl_.py\", line 493, in _ssl_wrap_socket_impl\n return ssl_context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib/python3.10/ssl.py\", line 513, in wrap_socket\n return self.sslsocket_class._create(\n File \"/usr/lib/python3.10/ssl.py\", line 1100, in _create\n self.do_handshake()\n File \"/usr/lib/python3.10/ssl.py\", line 1371, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/requests/adapters.py\", line 486, in send\n resp = conn.urlopen(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 799, in urlopen\n retries = retries.increment(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/retry.py\", line 592, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1021, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/adapters.py\", line 517, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 133, in _do_create_plugin\n disc = self.get_discovery(session,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 605, in get_discovery\n return discover.get_discovery(session=session, url=url,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 1459, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 539, in __init__\n self._data = get_version_data(session, url,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 106, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1154, in get\n return self.request(url, 'GET', **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 930, in request\n resp = send(**kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1025, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://openstack.cyberrange.rit.edu:5000: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 107, in <module>\n _ansiballz_main()\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.openstack.cloud.plugins.modules.catalog_service', init_globals=dict(_module_fqn='ansible_collections.openstack.cloud.plugins.modules.catalog_service', _modlib_path=modlib_path),\n File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 211, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 207, in main\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 415, in __call__\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 113, in run\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 175, in _find\n File \"/opt/ansible/lib/python3.10/site-packages/openstack/service_description.py\", line 89, in __get__\n proxy = self._make_proxy(instance)\n File \"/opt/ansible/lib/python3.10/site-packages/openstack/service_description.py\", line 289, in _make_proxy\n found_version = temp_adapter.get_api_major_version()\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/adapter.py\", line 352, in get_api_major_version\n return self.session.get_api_major_version(auth or self.auth, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1289, in get_api_major_version\n return auth.get_api_major_version(self, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 497, in get_api_major_version\n data = get_endpoint_data(discover_versions=discover_versions)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 268, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 131, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 203, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 155, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://openstack.cyberrange.rit.edu:5000: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} #config excerpts from globals.yml kolla_base_distro: "ubuntu" kolla_internal_vip_address: "100.70.0.20" # /etc/hosts updated on all hosts to have openstack.cyberrange.rit.edu point to the internal IP kolla_internal_fqdn: "openstack.cyberrange.rit.edu" kolla_external_vip_address: "129.21.246.130" kolla_external_fqdn: "openstack.cyberrange.rit.edu" kolla_enable_tls_internal: "yes" kolla_enable_tls_external: "yes" kolla_certificates_dir: "/etc/kolla/certificates" kolla_external_fqdn_cert: "{{ kolla_certificates_dir }}/haproxy.pem" kolla_internal_fqdn_cert: "{{ kolla_certificates_dir }}/haproxy.pem" #kolla_admin_openrc_cacert: "" kolla_copy_ca_into_containers: "yes" haproxy_backend_cacert: "{{ 'ca-certificates.crt' if kolla_base_distro in ['debian', 'ubuntu'] else 'ca-bundle.trust.crt' }}" haproxy_backend_cacert_dir: "/etc/ssl/certs" letsencrypt_email: "fffics(a)rit.edu" enable_letsencrypt: yes letsencrypt_cert_server: "https://acme-v02.api.letsencrypt.org/directory" # attempt to renew Let's Encrypt certificate every 12 hours letsencrypt_cron_renew_schedule: "0 */12 * * *"

3 2

PTG October 2024 Team Signup
by Kendall Nelson 29 Aug '24

29 Aug '24

Hello Everyone, Hopefully you have seen by now, the next OpenInfra PTG[1] which will take place October 21-25, 2024 virtually! Registration is now open[2]! To sign up your team, you must complete the survey[3] by August 30th at 7:00 UTC. We NEED accurate contact information (email and IRC nickname ideally) for the moderator of your team’s sessions. This is because the survey information will be used to organize the schedule signups which will be done via the PTGBot. If you are not on IRC, please get setup[4] on the OFTC network and join #openinfra-events. You are also encouraged to familiarize yourself with the PTGBot documentation[5] as well. If you have any questions, please reach out to ptg(a)openinfra.dev! Information about signing up for time slots will be sent to moderators after the team signup deadline. Continue to visit openinfra.dev/ptg for updates. -Kendall (diablo_rojo) [1] PTG Site: https://openinfra.dev/ptg/ [2] PTG Registration: http://ptg.openinfra.dev/ [3] Team Survey: https://openinfrafoundation.formstack.com/forms/oct2024_ptg_survey [4] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [5] PTGBot README: https://opendev.org/openstack/ptgbot/src/branch/master/README.rst

1 2

[MAGNUM] Fedora Core OS include NVIDIA GPU drivers
by Oliver Weinmann 19 Aug '24

19 Aug '24

Hi all, I'm running OpenStack 2023.2 (Bobcat) deployed with Kolla-Ansible. Currently I'm struggling to get the NVIDIA drivers to work in a FCOS 35 image. I know FCOS 35 is EOL and outdated, but for a K8s 1.21.11 cluster it is the only version that still works with Magnum. So far I managed to get the GPU Passthrough working for two NVIDIA Tesla T4 GPUs. I have attached the GPUs to an Ubuntu instance and ran geekbench without any issues. Then I tried to install the latest NVIDIA drivers in FCOS 35 by running: sudo rpm-ostree install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$ (rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$ (rpm -E %fedora).noarch.rpm systemctl reboot rpm-ostree install --apply-live akmod-nvidia akmods pciutils rpmdevtools xorg-x11-drv-nvidia-cuda It fails on: rpm-ostree install --apply-live akmod-nvidia akmods pciutils rpmdevtools xorg-x11-drv-nvidia-cuda error: Cannot download rpm-build-libs-4.17.0-4.fc35.x86_64.rpm: All mirrors were tried; Last error: Status code: 403 for https://fedoraproject-updates-archive.fedoraproject.org/fedora/35/x86_64/rp… I guess the binaries are no longer hosted? Running the same on FCOS 36 and latest FCOS 40 works fine. Next I have taken new images from the FCOS instances with the NVIDIA driver. openstack server image create --name fedora_coreos_36_nvidia_cuda fedora_coreos_35_latest_no_update --wait This takes ages but anyway I can successfully deploy new instances from the images with my gpu flavor and the GPU is working fine. Again tested with geekbench. But now when adding new K8s worker nodes in a fresh nodegroup on an existing K8s 1.21.11 cluster the nodes are not joining the cluster: openstack coe nodegroup create \ --node-count 1 \ --role worker-gpu \ --flavor gpuflavor \ --image fedora_coreos_36_nvidia_cuda \ $CLUSTER_ID worker-gpu I have tested the latest available images of FCOS36 and they work fine, so something must be wrong with how I created the new image. Is there some magic that needs to be done? I'm also not sure if this is the best way of doing this. Maybe there is already a ready FCOS image with nvidia drivers? I can find lots of outdated information on the internet. Generally I tend to use Vexxhost CAPI from now on, which works great. This is really just a one off attempt to have a working NVIDIA T4 GPU in a K8s 1.21.11 cluster. Best Regards, Oliver

5 16

networking-onos plugin
by Ganesan, Sathya 14 Aug '24

14 Aug '24

Good Day, Is there an alternative integration available for ONOS and Openstack since networking-onos is archived? Sathya Ganesan Geico Tech | Data Center Core Network Engineering - SDN | sganesan(a)geico.com<mailto:sganesan@geico.com> | 240.478.2121 | M-F 9a-5:30p EST Sensitivity: Company-Internal ==================== This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message.

2 2

[rabbitmq] Maximum Number of Sessions (8192) Reached
by Eugen Block 14 Aug '24

14 Aug '24

Hi *, I can't seem to find much about rabbitmq and logind, so I wanted to ask the list if anyone has encountered the same and if so, how they dealt with it. We're supporting a Victoria cluster (installed with our own deployment method) mostly controlled by pacemaker. And on the rabbit master node I see this warning constantly: ---snip--- 2024-07-29T14:09:23.552576+02:00 control01 su: pam_unix(su:session): session opened for user rabbitmq by (uid=0) 2024-07-29T14:09:24.450657+02:00 control01 su: pam_unix(su:session): session closed for user rabbitmq 2024-07-29T14:09:24.500356+02:00 control01 su: (to rabbitmq) root on none 2024-07-29T14:09:24.502370+02:00 control01 su: pam_systemd(su:session): Failed to create session: Maximum number of sessions (8192) reached, refusing further sessions. 2024-07-29T14:09:24.502681+02:00 control01 su: pam_unix(su:session): session opened for user rabbitmq by (uid=0) 2024-07-29T14:09:25.565203+02:00 control01 su: pam_unix(su:session): session closed for user rabbitmq 2024-07-29T14:09:25.609613+02:00 control01 su: (to rabbitmq) root on none ---snip--- Looking into loginctl list-sessions, almost all of them belong to rabbitmq and they have a very old timestamp (2023). I'm aware of older systemd versions which can't handle closing sessions correctly [0], but we can't upgrade at this time. Would enabling "linger" fix this (loginctl enable-linger rabbitmq) after a reboot? During the next maintenance window I would reboot control01 and watch how the other control nodes behave wrt rabbit login-sessions. But I'm wondering if somebody already has dealt with this? Thanks for any pointers! Eugen [0] https://www.suse.com/support/kb/doc/?id=000020549

1 2

Firewall as a Service - kolla ansible 2023.1
by Thamanna Farhath 07 Aug '24

07 Aug '24

Hello OpenStack Community, I hope this message finds you well. My name is Thamanna , and I am currently working with the OpenStack Kolla-Ansible deployment version 2023.1. I am interested in deploying Firewall as a Service (FWaaS) within our OpenStack environment. However, I have noticed that FWaaS is not available in the Kolla-Ansible 2023.1 version. Could anyone please provide guidance on the following points? 1. Is there an official or recommended way to integrate FWaaS with Kolla-Ansible 2023.1? 2. Are there any specific configurations or additional components required to enable FWaaS? 3. Has anyone in the community successfully deployed FWaaS with this version, and if so, could you share your experience and any potential challenges faced? 4. Are there alternative approaches or tools that you would recommend for deploying FWaaS in an OpenStack environment? Any help or pointers to relevant documentation would be greatly appreciated. Thank you in advance for your assistance! Best regards, Thamanna mailto:thamanna.f@zybisys.com Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

3 5

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

Assistance needed - Application Credential
by Naveen Anbarasu 05 Aug '24

05 Aug '24

Hi team, I have created an application credential for the compute service and successfully obtained a token. Access rule: "service": "compute", "method": "GET", "path": "/v2.1/servers" However, when using the token to retrieve the server list, I receive a Error 403 Forbidden error with the message: 'Policy doesn't allow os_compute_api:servers to be performed. But i have a necessary permission within the respective project (member role) How can I solve this problem? Thanks in advance

7 12

[openstack][kolla-ansible]about upgrading openstack
by Nguyễn Hữu Khôi 04 Aug '24

04 Aug '24

Hello. I read this doc about upgrading openstack https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html But I would like to know if We install a new kolla-ansible(version to upgrade) on the new deployment host, apply changes with password.yml, global.yml and multinode and keep old /etc/kolla/config. Finally, we use kolla-ansible deploy instead of kolla-ansible upgrade. Are these steps ok? Pls correct me if I am wrong. Thank you. Regards Nguyen Huu Khoi

3 8

[election] [tc] Notification of non-candidacy; how it impacts TC diversity
by Jay Faulkner 01 Aug '24

01 Aug '24

Hi all, The nomination period for TC members / PTLs is coming up soon. I wanted to inform the community that I do not intend on standing for election for another year on the TC. Now is a good time, especially if you work somewhere other than Red Hat, to consider serving as a TC member to ensure our leadership team contains representation for all perspectives of OpenStack use. I'm happy to speak with anyone privately who is interested in running about what it's like to be on the TC, and to share some strategies I exercised to ensure the value of my TC membership was being communicated to leadership at my job. Thanks, Jay Faulkner Technical Committee Member

6 5

oslo.service
by engineer2024 01 Aug '24

01 Aug '24

Can anyone guide me how to configure and use oslo.service for nova-api service for tracing and debugging ? Thanks elinux

3 2

[cinder] Stepping down from Cinder core
by Pete Zaitcev 01 Aug '24

01 Aug '24

Hello: Because of a job change, my time available is becoming quite limited, so I'm not going to be able to contribute at the appropriate level. So, I contacted our PTL to get me removed from the core team. Thanks everyone for the help and the journey in Cinder. Greetings, Pete

3 2

nova deploy issue
by Parsa Aminian 30 Jul '24

30 Jul '24

hello when I try to reconfigure nova with kolla ansible this error will be shown : TASK [nova-cell : Waiting for nova-compute services to register themselves] ******************************************************************************************* task path: /usr/local/share/kolla-ansible/ansible/roles/nova-cell/tasks/wait_discover_computes.yml:8 fatal: [control01 -> {{ groups[nova_cell_conductor_group][0] }}]: FAILED! => { "msg": "The conditional check 'inventory_hostname == all_computes_in_batch[0]' failed. The error was: error while evaluating conditional (inventory_hostname == all_computes_in_batch[0]): 'all_computes_in_batch' is undefined\n\nThe error appears to be in '/usr/local/share/kolla-ansible/ansible/roles/nova-cell/tasks/wait_discover_computes.yml': line 8, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n- block:\n - name: Waiting for nova-compute services to register themselves\n ^ here\n" Does anyone have any idea what I should do ?

1 0

Assistance Needed for Trove Instance Creation
by Thamanna Farhath 30 Jul '24

30 Jul '24

Hello community, I am currently deploying Trove using Kolla Ansible 2023.1, following the guide provided here: https://satishdotpatel.github.io/trove-dbaas-openstack/ . Despite cleaving to the instructions, I am facing an issue where the Trove instance ends up in an error state. trove-guestagent.conf [DEFAULT] transport_url = rabbit://openstack:USI0SmaT7lPF7MJCxHPZ4alm8wiyxoihssD1VlnA@192.168.1.101:5672,openstack:USI0SmaT7lPF7MJCxHPZ4alm8wiyxoihssD1VlnA@192.168.1.102:5672,openstack:USI0SmaT7lPF7MJCxHPZ4alm8wiyxoihssD1VlnA@192.168.1.103:5672// control_exchange = trove root_grant = ALL root_grant_option = True debug = False Docker Status Check: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f698ee14d928 mysql:5.7.29 "docker-entrypoint.s…" 18 minutes ago Up 18 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database Telnet Connection: telnet 192.168.50.78 3306 Trying 192.168.50.78... Connected to 192.168.50.78. Escape character is '^]'. J 5.7.29p+C5 }0^k[zH3mysql_native_password Syslog(SSH into the trove instance) ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)\n": Exception: Running command error: b"mysql: [Warning] Using a password on the command line interface can be insecure.ERROR trove.guestagent.datastore.service [None req-a03ff3dd-5a8f-4fd4-80d5-76e31fa16dd0 - - - - - -] Timeout while waiting for database status to change. Expected state healthy, current state is running When looking for the missing directory, I discovered it in /var/lib/mysql/mysqld.sock and /var/lib/mysql/mysqld.pid paths. How can I get an active instance?Kindly help me resolve this. Thank you in advance Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

3 2

[horizon] How to enable MFA using TOTP on Horizon (ZED)
by Shubham Kumar Yadav 30 Jul '24

30 Jul '24

Hi all, I was working on a MFA in openstack (ZED). Is it possible to enable MFA using TOTP on Horizon? I'm using a ZED Openstack with Ubuntu 22.04 , and I know that from the CLI point of view, you can have Multi Factor Authentication, but I don't see that implemented on Horizon. i added MFA using - https://docs.openstack.org/keystone/pike/advanced-topics/auth-totp.html but now the problem is i can use the token issued to run openstack cli but not able to login into Horizon with user and password/totp. I see The Totp support was added to Horizon in the Bobcat (2023.2) release So i would like to ask how it is possible to backport the feature in Horizon in the zed release. I would appreciate any help! Thanks!

2 1

[ironic] ironic-inspector bugfix branches
by Riccardo Pittau 30 Jul '24

30 Jul '24

Hello ironicers! With the deprecation and the migration of the ironic-inspector functionalities to the new agent interface in ironic we have decided to also stop cutting bugfix branches for the project. The last bugfix branch for ironic-inspector is bugfix/12.2 supported until December 2024. We'll continue with the normal release schedule until full retirement of the project. Thanks Ciao! Riccardo

1 0

[neutron] CI Meeting cancelled for today
by Yatin Karel 30 Jul '24

30 Jul '24

Hello Neutron Team !! I will not be able to chair the CI meeting today as I have to attend some internal meetings. Sorry for the late notice. We have some known issues in CI and all are under track with patches in gate or already merged. There are some issues in the periodic line specific to ovs-master jobs and 9-stream so will report issues for those. -- Thanks and Regards Yatin Karel

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-9)
by Goutham Pacha Ravi 29 Jul '24

29 Jul '24

Hello Stackers, We're counting down 9 weeks to the 2024.2 (Dalmatian) release [1]. OpenStack projects that follow the cycle-with-rc release model [2] will have their code freeze in 4 weeks. Between now and the release, the release team will communicate with us weekly to coordinate deliverables across the community. The elections for OpenStack PTLs and Technical Committee members are around the corner as well, so now is a good time to ensure that you are eligible to vote in these elections! Please navigate to: https://civs1.civs.us/cgi-bin/opt_in.pl Enter your preferred email address on Gerrit and request activation. It only takes a minute! :) The following are updates from the OpenStack Technical Committee during the past week. === Weekly Meeting === The TC had a productive meeting via IRC on OFTC's #openstack-tc channel last week [3]. We discussed follow-ups for the eventlet removal goal proposal [4] and recent updates to inactive and emerging OpenStack projects. We spent some time assessing gate health concerns, including an issue with a change to `setuptools` that requires us to bump the upper constraint on the "packaging" library in stable/2023.1 [5]. This was important to get some CVE fixes landed in various OpenStack code repositories. Finally, we refreshed our understanding of the TC's house rules. To summarize: - Changes to the TC charter require at least 2/3 of the TC voting in favor, while governance motions require at least 1/3 of the TC to vote in favor. - TC members can vote positively, negatively, or abstain (using the “RollCall-Vote” in Gerrit). - Decisions need more positive votes than negative votes (ties mean the proposal is rejected). - Any motion or charter-change is deliberately open at least a week after it was last proposed/updated, allowing the TC and the wider community to participate fully. - Significant governance changes take a while to merge because of the importance of ensuring full TC and community engagement in votes. Our next weekly meeting is tomorrow (2024-07-30) at 1800 UTC via IRC on OFTC's #openstack-tc channel. You can find the agenda on the OpenStack TC wiki page [6]. Please consider joining us and remember, you can edit the wiki and propose new topics! === Governance Proposals === ==== Merged ==== - Add `os-test-images` under glance | https://review.opendev.org/c/openstack/governance/+/925044 ==== Open for review/discussion ==== - [Follow-up] Fix typos in the eventlet proposal | https://review.opendev.org/c/openstack/governance/+/924944 - Update criteria for inactive projects to become active again | https://review.opendev.org/c/openstack/governance/+/921500 - Follow up on the affiliation diversity handling | https://review.opendev.org/c/openstack/governance/+/923876 - Inactive state extensions: Monasca | https://review.opendev.org/c/openstack/governance/+/923919 === Upcoming Events === - 2024-08-12: Call for Proposals due for the OpenInfra Days NA [7] - 2024-08-14: Nominations open for 2025.1 OpenStack PTL+TC elections - 2024-08-24: OpenInfra Days, Vietnam: https://2024.vietopeninfra.org/ - 2024-08-29: Dalmatian-3 milestone - 2024-09-03: OpenInfra Summit Asia, Suwon The OpenStack Technical Committee serves you! We're always willing to chat and would love feedback: === How to contact the TC === You can reach out to the TC in multiple ways: 1. Email: Send an email with the tag [tc] on this email list. 2. Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel. 3. Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [7]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] OpenStack cycle-with-rc model: https://releases.openstack.org/reference/release_models.html#cycle-with-rc [3] TC Meeting IRC Logs, 2024-07-23: https://meetings.opendev.org/meetings/tc/2024/tc.2024-07-23-18.00.log.html [4] Remove Eventlet From Openstack: https://review.opendev.org/c/openstack/governance/+/902585 [5] Update packaging constraint in stable/2023.1: https://review.opendev.org/c/openstack/requirements/+/924764 [6] TC Meeting, 2024-07-30: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [7] Call for Proposals, OpenInfra Days NA: https://docs.google.com/forms/d/e/1FAIpQLSeFQBL1qGNegDWDHCVTLtu_wLf06w7JJkc…

1 0

Cinder Bug Report 2024-07-29
by Jon Bernard 29 Jul '24

29 Jul '24

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - When restoring volume using the imported backup, it is always scheduled to the same host. <https://bugs.launchpad.net/cinder/+bug/2074058> - Status: In Progress (assigned) - The cinder-backup swift driver connects to swift in a different region. <https://bugs.launchpad.net/cinder/+bug/2074126> - Status: In Progress (assigned) - allow_qcow2_backing_file flag cannot be passed into some image_utils operations <https://bugs.launchpad.net/cinder/+bug/2074377> - Status: New (unassigned) Thanks, -- Jon

1 0