July 2024 - openstack-discuss - lists.openstack.org

[Swift - proxy server] manually exclude nodes via new middleware
by Olivier Chaze 11 Jul '24

11 Jul '24

Hi all, I am writing to introduce a new middleware component, ExcludeNode, which we've just started to develop to enhance node management within the Swift proxy server. [Middleware Overview] ExcludeNode is designed to dynamically exclude specific nodes from the proxy's operations. This middleware reads a list of nodes to be excluded from a specified file and prevents these nodes from being considered during normal operations. Main reason for developing this middleware is to avoid objects request latency/failure when nodes are down or when rebalancing objects after adding new nodes for example. The main functionalities include: - Node Exclusion Check: Validates if a node (IP:PORT) is listed in the exclusion file. - Dynamic Updates: Allows updating the exclusion list by writing node information to the specified file (IP:PORT). - Clear the exclusion list How to use it ? Assuming 1.2.3.4 storage node is down, requests will fail with : proxy-server: ERROR with Object server 1.2.3.4:6210/sdm re: Trying to GET /v1/AUTH_[...]: ConnectionTimeout (2.0s) (txn: tx19b7067dcb0c442e96a10-0066867068) To ban the disk 1.2.3.4:6210/sdm , on each proxy-server : curl -X POST `hostname`:8080/exclude_node -H "Content-Type: application/json" -d {"ip":"'1.2.3.4","port":6210}"' in proxy-server.log will be logged : proxy-server: Node added to exclusion list: 1.2.3.4:6210 (txn: tx0f292e3b08f940b5be3d9-0066867068) This way, the node won't serve any request and depending the request type, this latter will be redirected to another primary or handoff node. It will remain like this until the disk is manually removed from the exclusion list and every time this node would have been chosen to serve a request you'll get a log : proxy-server: Node 1.2.3.4:6210 (sdm) explicitly excluded by exclude_node middleware. (txn: txeb9f5953ee3343dbb9699-006686707a) I would greatly appreciate the community's feedback on the following: * FUNCTIONALITY: Does the middleware fulfill a common need? * IMPLEMENTATION: Are there any concerns regarding the code implementation or its impact on existing functionalities? We'd like to submmit this middleware upstream but to be efficient and avoid wasting everybody's time I'd like some feedbacks first :) The code itself is quite simple today but we plan to add features like auto excluding a node reported as unmounted by swift-recon for example. exclude_node.py middleware : https://kpaste.infomaniak.com/nRp8AdJGnMuLCoP7czIbL9IBJjlSkyYS#ARXTJo5ZnGa5… server.py modification : https://kpaste.infomaniak.com/Qvy992wlRbc6yEgrIPhAhyrh2V3-Gipy#E2TaKDAXQdXp… proxy-server.conf : [pipeline:main] - pipeline = [...] proxy-server + pipeline = [...] exclude_node proxy-server [filter:exclude_node] use=egg:swift#exclude_node exclude_nodes_filename = /dev/shm/excluded_nodes Thank you in advance for your feedback, Olivier

3 3

Cannot create windows image on Openstack
by Ramzi Abdallah 11 Jul '24

11 Jul '24

I have installed openstack version bobcat on Centos 9 Stream. The installation went fine and I am able to login and launch the cirros image. The problem I am facing is with windows images. When I try to upload the Windows Server 2012 R2 Standard Evaluation for OpenStack which I downloaded from cloudebase I get this error: TypeError: Cannot read properties of undefined (reading 'data') - Error: Unable to create the image. At first I though the problem is from the image so I created my own with all the needed drivers and got the same problem. Appreciate any help is someone experienced this before Ramzi

3 4

[nova][libvirt] Timed out during operation: cannot acquire state change lock
by Satish Patel 11 Jul '24

11 Jul '24

Folks, We had ceph storage failure and finally its up and running and in HEALTH_OK state but now I am not able to bring up openstack vms. I am running the Zed release of openstack. I have noticed a very odd error on nova_libvirt and am not sure what the solution is here. root@comp1:~# tail -f /var/log/kolla/libvirt/libvirtd.log 2024-07-08 07:22:07.420+0000: 3160: warning : qemuDomainObjBeginJobInternal:934 : Cannot start job (query, none, none) for domain instance-000014b0; current job is (async nested, none, start) owned by (3159 remoteDispatchDomainCreateWithFlags, 0 <null>, 3159 remoteDispatchDomainCreateWithFlags (flags=0x1)) for (1563s, 0s, 1563s) 2024-07-08 07:22:07.420+0000: 3160: error : qemuDomainObjBeginJobInternal:968 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainCreateWithFlags) 2024-07-08 07:22:37.444+0000: 41620: warning : qemuDomainObjBeginJobInternal:934 : Cannot start job (query, none, none) for domain instance-000015b2; current job is (async nested, none, start) owned by (3162 remoteDispatchDomainCreateWithFlags, 0 <null>, 3162 remoteDispatchDomainCreateWithFlags (flags=0x1)) for (1671s, 0s, 1672s) 2024-07-08 07:22:37.444+0000: 41620: error : qemuDomainObjBeginJobInternal:968 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainCreateWithFlags) I am noticing the VM in a pause state. Any idea what I can do to fix it. I have rebooted compute nodes also in hope it will free up lock or bad process but no luck root@comp1:~# docker exec -it nova_libvirt virsh list Id Name State ---------------------------------- 1 instance-00001751 paused 2 instance-000015b2 paused 3 instance-000014b0 paused When I try to reboot an instance I see the following error. look like qemu acting up.. root@comp1:~# docker exec -it nova_libvirt virsh reboot instance-00001751 error: Failed to reboot domain 'instance-00001751' error: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params)

2 1

[All] CentOS stream 8 jobs
by Tony Breeds 11 Jul '24

11 Jul '24

Hello all, Letting you all know that I, on behalf of the OpenDev infra team bypassed Zuul to merge: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/922649 This means that any job on any branch testing with FIPS need to be switched to CentOS-Stream-9. Yours Tony.

1 0

[python-openstackclient] openstack client can't use swift command
by chunyang wu 10 Jul '24

10 Jul '24

Hello team: The latest version of python-openstackclient on pypi.org is 6.0.0[0], but this version lacks the ability to use commands related to the Swift project due to the absence of the simplejson package. Although this issue was addressed in a recent commit [1]. The fix has not yet been released on pypi.org. I kindly request that the team release a new version to address this issue as soon as possible. thanks [0]: https://pypi.org/project/python-openstackclient/6.6.0/#history [1]: https://review.opendev.org/c/openstack/python-openstackclient/+/920001

1 0

[NON-ZIP][tacker]Troubleshooting Error in CNF Deployment with Tacker and Helm
by yo-kumamoto＠kddi.com 10 Jul '24

10 Jul '24

Hello, I hope this message finds you well. I am currently working on deploying a Cloud-Native Network Function (CNF) using Tacker. For this deployment, I am utilizing Helm charts as the method of deployment. Following the guidelines provided in the official Tacker documentation (https://docs.openstack.org/tacker/latest/user/v2/cnf/deployment_using_helm/…) I encountered an error during the instantiation process. Despite following the steps closely, the deployment did not proceed as expected. I have attached the error logs to this email for your reference. Could you please provide assistance in troubleshooting this issue? Any insights or suggestions would be greatly appreciated. Thank you for your support. Best regards, Yohei Kumamoto

3 3

Magnum Caracal: magnum-conductor fails with after sql QueuePool limit exhaustion
by Michel Jouvin 10 Jul '24

10 Jul '24

Hi, In our tests of Magnum Caracal, we are experiencing a bad error in openstack-magnum-conductor causing it not to work until it is restarted. The problem is caused by a high number of connections to the magnum DB (compared to what can be seen in Antelope version), exceeding the QueuePool limit configured with max_pool_size (+ an overflow of 50). This is described with more details at https://bugs.launchpad.net/magnum/+bug/2067345 that remained mostly unanswered (but with several people joining to say they were experiencing the same issue). Are they people running Magnum Caracal on this list? Are you experiencing the same issue? In our case we are using RDO distribution (EL9-based) and the Magnum DB is on a remote server. Any suggestion welcome! Best regards, Michel

2 2

[magnum] Cancelling meeting
by Jake Yip 10 Jul '24

10 Jul '24

Hi all, I have to cancel today's meeting due to clashing appointments. Regards, Jake

1 0

[adjutant][tc] Checking in on project health
by Goutham Pacha Ravi 10 Jul '24

10 Jul '24

Hello, During a Technical Committee meeting [1], we were made aware that adjutant's CI jobs are perma-failing; and trivial bot-created patches as well as patches from folks addressing CI issues [2] haven't received any review feedback. I wanted to check on the status of development of this project and its deliverables (adjutant, python-adjutantclient, adjutant-ui). If these are not actively developed, perhaps it is time to retire them [3]. Let me know what you think. Thanks, Goutham [1] https://meetings.opendev.org/meetings/tc/2024/tc.2024-07-09-18.00.log.html#… [2] https://review.opendev.org/q/project:openstack/adjutant+status:open [3] https://docs.openstack.org/project-team-guide/repository.html#retiring-a-re…

2 1

Network demand for controller nodes
by William Muriithi 10 Jul '24

10 Jul '24

Hello, Does anyone know if controller nodes are demanding on network needs? I have a situation where I don't have enough 10GB/s port in the switch. Would a cluster be network constrained if I have three controllers using 1GB/s connections and the execution and storage nodes are on 10GB/s connections? Does the VM traffic ever transverse the controller's network cards? In another word, is there a valid requirement for 10GB/s on openstack controllers? Regards, William

4 3