October 2024 - openstack-discuss - lists.openstack.org

Improving stestr... or switching to a (better) compatible test runner?
by Thomas Goirand 09 Oct '24

09 Oct '24

Hi, This is the 2nd time I'm opening a thread about it. Sorry for this, but it really bothers me. Today, running tempest, I got these stats from stestr: - Worker 0 (132 tests) => 0:24:49.209885 - Worker 1 (100 tests) => 0:17:14.791845 - Worker 2 (124 tests) => 0:42:52.690906 - Worker 3 (189 tests) => 0:41:21.307241 - Worker 4 (159 tests) => 0:45:49.503031 - Worker 5 (143 tests) => 0:28:13.282371 - Worker 6 (156 tests) => 3:16:52.364976 - Worker 7 (103 tests) => 0:46:05.366089 So, thread #1 ran 17 minutes, sitting idle for the rest of the 3 hours run of thread #6. While I thought stestr was deviding the number of tests by the number of thread, I don't get why worker #1 only had 100 tests assigned. All together, all tests could have been ran within maybe less than an hour of time (rather than 3h16 above), if not-yet-ran-tests were reassigned to idle treads. I'm currently spending a lot of time on running tempest. Indeed, I'm running tempest on each OpenStack upgrade, from Victoria up to Dalmatian. With the current way stestr run, it may take 2 full days to do that (and that's not even counting when upgrade will fail and will need fixing...), when it could be done in maybe 8 hours (if I count 1h per OpenStack release). So, knowing the above, it might be a good use of my time to dig into stestr and see if I can fix this... or not! Does anyone have a suggestion for another test runner, that's compatible with stestr, at least for the tests selection with a regular expression? As much as I know, pytest cannot take a regexp for test selection, can it? Or is there maybe a plugin for it? If there's no compatible test runner, where should I dig in the stestr code to rewrite things in a smarter way? Cheers, Thomas Goirand (zigo) P.S: On Caracal today, I just had this: ============== Worker Balance ============== - Worker 0 (117 tests) => 0:59:32.291385 - Worker 1 (147 tests) => 1:22:28.228980 - Worker 2 (125 tests) => 0:45:20.969397 - Worker 3 (114 tests) => 1:46:21.667579 - Worker 4 (170 tests) => 0:45:27.577738 - Worker 5 (190 tests) => 2:28:39.744920 - Worker 6 (182 tests) => 2:29:01.402255 - Worker 7 (152 tests) => 2:29:10.183359 this looks better, but that's probably 1/ random 2/ still not perfect, with worker #0, #2 and #4 doing nothing 2/3rd of the time.

2 1

[tc][watcher] No leaders for project team, heading to retirement
by Goutham Pacha Ravi 09 Oct '24

09 Oct '24

Hello Stackers, The "watcher" project [1] did not have any volunteers to lead the project in the 2025.1 election cycle. In the past few years, the project's team has slowly disbanded and there have been no significant changes proposed/merged in the project's repositories [2]. We noticed that there were three deployers that noted their use of Watcher in production in the past user survey [3]. It's possible that the project failed to gather any significant adoption, and hence there's less interest in contributing to, and maintaining the project. I'm sending this notification pursuant to the OpenStack governance recommendations [4] to retire project teams. We can mark a project team "inactive", however, an "inactive" project must have some contributors aiming to build-back a maintainers team around it - and it must pick a project team lead, or adopt a distributed project leadership model by nominating liaisons. Since there are no PTL nominees, we think marking this project "inactive" serves no purpose. Is there any objection to retiring the "openstack/watcher" project (and associated deliverables: python-watcherclient, watcher-dashboard and watcher-tempest-plugin)? Once the project is retired, configuration management and deployment projects can drop their integrations as well. Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi [1] https://wiki.openstack.org/wiki/Watcher [2] https://review.opendev.org/q/(project:%5Eopenstack/watcher.*+OR+project:ope… [3] https://governance.openstack.org/tc/user_survey/analysis-2023.html [4] https://governance.openstack.org/tc/reference/dropping-projects.html

5 7

[all][manila][ironic][swift][bandit] removal of Python standard libraries in Python 3.13
by Thomas Goirand 09 Oct '24

09 Oct '24

Hi, OpenStack Dalmatian is currently fully in Debian Testing, and in a quite good shape. However, as you may know, Python 3.13 is about to be uploaded to Debian unstable, and we intend to release Debian 13 (aka: Trixie) with it. A number of OpenStack (or related) packages are affected by the removal of Python standard libraries in Python 3.13. We need to fix these ASAP (and not only because of Debian). Here's the list of affected projects, with the name of the bad import that was removed from Python 3.13: manila https://bugs.debian.org/1084543 (pipes) swift https://bugs.debian.org/1084544 (cgi) ironic https://bugs.debian.org/1084566 (crypt) ironic-python-agent https://bugs.debian.org/1084680 (crypt) bandit https://bugs.debian.org/1084700 (crypt, telnetlib) There's also a number of libraries that we don't directly maintain, but that are in our requirements: qpid-proton https://bugs.debian.org/1084552 (cgi) python-wsme https://bugs.debian.org/1084594 (cgi) python-falcon https://bugs.debian.org/1084623 (cgi) python-pysaml2 https://bugs.debian.org/1084670 (cgi) These will need to be fixed. For more details with were the bad import was found, please dive into the individual bug reports. While pipes and crypt have direct replacement (subprocess and for example the pypi module bcrypt), there's a few comments here: https://docs.python.org/3.13/whatsnew/3.13.html While I will for sure attempt to fix some of the above, I cannot be held responsible for all of OpenStack code, and I'd appreciate a lot having some help. Unless I'm being pointed at fixes patches in Gerrit, I'll start filling bugs for the above once I'm done with other tasks (currently, I'm working on validation of OpenStack upgrades in Debian), and hopefully, get all in order before Trixie's freeze in early 2025. I know there's probably going to be more Python 3.13 fixes to come, and I expect some package to fail to build from source (FTBFS) due to some tests failing in 3.13. Then I'll run my usual tempest validation CI, and hopefully, Dalmatian on Trixie will be the best OpenStack ever... :) In the mean time, let's tackle the above. Cheers, Thomas Goirand (zigo)

7 7

[publiccloud-sig] Heads-Up: next meeting October 9th (tomorrow) - 0700 UTC
by Felix Kronlage-Dammers 08 Oct '24

08 Oct '24

Hi everyone, This Wednesday, tomorrow, (October 9th) the next meeting of the publiccloud-sig is going to happen. Thanks to Artem (gtema) and Joern (joek-office) who helped prepare the meeting agenda for this wednesday: - Housekeeping - Upcoming vPTG - Reminder to sign up for PTG: https://openinfra.dev/ptg/ - CfPs for Devrooms @ FOSDEM 2025 -> Sovereign Cloud Devroom? - Continued discussion regarding IAM Federation If you’re interesting in IAM federation, be sure to join! We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> Due to vacation, I’ll not be around so I wish a fruitful meeting :) felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

Configuring Multiple IdPs with OpenStack and Shibboleth
by Abubakur Sait 08 Oct '24

08 Oct '24

Hello everyone, I have successfully configured a single Shibboleth Federation system with OpenStack (using a single IdP and SP) on a DevStack deployment. Now, I’m looking to expand the setup to support multiple IdPs with OpenStack. I followed the instructions on the Shibboleth SessionInitiator page ( https://shibboleth.atlassian.net/wiki/spaces/SP3/pages/2065334685/SessionIn…) but I haven’t had any success so far. Does anyone have experience or suggestions on how to configure multiple IdPs in this setup? Thank you in advance for any guidance! Best regards, Abubakur Sait

1 0

How to create snapshot via Python module
by Saeed Fazlollahzadeh 08 Oct '24

08 Oct '24

Hello, this is my full code to create a snapshot from an instance. The code creates a new image with type snapshot, but after around 2 hours, its status is still in queue. But when I create another snapshot via the Openstack UI, it works fine. Would you please help me how can I create a snapshot? import openstack import datetime now = datetime.datetime.now() today = now.strftime('%Y_%m_%d_%H_%M_%S') openstack.enable_logging(debug=False) servers = [] images = [] openstack_connection = openstack.connect(cloud='region1') # clouds.yaml for server in openstack_connection.compute.servers(details=True, hostname='my_server'): servers.append(server.to_dict()) servers = servers[0] for image in openstack_connection.compute.images(details=True, id=servers['image']['id']): images.append(image.to_dict()) images = images[0] name = f'my_server_{today}' openstack_connection.image.create_image( name=name, disk_format='raw', container_format='bare', visibility='private', timeout=1, properties={ 'hw_disk_bus': 'scsi', 'hw_qemu_guest_agent': 'yes', 'hw_scsi_model': 'virtio-scsi', 'hw_watchdog_action': 'reset', 'base_image_ref': servers['image']['id'], 'owner_user_name': 'whmcs', 'owner_project_name': 'admin', 'boot_roles': 'admin,reader,member', 'os_admin_user': 'administrator', 'os_distro': 'windows', 'os_type': 'windows', 'owner_specified.openstack.object': images['metadata']['owner_specified.openstack.object'], 'clean_attempts': '1', 'hw_machine_type': 'pc', 'password_0': '', 'password_1': '', 'password_2': '', 'password_3': '', 'hw_cdrom_bus': 'ide', 'hw_input_bus': 'usb', 'hw_pointer_model': 'usbtablet', 'hw_video_model': 'virtio', 'hw_vif_model': 'virtio', 'instance_uuid': servers['id'], 'image_type': 'snapshot', } )

2 2

How to create snapshot via python module?
by Saeed 08 Oct '24

08 Oct '24

Hello, this is my full code to create a snapshot from an instance. The code creates a new image with type snapshot, but after around 2 hours, its status is still in queue. But when I create another snapshot via the Openstack UI, it works fine. Would you please help me how can I create a snapshot? import openstack import datetime now = datetime.datetime.now() today = now.strftime('%Y_%m_%d_%H_%M_%S') openstack.enable_logging(debug=False) servers = [] images = [] openstack_connection = openstack.connect(cloud='region1') # clouds.yaml for server in openstack_connection.compute.servers(details=True, hostname='my_server'): servers.append(server.to_dict()) servers = servers[0] for image in openstack_connection.compute.images(details=True, id=servers['image']['id']): images.append(image.to_dict()) images = images[0] name = f'my_server_{today}' openstack_connection.image.create_image( name=name, disk_format='raw', container_format='bare', visibility='private', timeout=1, properties={ 'hw_disk_bus': 'scsi', 'hw_qemu_guest_agent': 'yes', 'hw_scsi_model': 'virtio-scsi', 'hw_watchdog_action': 'reset', 'base_image_ref': servers['image']['id'], 'owner_user_name': 'whmcs', 'owner_project_name': 'admin', 'boot_roles': 'admin,reader,member', 'os_admin_user': 'administrator', 'os_distro': 'windows', 'os_type': 'windows', 'owner_specified.openstack.object': images['metadata']['owner_specified.openstack.object'], 'clean_attempts': '1', 'hw_machine_type': 'pc', 'password_0': '', 'password_1': '', 'password_2': '', 'password_3': '', 'hw_cdrom_bus': 'ide', 'hw_input_bus': 'usb', 'hw_pointer_model': 'usbtablet', 'hw_video_model': 'virtio', 'hw_vif_model': 'virtio', 'instance_uuid': servers['id'], 'image_type': 'snapshot', } )

1 0

[neutron] OVS Agent constant bridges recreation
by Artsiom Antropau 08 Oct '24

08 Oct '24

Hi, Team! Pretty long time we’re experiencing some troubles with auto-restoration of ovs after some storms. We’re using neutron-wallaby and openvswitch 2.15. The problem is the next: After some storms we’re receiving situation, when ovs agent constantly recreating bridges and doing full syncs, which affects our network stability. All versions (incl. 2024.2), containing some parts of code that I want to highlight: 1. Non-empty `idl_monitor.bridges_added` triggers 2. `_do_reconfigure_physical_bridges`, then it triggers 3. `setup_physical_bridges(bridge_mappings)` then it contains 4. br.create() I’m seeing note in code «The bridge already exists, so create won't recreate it», BUT at the same time we’re receiving in logs: ``` Matched CREATE: BridgeCreateEvent(events=('create',), table='Bridge', conditions=None, old_conditions=None) to row=Bridge(name=br-tun ``` and ``` BridgeCreateEvent, bridge name: br-tun ``` Constantly – by hours and for all our bridges (we have four: br-tun, br-ex, br-int, br-rmi) while we will not restart ovs agent. So this event filling idl_monitor.bridges_added then ovs reacting on it by _do_reconfigure_physical_bridges, then it triggers br.create() and we’re receiving event once again – we’re in loop. Want to add that our datapath-id NOT changing between these runs – keeps same. Any ideas or ways how to fix this problem and never experience it next? [https://my.gcdn.co/images/signature/gcore_logo.png]<https://gcore.com/> Artsiom Antropau Software Engineer artsiom.antropau(a)gcore.com<mailto:artsiom.antropau@gcore.com> +48796909739<tel:+48796909739> gcore.com<https://gcore.com/> [https://my.gcdn.co/images/signature/linkedin.png]<https://www.linkedin.com/company/g-core> [https://my.gcdn.co/images/signature/x.png] <https://x.com/gcore_official> [https://my.gcdn.co/images/signature/youtube.png] <https://www.youtube.com/channel/UCFhtQE7HB097a1_Nfv-Ma_Q> [https://my.gcdn.co/images/signature/facebook.png] <https://www.facebook.com/gcorelabscom/> [https://my.gcdn.co/images/signature/instagram.png] <https://www.instagram.com/gcore.official/> This message is intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to us, and immediately and permanently delete it. Do not use, copy or disclose the information contained in this message or in any attachment.

2 1

image creation error
by BEDDA Fadhel 08 Oct '24

08 Oct '24

I'm trying to create a debian image for openstack from a file. on the graphical interface it still remains waiting. On the command line: I receive the message: HttpException: 502: Server Error for url: http://192.168.11.40/image/v2/images/1bb75b80-f6d2-4a26-a941-a52b03e48455/f…, : 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.52 (Ubuntu) Server at 192.168.11.40 Port 80 Merci

1 0

[manila] stuck at connecting ceph backend with manila
by Shubham Kumar Yadav 08 Oct '24

08 Oct '24

hello, I'm trying to install manila in my openstack cloud (deployed on k8s cluster) but having problem. 1 -> i created the auth using ceph auth get-or-create client.manila -o manila.keyring \ mgr 'allow rw' \ mon 'allow r' source - https://docs.openstack.org/manila/latest/configuration/shared-file-systems/… 2 -> in manila-share pod i see the error 2024-10-08 07:37:25.090 7 INFO manila.share.drivers.cephfs.driver [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... 2024-10-08 07:37:25.104 7 ERROR manila.share.manager [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-1@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-10-08 07:37:25.104 7 ERROR manila.share.manager Traceback (most recent call last): 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client 2024-10-08 07:37:25.104 7 ERROR manila.share.manager self._rados_client.connect() 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect 2024-10-08 07:37:25.104 7 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) 2024-10-08 07:37:25.104 7 ERROR manila.share.manager 2024-10-08 07:37:25.104 7 ERROR manila.share.manager During handling of the above exception, another exception occurred: 2024-10-08 07:37:25.104 7 ERROR manila.share.manager 2024-10-08 07:37:25.104 7 ERROR manila.share.manager Traceback (most recent call last): 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup 2024-10-08 07:37:25.104 7 ERROR manila.share.manager self.driver.do_setup(ctxt) 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup 2024-10-08 07:37:25.104 7 ERROR manila.share.manager rados_client=self.rados_client, 2024-10-08 07:37:25.104 7 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client 2024-10-08 07:37:25.104 7 ERROR manila.share.manager raise exception.ShareBackendException( 2024-10-08 07:37:25.104 7 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-10-08 07:37:25.104 7 ERROR manila.share.manager 2024-10-08 07:37:25.105 7 DEBUG manila.utils [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 62.106(s), this was the 6th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 2024-10-08 07:37:25.105 7 DEBUG manila.utils [None req-fffe12f7-c7aa-42dc-aea1-bed6f918b278 - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 64.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 3 -> when i try to use the maila keyring i get the error *[rook@cmp-2 ceph]$ ceph --key AQBip/9mCMf4N/jFNKFlwl/hA4j64esWuA== osd ls[errno 13] RADOS permission denied (error connecting to the cluster)*this is what i get after running that not sure what permissions any idea? -> i also tried with admin rights to the auth *[rook@cmp-2 ceph]$ ceph auth get client.manila[client.manila] key = AQBip/9mCMf4N/jFNKFlwl/hA4j64esWuA== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"exported keyring for client.manila * 4-> my manila conf looks like conf: manila: DEFAULT: debug: true enabled_share_protocols: NFS,CIFS,CEPHFS enabled_share_backends: cephfsnative1 cephfsnative1: driver_handles_share_servers: False share_backend_name: CEPHFSNATIVE1 share_driver: manila.share.drivers.cephfs.driver.CephFSDriver cephfs_conf_path: /etc/ceph/ceph.conf cephfs_protocol_helper_type: CEPHFS cephfs_auth_id: manila cephfs_cluster_name: rook-ceph cephfs_filesystem_name: myfs any idea what could be wrong? Hope to hear from you soon Thank you!

1 0