September 2023 - openstack-discuss

[openstack-ansible] Meeting on 12.09.2023 is cancelled
by Dmitriy Rabotyagov 12 Sep '23

12 Sep '23

Hi everyone, Due to the absence of multiple core members of the group during this week, it has been decided to cancel our regular community meeting on 12.09.2023.

1 0

[Diversity] Diversity and Inclusion WG Meeting reminder
by Amy Marrich 11 Sep '23

11 Sep '23

This is a reminder that the Diversity and Inclusion WG will be meeting tomorrow at 14:00 UTC in the #openinfra-diversity channel on OFTC. We hope members of all OpenInfra projects join us as we look at the progress on responses to the Foundation-wide diversity survey. Thanks, Amy (spotz) 0 - https://etherpad.opendev.org/p/diversity-wg-agenda

1 0

Re: [openstack-ansible] Dedicated gateway hosts not working with OVN
by Dmitriy Rabotyagov 11 Sep '23

11 Sep '23

Hey, 1. Sorry, my bad, was copying you from my phone, so the extra section ( container_skel) that is required has slipped my paste. So /etc/openstack_deploy/env.d/nova.yml should look like this: container_skel: nova_compute_container: belongs_to: - compute_containers - kvm-compute_containers - qemu-compute_containers contains: - neutron_sriov_nic_agent - neutron_ovn_controller - nova_compute properties: is_metal: true 2. Now I actually see more issues in defined openstack_user_config. I'm not sure if that is the reason of the error or not, but it still must be adjusted: a) replace network_hosts with network-infra_hosts. Defining network_hosts also adds infra servers to neutron_l3_agent (and other agents) which has in fact no effect, but triggers a bug, where run_once is treated wrongly. But this will cause failure down the line and I assume that's not it yet. You might need to clean up inventory as a result. b) also define network-northd_hosts - this usually is usually set to infra nodes, and spawns inside LXC. I would also suggest to check out doc on OVN configuration: https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html c) For the issue itself. Most likely, it is looking for `container_bridge` or `host_bind_override` key for some network. As one of these keys are expected in order to create a mapping and ovs bridges for you. It does combine net_name and one of these keys. So it would be interesting to see adjusted openstack_user_config once the above issues are sorted out. I can also suggest defining mappings in neutron_provider_networks directly, like mentioned in the documentation above.

4 12

[publiccloud-sig] Reminder - next meeting September 13th - 0700 UTC
by Felix Kronlage-Dammers 11 Sep '23

11 Sep '23

Hi everyone, this Wednesday the next meeting of the public cloud sig is going to happen. As announced a few weeks ago, we want to try out to have a videocall format every other meeting of the public cloud SIG, so this time we will meet in a video call. Part of the meeting will be a lightning talk. For the first lightning talk Artem (gtema@) will talk on his work on “OpenAPI support for OpenStack and making a new faster CLI on Rust”. (There is a detailled mail about this work that Artem sent to this list End of August[1]). Thanks a lot Artem for giving the first lightning talk in the new format. We will meet at 0700 UTC here: https://conf.scs.koeln:8443/OIF-public-cloud-sig In parallel we will be on irc on #openstack-operators as well of course. Am very much looking forward to wednesday morning! :) felix [1] <url: https://lists.openstack.org/pipermail/openstack-discuss/2023-August/034934.…> -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[ptl][tc][winstackers] Final call for Winstackers PTL and maintainers
by James Page 11 Sep '23

11 Sep '23

Hi All As announced by Lucian last November (see [0]) Cloudbase Solutions are no longer in a position to maintain support for running OpenStack on Windows and have also ceased operation of their 3rd party CI for the windows support across a number of OpenStack projects. This situation has resulted in the Winstackers project becoming PTL-less for the 2023.2 cycle with no volunteers responding to the TC's call to fill this role and take this feature in OpenStack forward (see [1]). This is the final call for any maintainers to step forward if this feature is important to them in OpenStack. The last user survey in 2022 indicated that 2% of respondents were running on Hyper-V so this might be important enough to warrant a commitment from someone operating OpenStack on Windows to maintain these features going forward. Here is a reminder from Lucian's original email on the full list of projects which are impacted in some way: * nova hyper-v driver - in-tree plus out-of-tree compute-hyperv driver * os-win - common Windows library for Openstack * neutron hyperv ml2 plugin and agent * ovs on Windows and neutron ovs agent support * cinder drivers - SMB and Windows iSCSI * os-brick Windows connectors - iSCSI, FC, SMB, RBD * ceilometer Windows poller * manila Windows driver * glance Windows support * freerdp gateway The lack of 3rd party CI for testing all of this really needs to be addressed as well. If no maintainers are forthcoming between now and the next PTG in June the TC will need to officially retire the project and start the process of removing support for Windows across the various projects that support this operating system in some way - either directly or through the use of os-win. For clarity this call refers to the use of the Hyper-V virtualisation driver and associated Windows server components to provide WIndows based OpenStack Hypervisors and does not relate to the ability to run Windows images as guests on OpenStack. Regards James [0] https://lists.openstack.org/pipermail/openstack-discuss/2022-November/03104… [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032888.h…

5 9

[neutron] Bug Deputy Report September 4 - September 10
by Yatin Karel 11 Sep '23

11 Sep '23

Hi Team, Below is the bug deputy report for last week:- High ------ - https://bugs.launchpad.net/neutron/+bug/2033887 [OVN][Trunk] The cold migration process is broken since patch 882581 Fix Proposed: https://review.opendev.org/q/topic:bug%252F2018289 Assigned to Rodolfo - https://bugs.launchpad.net/neutron/+bug/2033932 Add support for OVN MAC_Binding aging Fix Proposed: https://review.opendev.org/c/openstack/neutron/+/893575 Assigned to Terry - https://bugs.launchpad.net/neutron/+bug/2034522 Fake members operating_status ONLINE Fix Proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/893839 Assigned to Fernando Medium ---------- - https://bugs.launchpad.net/neutron/+bug/2034589 [FT][OVN] "ovsdb_connection.stop()" failing during the test cleanup process Unassigned Undecided -------------- - https://bugs.launchpad.net/neutron/+bug/2034761 test discovery fail with latest neutron-lib Marked as Invalid as it was due to version mismatch of neutron-lib and neutron - https://bugs.launchpad.net/neutron/+bug/2034684 UEFI (edk2/ovmf) network boot with OVN fail because no DHCP release reply Looks like issue only against Core OVN, asked reporter for it Thanks and Regards Yatin Karel

1 0

[Skyline] error username or password incorrect
by Satish Patel 10 Sep '23

10 Sep '23

Folks, Try to install skyline UI to replace horizon using doc: https://docs.openstack.org/skyline-apiserver/latest/install/docker-install-… Everything went well and I got a login page on http://x.x.x.x:9999 also it pulled Region/Domains. When I am trying to login with my account, I get an error: Username or Password is incorrect. I am using sqlite DB for skyline as per documents. No errors in logs command $ docker logs skyline When I use Chrome Developer Tools then it was indicating an error in these URLs. http://openstack.example.com:9999/api/openstack/skyline/api/v1/profile http://openstack.example.com:9999/api/openstack/skyline/api/v1/policies 401 Unauthorized ( {"detail":"no such table: revoked_token"} )

2 16

No valid host was found. There are not enough hosts available
by - 10 Sep '23

10 Sep '23

I added a new node to the OpenStack cluster and when creating a new instance I have this error. Maybe someone can tell me what else I can check? No valid host was found. There are not enough hosts available. No valid host was found. There are not enough hosts available. Kod 500 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1548, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 908, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Utworzono

2 1

Re: [openstack][neutron][openvswitch] Openvswitch Packet loss when high throughput (pps)
by Ha Noi 10 Sep '23

10 Sep '23

Hi Satish, Actually, the guess interface is not using tap anymore. <interface type='vhostuser'> <mac address='fa:16:3e:76:77:dd'/> <source type='unix' path='/var/run/openvswitch/vhu3766ee8a-86' mode='server'/> <target dev='vhu3766ee8a-86'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> It's totally bypass the kernel stack ? On Fri, Sep 8, 2023 at 5:02 AM Satish Patel <satish.txt(a)gmail.com> wrote: > I did test OVS-DPDK and it helps offload the packet process on compute > nodes, But what about VMs it will still use a tap interface to attach from > compute to vm and bottleneck will be in vm. I strongly believe that we have > to run DPDK based guest to pass through the kernel stack. > > I love to hear from other people if I am missing something here. > > On Thu, Sep 7, 2023 at 5:27 PM Ha Noi <hanoi952022(a)gmail.com> wrote: > >> Oh. I heard from someone on the reddit said that Ovs-dpdk is transparent >> with user? >> >> So It’s not correct? >> >> On Thu, 7 Sep 2023 at 22:13 Satish Patel <satish.txt(a)gmail.com> wrote: >> >>> Because DPDK required DPDK support inside guest VM. It's not >>> suitable for general purpose workload. You need your guest VM network to >>> support DPDK to get 100% throughput. >>> >>> On Thu, Sep 7, 2023 at 8:06 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>> >>>> Hi Satish, >>>> >>>> Why dont you use DPDK? >>>> >>>> Thanks >>>> >>>> On Thu, 7 Sep 2023 at 19:03 Satish Patel <satish.txt(a)gmail.com> wrote: >>>> >>>>> I totally agreed with Sean on all his points but trust me, I have >>>>> tried everything possible to tune OS, Network stack, multi-queue, NUMA, CPU >>>>> pinning and name it.. but I didn't get any significant improvement. You may >>>>> gain 2 to 5% gain with all those tweek. I am running the entire workload on >>>>> sriov and life is happy except no LACP bonding. >>>>> >>>>> I am very interesting is this project >>>>> https://docs.openvswitch.org/en/latest/intro/install/afxdp/ >>>>> >>>>> On Thu, Sep 7, 2023 at 6:07 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>>>> >>>>>> Dear Smoney, >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 7, 2023 at 12:41 AM <smooney(a)redhat.com> wrote: >>>>>> >>>>>>> On Wed, 2023-09-06 at 11:43 -0400, Satish Patel wrote: >>>>>>> > Damn! We have noticed the same issue around 40k to 55k PPS. Trust >>>>>>> me >>>>>>> > nothing is wrong in your config. This is just a limitation of the >>>>>>> software >>>>>>> > stack and kernel itself. >>>>>>> its partly determined by your cpu frequency. >>>>>>> kernel ovs of yesteryear could handel about 1mpps total on a ~4GHZ >>>>>>> cpu. with per port troughpuyt being lower dependin on what >>>>>>> qos/firewall >>>>>>> rules that were apllied. >>>>>>> >>>>>>> >>>>>> >>>>>> My CPU frequency is 3Ghz and using CPU Intel Gold 2nd generation. I >>>>>> think the problem is tuning in the compute node inside. But I cannot find >>>>>> any guide or best practices for it. >>>>>> >>>>>> >>>>>> >>>>>>> moving form iptables firewall to ovs firewall can help to some degree >>>>>>> but your partly trading connection setup time for statead state >>>>>>> troughput >>>>>>> with the overhead of the connection tracker in ovs. >>>>>>> >>>>>>> using stateless security groups can help >>>>>>> >>>>>>> we also recently fixed a regression cause by changes in newer >>>>>>> versions of ovs. >>>>>>> this was notable in goign form rhel 8 to rhel 9 where litrally it >>>>>>> reduced >>>>>>> small packet performce to 1/10th and jumboframes to about 1/2 >>>>>>> on master we have a config option that will set the default qos on a >>>>>>> port to linux-noop >>>>>>> >>>>>>> https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L106-L1… >>>>>>> >>>>>>> the backports are propsoed upstream >>>>>>> https://review.opendev.org/q/Id9ef7074634a0f23d67a4401fa8fca363b51bb43 >>>>>>> and we have backported this downstream to adress that performance >>>>>>> regression. >>>>>>> the upstram backport is semi stalled just ebcasue we wanted to >>>>>>> disucss if we shoudl make ti opt in >>>>>>> by default upstream while backporting but it might be helpful for >>>>>>> you if this is related to yoru current >>>>>>> issues. >>>>>>> >>>>>>> 40-55 kpps is kind of low for kernel ovs but if you have a low >>>>>>> clockrate cpu, hybrid_plug + incorrect qos >>>>>>> then i could see you hitting such a bottelneck. >>>>>>> >>>>>>> one workaround by the way without the os-vif workaround backported >>>>>>> is to set >>>>>>> /proc/sys/net/core/default_qdisc to not apply any qos or a low >>>>>>> overhead qos type >>>>>>> i.e. sudo sysctl -w net.core.default_qdisc=pfifo_fast >>>>>>> >>>>>>> >>>>>> >>>>>>> that may or may not help but i would ensure that your are not usign >>>>>>> somting like fqdel or cake >>>>>>> for net.core.default_qdisc and if you are try changing it to >>>>>>> pfifo_fast and see if that helps. >>>>>>> >>>>>>> there isnet much you can do about the cpu clock rate but ^ is >>>>>>> somethign you can try for free >>>>>>> note it wont actully take effect on an exsitng vm if you jsut change >>>>>>> the default but you can use >>>>>>> tc to also chagne the qdisk for testing. hard rebooting the vm >>>>>>> shoudl also make the default take effect. >>>>>>> >>>>>>> the only other advice i can give assuming kernel ovs is the only >>>>>>> option you have is >>>>>>> >>>>>>> to look at >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rx… >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.tx… >>>>>>> and >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/extra-specs.html#hw:vi… >>>>>>> >>>>>>> if the bottelneck is actully in qemu or the guest kernel rather then >>>>>>> ovs adjusting the rx/tx queue size and >>>>>>> using multi queue can help. it will have no effect if ovs is the >>>>>>> bottel neck. >>>>>>> >>>>>>> >>>>>>> >>>>>> I have set this option to 1024, and enable multiqueue as well. But it >>>>>> did not help. >>>>>> >>>>>> >>>>>>> > >>>>>>> > On Wed, Sep 6, 2023 at 9:21 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > > Hi Satish, >>>>>>> > > >>>>>>> > > Actually, our customer get this issue when the tx/rx above only >>>>>>> 40k pps. >>>>>>> > > So what is the threshold of this throughput for OvS? >>>>>>> > > >>>>>>> > > >>>>>>> > > Thanks and regards >>>>>>> > > >>>>>>> > > On Wed, 6 Sep 2023 at 20:19 Satish Patel <satish.txt(a)gmail.com> >>>>>>> wrote: >>>>>>> > > >>>>>>> > > > Hi, >>>>>>> > > > >>>>>>> > > > This is normal because OVS or LinuxBridge wire up VMs using >>>>>>> TAP interface >>>>>>> > > > which runs on kernel space and that drives higher interrupt >>>>>>> and that makes >>>>>>> > > > the kernel so busy working on handling packets. Standard >>>>>>> OVS/LinuxBridge >>>>>>> > > > are not meant for higher PPS. >>>>>>> > > > >>>>>>> > > > If you want to handle higher PPS then look for DPDK or SRIOV >>>>>>> deployment. >>>>>>> > > > ( We are running everything in SRIOV because of high PPS >>>>>>> requirement) >>>>>>> > > > >>>>>>> > > > On Tue, Sep 5, 2023 at 11:11 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > > > >>>>>>> > > > > Hi everyone, >>>>>>> > > > > >>>>>>> > > > > I'm using Openstack Train and Openvswitch for ML2 driver and >>>>>>> GRE for >>>>>>> > > > > tunnel type. I tested our network performance between two >>>>>>> VMs and suffer >>>>>>> > > > > packet loss as below. >>>>>>> > > > > >>>>>>> > > > > VM1: IP: 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > VM2: IP: 10.20.1.154 <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > VM3: IP: 10.20.1.72 >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using iperf3 to testing performance between VM1 and VM2. >>>>>>> > > > > >>>>>>> > > > > Run iperf3 client and server on both VMs. >>>>>>> > > > > >>>>>>> > > > > On VM2: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > On VM1: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.154 >>>>>>> > > > > <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using VM3 ping into VM1, then the packet is lost and the >>>>>>> latency is >>>>>>> > > > > quite high. >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > ping -i 0.1 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > PING 10.20.1.206 (10.20.1.206) 56(84) bytes of data. >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=1 ttl=64 time=7.70 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=2 ttl=64 time=6.90 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=3 ttl=64 time=7.71 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=4 ttl=64 time=7.98 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=6 ttl=64 time=8.58 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=7 ttl=64 time=8.34 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=8 ttl=64 time=8.09 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=10 ttl=64 time=4.57 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=11 ttl=64 time=8.74 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=12 ttl=64 time=9.37 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=14 ttl=64 time=9.59 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=15 ttl=64 time=7.97 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=16 ttl=64 time=8.72 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=17 ttl=64 time=9.23 ms >>>>>>> > > > > >>>>>>> > > > > ^C >>>>>>> > > > > >>>>>>> > > > > --- 10.20.1.206 ping statistics --- >>>>>>> > > > > >>>>>>> > > > > 34 packets transmitted, 28 received, 17.6471% packet loss, >>>>>>> time 3328ms >>>>>>> > > > > >>>>>>> > > > > rtt min/avg/max/mdev = 1.396/6.266/9.590/2.805 ms >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Does any one get this issue ? >>>>>>> > > > > >>>>>>> > > > > Please help me. Thanks >>>>>>> > > > > >>>>>>> > > > >>>>>>> >>>>>>>

3 5

[release] Release countdown for week R-3, Sep 11-15
by Herve Beraud 08 Sep '23

08 Sep '23

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, September 14. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/2023.2 branch will be cut. This branch will track the 2023.2 Bobcat release. Once stable/2023.2 has been created, the master branch will be ready to switch to 2024.1 Caracal development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing 2023.2 Bobcat plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/2023.2 branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/2023.2 branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be 2024.1 Bobcat) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any 2023.2 Bobcat release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: September 14 (R-3 week) Final RC deadline: September 28 (R-1 week) Final 2023.2 Bobcat release: October 4 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0