September 2023 - openstack-discuss

[Diversity] Diversity and Inclusion WG Meeting reminder
by Amy Marrich 11 Sep '23

11 Sep '23

This is a reminder that the Diversity and Inclusion WG will be meeting tomorrow at 14:00 UTC in the #openinfra-diversity channel on OFTC. We hope members of all OpenInfra projects join us as we look at the progress on responses to the Foundation-wide diversity survey. Thanks, Amy (spotz) 0 - https://etherpad.opendev.org/p/diversity-wg-agenda

1 0

Re: [openstack-ansible] Dedicated gateway hosts not working with OVN
by Dmitriy Rabotyagov 11 Sep '23

11 Sep '23

Hey, 1. Sorry, my bad, was copying you from my phone, so the extra section ( container_skel) that is required has slipped my paste. So /etc/openstack_deploy/env.d/nova.yml should look like this: container_skel: nova_compute_container: belongs_to: - compute_containers - kvm-compute_containers - qemu-compute_containers contains: - neutron_sriov_nic_agent - neutron_ovn_controller - nova_compute properties: is_metal: true 2. Now I actually see more issues in defined openstack_user_config. I'm not sure if that is the reason of the error or not, but it still must be adjusted: a) replace network_hosts with network-infra_hosts. Defining network_hosts also adds infra servers to neutron_l3_agent (and other agents) which has in fact no effect, but triggers a bug, where run_once is treated wrongly. But this will cause failure down the line and I assume that's not it yet. You might need to clean up inventory as a result. b) also define network-northd_hosts - this usually is usually set to infra nodes, and spawns inside LXC. I would also suggest to check out doc on OVN configuration: https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html c) For the issue itself. Most likely, it is looking for `container_bridge` or `host_bind_override` key for some network. As one of these keys are expected in order to create a mapping and ovs bridges for you. It does combine net_name and one of these keys. So it would be interesting to see adjusted openstack_user_config once the above issues are sorted out. I can also suggest defining mappings in neutron_provider_networks directly, like mentioned in the documentation above.

4 12

[publiccloud-sig] Reminder - next meeting September 13th - 0700 UTC
by Felix Kronlage-Dammers 11 Sep '23

11 Sep '23

Hi everyone, this Wednesday the next meeting of the public cloud sig is going to happen. As announced a few weeks ago, we want to try out to have a videocall format every other meeting of the public cloud SIG, so this time we will meet in a video call. Part of the meeting will be a lightning talk. For the first lightning talk Artem (gtema@) will talk on his work on “OpenAPI support for OpenStack and making a new faster CLI on Rust”. (There is a detailled mail about this work that Artem sent to this list End of August[1]). Thanks a lot Artem for giving the first lightning talk in the new format. We will meet at 0700 UTC here: https://conf.scs.koeln:8443/OIF-public-cloud-sig In parallel we will be on irc on #openstack-operators as well of course. Am very much looking forward to wednesday morning! :) felix [1] <url: https://lists.openstack.org/pipermail/openstack-discuss/2023-August/034934.…> -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[ptl][tc][winstackers] Final call for Winstackers PTL and maintainers
by James Page 11 Sep '23

11 Sep '23

Hi All As announced by Lucian last November (see [0]) Cloudbase Solutions are no longer in a position to maintain support for running OpenStack on Windows and have also ceased operation of their 3rd party CI for the windows support across a number of OpenStack projects. This situation has resulted in the Winstackers project becoming PTL-less for the 2023.2 cycle with no volunteers responding to the TC's call to fill this role and take this feature in OpenStack forward (see [1]). This is the final call for any maintainers to step forward if this feature is important to them in OpenStack. The last user survey in 2022 indicated that 2% of respondents were running on Hyper-V so this might be important enough to warrant a commitment from someone operating OpenStack on Windows to maintain these features going forward. Here is a reminder from Lucian's original email on the full list of projects which are impacted in some way: * nova hyper-v driver - in-tree plus out-of-tree compute-hyperv driver * os-win - common Windows library for Openstack * neutron hyperv ml2 plugin and agent * ovs on Windows and neutron ovs agent support * cinder drivers - SMB and Windows iSCSI * os-brick Windows connectors - iSCSI, FC, SMB, RBD * ceilometer Windows poller * manila Windows driver * glance Windows support * freerdp gateway The lack of 3rd party CI for testing all of this really needs to be addressed as well. If no maintainers are forthcoming between now and the next PTG in June the TC will need to officially retire the project and start the process of removing support for Windows across the various projects that support this operating system in some way - either directly or through the use of os-win. For clarity this call refers to the use of the Hyper-V virtualisation driver and associated Windows server components to provide WIndows based OpenStack Hypervisors and does not relate to the ability to run Windows images as guests on OpenStack. Regards James [0] https://lists.openstack.org/pipermail/openstack-discuss/2022-November/03104… [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032888.h…

5 9

[neutron] Bug Deputy Report September 4 - September 10
by Yatin Karel 11 Sep '23

11 Sep '23

Hi Team, Below is the bug deputy report for last week:- High ------ - https://bugs.launchpad.net/neutron/+bug/2033887 [OVN][Trunk] The cold migration process is broken since patch 882581 Fix Proposed: https://review.opendev.org/q/topic:bug%252F2018289 Assigned to Rodolfo - https://bugs.launchpad.net/neutron/+bug/2033932 Add support for OVN MAC_Binding aging Fix Proposed: https://review.opendev.org/c/openstack/neutron/+/893575 Assigned to Terry - https://bugs.launchpad.net/neutron/+bug/2034522 Fake members operating_status ONLINE Fix Proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/893839 Assigned to Fernando Medium ---------- - https://bugs.launchpad.net/neutron/+bug/2034589 [FT][OVN] "ovsdb_connection.stop()" failing during the test cleanup process Unassigned Undecided -------------- - https://bugs.launchpad.net/neutron/+bug/2034761 test discovery fail with latest neutron-lib Marked as Invalid as it was due to version mismatch of neutron-lib and neutron - https://bugs.launchpad.net/neutron/+bug/2034684 UEFI (edk2/ovmf) network boot with OVN fail because no DHCP release reply Looks like issue only against Core OVN, asked reporter for it Thanks and Regards Yatin Karel

1 0

[Skyline] error username or password incorrect
by Satish Patel 10 Sep '23

10 Sep '23

Folks, Try to install skyline UI to replace horizon using doc: https://docs.openstack.org/skyline-apiserver/latest/install/docker-install-… Everything went well and I got a login page on http://x.x.x.x:9999 also it pulled Region/Domains. When I am trying to login with my account, I get an error: Username or Password is incorrect. I am using sqlite DB for skyline as per documents. No errors in logs command $ docker logs skyline When I use Chrome Developer Tools then it was indicating an error in these URLs. http://openstack.example.com:9999/api/openstack/skyline/api/v1/profile http://openstack.example.com:9999/api/openstack/skyline/api/v1/policies 401 Unauthorized ( {"detail":"no such table: revoked_token"} )

2 16

No valid host was found. There are not enough hosts available
by - 10 Sep '23

10 Sep '23

I added a new node to the OpenStack cluster and when creating a new instance I have this error. Maybe someone can tell me what else I can check? No valid host was found. There are not enough hosts available. No valid host was found. There are not enough hosts available. Kod 500 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1548, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 908, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Utworzono

2 1

Re: [openstack][neutron][openvswitch] Openvswitch Packet loss when high throughput (pps)
by Ha Noi 10 Sep '23

10 Sep '23

Hi Satish, Actually, the guess interface is not using tap anymore. <interface type='vhostuser'> <mac address='fa:16:3e:76:77:dd'/> <source type='unix' path='/var/run/openvswitch/vhu3766ee8a-86' mode='server'/> <target dev='vhu3766ee8a-86'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> It's totally bypass the kernel stack ? On Fri, Sep 8, 2023 at 5:02 AM Satish Patel <satish.txt(a)gmail.com> wrote: > I did test OVS-DPDK and it helps offload the packet process on compute > nodes, But what about VMs it will still use a tap interface to attach from > compute to vm and bottleneck will be in vm. I strongly believe that we have > to run DPDK based guest to pass through the kernel stack. > > I love to hear from other people if I am missing something here. > > On Thu, Sep 7, 2023 at 5:27 PM Ha Noi <hanoi952022(a)gmail.com> wrote: > >> Oh. I heard from someone on the reddit said that Ovs-dpdk is transparent >> with user? >> >> So It’s not correct? >> >> On Thu, 7 Sep 2023 at 22:13 Satish Patel <satish.txt(a)gmail.com> wrote: >> >>> Because DPDK required DPDK support inside guest VM. It's not >>> suitable for general purpose workload. You need your guest VM network to >>> support DPDK to get 100% throughput. >>> >>> On Thu, Sep 7, 2023 at 8:06 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>> >>>> Hi Satish, >>>> >>>> Why dont you use DPDK? >>>> >>>> Thanks >>>> >>>> On Thu, 7 Sep 2023 at 19:03 Satish Patel <satish.txt(a)gmail.com> wrote: >>>> >>>>> I totally agreed with Sean on all his points but trust me, I have >>>>> tried everything possible to tune OS, Network stack, multi-queue, NUMA, CPU >>>>> pinning and name it.. but I didn't get any significant improvement. You may >>>>> gain 2 to 5% gain with all those tweek. I am running the entire workload on >>>>> sriov and life is happy except no LACP bonding. >>>>> >>>>> I am very interesting is this project >>>>> https://docs.openvswitch.org/en/latest/intro/install/afxdp/ >>>>> >>>>> On Thu, Sep 7, 2023 at 6:07 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>>>> >>>>>> Dear Smoney, >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 7, 2023 at 12:41 AM <smooney(a)redhat.com> wrote: >>>>>> >>>>>>> On Wed, 2023-09-06 at 11:43 -0400, Satish Patel wrote: >>>>>>> > Damn! We have noticed the same issue around 40k to 55k PPS. Trust >>>>>>> me >>>>>>> > nothing is wrong in your config. This is just a limitation of the >>>>>>> software >>>>>>> > stack and kernel itself. >>>>>>> its partly determined by your cpu frequency. >>>>>>> kernel ovs of yesteryear could handel about 1mpps total on a ~4GHZ >>>>>>> cpu. with per port troughpuyt being lower dependin on what >>>>>>> qos/firewall >>>>>>> rules that were apllied. >>>>>>> >>>>>>> >>>>>> >>>>>> My CPU frequency is 3Ghz and using CPU Intel Gold 2nd generation. I >>>>>> think the problem is tuning in the compute node inside. But I cannot find >>>>>> any guide or best practices for it. >>>>>> >>>>>> >>>>>> >>>>>>> moving form iptables firewall to ovs firewall can help to some degree >>>>>>> but your partly trading connection setup time for statead state >>>>>>> troughput >>>>>>> with the overhead of the connection tracker in ovs. >>>>>>> >>>>>>> using stateless security groups can help >>>>>>> >>>>>>> we also recently fixed a regression cause by changes in newer >>>>>>> versions of ovs. >>>>>>> this was notable in goign form rhel 8 to rhel 9 where litrally it >>>>>>> reduced >>>>>>> small packet performce to 1/10th and jumboframes to about 1/2 >>>>>>> on master we have a config option that will set the default qos on a >>>>>>> port to linux-noop >>>>>>> >>>>>>> https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L106-L1… >>>>>>> >>>>>>> the backports are propsoed upstream >>>>>>> https://review.opendev.org/q/Id9ef7074634a0f23d67a4401fa8fca363b51bb43 >>>>>>> and we have backported this downstream to adress that performance >>>>>>> regression. >>>>>>> the upstram backport is semi stalled just ebcasue we wanted to >>>>>>> disucss if we shoudl make ti opt in >>>>>>> by default upstream while backporting but it might be helpful for >>>>>>> you if this is related to yoru current >>>>>>> issues. >>>>>>> >>>>>>> 40-55 kpps is kind of low for kernel ovs but if you have a low >>>>>>> clockrate cpu, hybrid_plug + incorrect qos >>>>>>> then i could see you hitting such a bottelneck. >>>>>>> >>>>>>> one workaround by the way without the os-vif workaround backported >>>>>>> is to set >>>>>>> /proc/sys/net/core/default_qdisc to not apply any qos or a low >>>>>>> overhead qos type >>>>>>> i.e. sudo sysctl -w net.core.default_qdisc=pfifo_fast >>>>>>> >>>>>>> >>>>>> >>>>>>> that may or may not help but i would ensure that your are not usign >>>>>>> somting like fqdel or cake >>>>>>> for net.core.default_qdisc and if you are try changing it to >>>>>>> pfifo_fast and see if that helps. >>>>>>> >>>>>>> there isnet much you can do about the cpu clock rate but ^ is >>>>>>> somethign you can try for free >>>>>>> note it wont actully take effect on an exsitng vm if you jsut change >>>>>>> the default but you can use >>>>>>> tc to also chagne the qdisk for testing. hard rebooting the vm >>>>>>> shoudl also make the default take effect. >>>>>>> >>>>>>> the only other advice i can give assuming kernel ovs is the only >>>>>>> option you have is >>>>>>> >>>>>>> to look at >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rx… >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.tx… >>>>>>> and >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/extra-specs.html#hw:vi… >>>>>>> >>>>>>> if the bottelneck is actully in qemu or the guest kernel rather then >>>>>>> ovs adjusting the rx/tx queue size and >>>>>>> using multi queue can help. it will have no effect if ovs is the >>>>>>> bottel neck. >>>>>>> >>>>>>> >>>>>>> >>>>>> I have set this option to 1024, and enable multiqueue as well. But it >>>>>> did not help. >>>>>> >>>>>> >>>>>>> > >>>>>>> > On Wed, Sep 6, 2023 at 9:21 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > > Hi Satish, >>>>>>> > > >>>>>>> > > Actually, our customer get this issue when the tx/rx above only >>>>>>> 40k pps. >>>>>>> > > So what is the threshold of this throughput for OvS? >>>>>>> > > >>>>>>> > > >>>>>>> > > Thanks and regards >>>>>>> > > >>>>>>> > > On Wed, 6 Sep 2023 at 20:19 Satish Patel <satish.txt(a)gmail.com> >>>>>>> wrote: >>>>>>> > > >>>>>>> > > > Hi, >>>>>>> > > > >>>>>>> > > > This is normal because OVS or LinuxBridge wire up VMs using >>>>>>> TAP interface >>>>>>> > > > which runs on kernel space and that drives higher interrupt >>>>>>> and that makes >>>>>>> > > > the kernel so busy working on handling packets. Standard >>>>>>> OVS/LinuxBridge >>>>>>> > > > are not meant for higher PPS. >>>>>>> > > > >>>>>>> > > > If you want to handle higher PPS then look for DPDK or SRIOV >>>>>>> deployment. >>>>>>> > > > ( We are running everything in SRIOV because of high PPS >>>>>>> requirement) >>>>>>> > > > >>>>>>> > > > On Tue, Sep 5, 2023 at 11:11 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > > > >>>>>>> > > > > Hi everyone, >>>>>>> > > > > >>>>>>> > > > > I'm using Openstack Train and Openvswitch for ML2 driver and >>>>>>> GRE for >>>>>>> > > > > tunnel type. I tested our network performance between two >>>>>>> VMs and suffer >>>>>>> > > > > packet loss as below. >>>>>>> > > > > >>>>>>> > > > > VM1: IP: 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > VM2: IP: 10.20.1.154 <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > VM3: IP: 10.20.1.72 >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using iperf3 to testing performance between VM1 and VM2. >>>>>>> > > > > >>>>>>> > > > > Run iperf3 client and server on both VMs. >>>>>>> > > > > >>>>>>> > > > > On VM2: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > On VM1: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.154 >>>>>>> > > > > <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using VM3 ping into VM1, then the packet is lost and the >>>>>>> latency is >>>>>>> > > > > quite high. >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > ping -i 0.1 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > PING 10.20.1.206 (10.20.1.206) 56(84) bytes of data. >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=1 ttl=64 time=7.70 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=2 ttl=64 time=6.90 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=3 ttl=64 time=7.71 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=4 ttl=64 time=7.98 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=6 ttl=64 time=8.58 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=7 ttl=64 time=8.34 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=8 ttl=64 time=8.09 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=10 ttl=64 time=4.57 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=11 ttl=64 time=8.74 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=12 ttl=64 time=9.37 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=14 ttl=64 time=9.59 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=15 ttl=64 time=7.97 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=16 ttl=64 time=8.72 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=17 ttl=64 time=9.23 ms >>>>>>> > > > > >>>>>>> > > > > ^C >>>>>>> > > > > >>>>>>> > > > > --- 10.20.1.206 ping statistics --- >>>>>>> > > > > >>>>>>> > > > > 34 packets transmitted, 28 received, 17.6471% packet loss, >>>>>>> time 3328ms >>>>>>> > > > > >>>>>>> > > > > rtt min/avg/max/mdev = 1.396/6.266/9.590/2.805 ms >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Does any one get this issue ? >>>>>>> > > > > >>>>>>> > > > > Please help me. Thanks >>>>>>> > > > > >>>>>>> > > > >>>>>>> >>>>>>>

3 5

[release] Release countdown for week R-3, Sep 11-15
by Herve Beraud 08 Sep '23

08 Sep '23

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, September 14. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/2023.2 branch will be cut. This branch will track the 2023.2 Bobcat release. Once stable/2023.2 has been created, the master branch will be ready to switch to 2024.1 Caracal development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing 2023.2 Bobcat plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/2023.2 branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/2023.2 branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be 2024.1 Bobcat) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any 2023.2 Bobcat release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: September 14 (R-3 week) Final RC deadline: September 28 (R-1 week) Final 2023.2 Bobcat release: October 4 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 08 Sep '23

08 Sep '23

Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Have a nice weekend. [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[openstack][neutron][openvswitch] Openvswitch Packet loss when high throughput (pps)
by Ha Noi 07 Sep '23

07 Sep '23

Hi everyone, I'm using Openstack Train and Openvswitch for ML2 driver and GRE for tunnel type. I tested our network performance between two VMs and suffer packet loss as below. VM1: IP: 10.20.1.206 VM2: IP: 10.20.1.154 <https://10.20.1.154/24> VM3: IP: 10.20.1.72 Using iperf3 to testing performance between VM1 and VM2. Run iperf3 client and server on both VMs. On VM2: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.206 On VM1: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.154 <https://10.20.1.154/24> Using VM3 ping into VM1, then the packet is lost and the latency is quite high. ping -i 0.1 10.20.1.206 PING 10.20.1.206 (10.20.1.206) 56(84) bytes of data. 64 bytes from 10.20.1.206: icmp_seq=1 ttl=64 time=7.70 ms 64 bytes from 10.20.1.206: icmp_seq=2 ttl=64 time=6.90 ms 64 bytes from 10.20.1.206: icmp_seq=3 ttl=64 time=7.71 ms 64 bytes from 10.20.1.206: icmp_seq=4 ttl=64 time=7.98 ms 64 bytes from 10.20.1.206: icmp_seq=6 ttl=64 time=8.58 ms 64 bytes from 10.20.1.206: icmp_seq=7 ttl=64 time=8.34 ms 64 bytes from 10.20.1.206: icmp_seq=8 ttl=64 time=8.09 ms 64 bytes from 10.20.1.206: icmp_seq=10 ttl=64 time=4.57 ms 64 bytes from 10.20.1.206: icmp_seq=11 ttl=64 time=8.74 ms 64 bytes from 10.20.1.206: icmp_seq=12 ttl=64 time=9.37 ms 64 bytes from 10.20.1.206: icmp_seq=14 ttl=64 time=9.59 ms 64 bytes from 10.20.1.206: icmp_seq=15 ttl=64 time=7.97 ms 64 bytes from 10.20.1.206: icmp_seq=16 ttl=64 time=8.72 ms 64 bytes from 10.20.1.206: icmp_seq=17 ttl=64 time=9.23 ms ^C --- 10.20.1.206 ping statistics --- 34 packets transmitted, 28 received, 17.6471% packet loss, time 3328ms rtt min/avg/max/mdev = 1.396/6.266/9.590/2.805 ms Does any one get this issue ? Please help me. Thanks

3 9

[ops]
by Chris Morgan 07 Sep '23

07 Sep '23

I'm trying to reanimate the openstack ops meetups. I moved the social media account used for notifications from X to mastodon (using the fosstodon instance). Please follow there if interested (link below). I also posted a similar announcement to the openstack group on linkedin. One of the first tasks (besides reforming the team) is finding which communications channels actually work. https://fosstodon.org/@osopsmeetup The old account was here https://twitter.com/osopsmeetup now "sepia-tinted" because it's just historical. Cheers, Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[i18n] New I18n SIG Core
by Ian Y. Choi 07 Sep '23

07 Sep '23

Hello, I would like to happily announce a new i18n core: Seongsoo Cho, considering his notable contributions on I18n with volunteering: 1. PDF Docs contribution with I18n issues (font issues, latex with Asian languages) [1] 2. Zanata to Weblate migration volunteering effort [2] 3. Participation of I18n SIG PTG on March [3] and Forum on May [4] Also, he is active on #openstack-i18n IRC channel (nick name: "seongsoocho"). All, please welcome Seongsoo as I18n-core. I am looking forward to more active contributions from him and interactions with other wonderful I18n members and OpenStack team! Thank you, /Ian [1] https://review.opendev.org/q/topic:bp%252Fbuild-pdf-from-rst-guides [2] https://lists.openstack.org/pipermail/openstack-i18n/2022-October/003568.ht… [3] https://etherpad.opendev.org/p/march2023-ptg-i18n [4] https://etherpad.opendev.org/p/vancouver-2023-i18n-forum

4 3

[Glance] Cancelling Weekly Meeting 7th Sept
by Pranali Deore 07 Sep '23

07 Sep '23

Hello, I won't be able to chair today's meeting, my kid is not well, going to take him to the doctor. so cancelling the meeting for this week. Let's meet next week ! Thanks, Pranali D

1 0

Windows Image failed to access network
by Richard Buregeya 07 Sep '23

07 Sep '23

Hello Team, I created the windows image and added the N/w drivers, but it can't be able to get from DHCP. Any idea? Regards Richard.

2 1

[neutron][release] Proposing transition to EOL Train (Neutron and neutron-lib)
by Rodolfo Alonso Hernandez 07 Sep '23

07 Sep '23

Hello: I'm sending this mail in advance to propose transitioning the Neutron and the neutron-lib Train branch to EOL. This topic was proposed and approved during the last Neutron team meeting [1]. These are the only two Neutron related projects still in EM. The announcement is the first step [2] to transition a stable branch to EOL. The patch to mark these branches as EOL will be pushed in two weeks. If you have any inconvenience, please let me know in this mail chain or in IRC (ralonsoh, #openstack-neutron channel). You can also contact any Neutron core reviewer in the IRC channel. Regards. [2] https://meetings.opendev.org/meetings/networking/2023/networking.2023-09-05… [1] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l…

1 0

[all][elections][ptl][tc] Conbined PTL/TC 2024.1 cycle Election Voting Kickoff
by Ian Y. Choi 06 Sep '23

06 Sep '23

Polls for PTL and TC elections are now open and will remain open for you to cast your vote until Sep 20, 2023 23:45 UTC. We are selecting 4 TC members, and are having PTL elections for OpenStack_Helm. Please rank all candidates in your order of preference. You are eligible to vote in the TC election if you are a Foundation individual member[0] that also has committed to any official project team's deliverable repositories[1] over the Sep 16, 2022 00:00 UTC - Aug 30, 2023 00:00 UTC timeframe (2023.1 to 2023.2) or if you are in the list of extra-atcs[2] for any official project team. You are eligible to vote in a PTL election if you are a Foundation individual member[0] and had a commit in one of that team's deliverable repositories[1] over the Sep 16, 2022 00:00 UTC - Aug 30, 2023 00:00 UTC timeframe (2023.1 to 2023.2) or if you are in that team's list of extra-atcs[2]. If you are eligible to vote in an election, you should find your email with a link to the Condorcet Internet Voting Service (CIVS) page to cast your vote in the inbox of your Gerrit preferred email[3]. What to do if you don't see the email and have a commit in at least one of the projects having an election: * due to a new CIVS policy, to get the email from poll supervisors, you must opt in to email communication opt in with your Gerrit email address at: https://civs1.civs.us/cgi-bin/opt_in.pl * eligible voters who opt in to email communication late can see a pending poll invitation and vote until Sep 20, 2023 23:45 UTC. * check the trash or spam folders of your Gerrit Preferred Email address, in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from the project's deliverable repos[0] and email the election officials[4]. If we can confirm that you are entitled to vote, we will add you to the voters list for the appropriate election. Our democratic process is important to the health of OpenStack, please exercise your right to vote! Candidate statements/platforms can be found linked to Candidate names on this page: https://governance.openstack.org/election/ Happy voting, [0] https://www.openstack.org/community/members/ [1] The list of the repositories eligible for electoral status: https://opendev.org/openstack/governance/src/tag/0.15.0/reference/projects.… [2] Look for the extra-atcs element in [1] [3] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your preferred email. That is where the ballot has been sent. [4] https://governance.openstack.org/election/#election-officials

1 0

[manila] Cancelling Sep 7 weekly meeting
by Carlos Silva 06 Sep '23

06 Sep '23

Hello, Zorillas! As discussed in the previous meeting, we are cancelling tomorrow's weekly meeting, as a big part of the audience is on PTO tomorrow/this week. The next Manila weekly meeting will be on Sep 14th. If you have something urgent to bring up, please let me know on the #openstack-manila IRC channel. Regards, carloss

1 0

[largescale-sig] Next meeting: Sept 6, 15utc
by Thierry Carrez 06 Sep '23

06 Sep '23

Hi everyone, The Large Scale SIG is back after the northern hemisphere summer break and will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC, our EU+US-friendly time. I will be chairing. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230906T15 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[kolla-ansible][octavia] octavia management network setup for vlan provider
by Satish Patel 06 Sep '23

06 Sep '23

Folks, I have setup kolla-ansible and configured octavia using the o-hm0 interface with the tenant and it works. For production I think I should use VLAN based provider for octavia management network so this is what I did I have created a bond0.41 dedicated interface on all 3 controller nodes and created vlan 41 on all network switches. This is what my global.yml looks like ## Octivia enable_octavia: "yes" octavia_network_interface: "bond0.41" octavia_amp_flavor: name: "amphora" is_public: no vcpus: 2 ram: 2048 disk: 5 octavia_amp_network: name: lb-mgmt-net provider_network_type: vlan provider_segmentation_id: 41 provider_physical_network: physnet1 external: false shared: false subnet: name: lb-mgmt-subnet cidr: "192.168.41.0/24" allocation_pool_start: "192.168.41.100" allocation_pool_end: "192.168.41.200" enable_dhcp: yes After running the playbook all get setup as per document. When I create loadbalancer it just get stuck in PENDING status. [1] Document saying make sure your octavia_network_interface is connected to openvswitch. Do I need to connect manually or will kolla-ansible do that for me? If I am going to do that then on which bridge I should attach br-ex or br-int ? [1] https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavi…

3 3

Cinder Bug Report 2023-09-06
by Jon Bernard 06 Sep '23

06 Sep '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - Weak Cryptographic Algorithm (MD5) used <https://bugs.launchpad.net/cinder/+bug/2033612> - Status: New - group list doesn't parse 'all_tenants' parameter correctly <https://bugs.launchpad.net/cinder/+bug/2034248> - Status: In Progress - Documentation check and correction for PowerStore NFS driver <https://bugs.launchpad.net/cinder/+bug/2034482> - Status: New Thanks -- Jon

1 0

[all][elections][ptl][tc] Opt in to CIVS voting system
by Ian Y. Choi 05 Sep '23

05 Sep '23

The election uses the Condorcet Internet Voting Service (CIVS). Due to CIVS policy, to vote in private CIVS polls, you must opt in to email communication. To opt in, please enter your Gerrit email address in the following page before the start of the election Sep 06, 2023 23:45 UTC, and confirm with the code that will sent to you via email. https://civs1.civs.us/cgi-bin/opt_in.pl If you have any question, please contact the election officials. https://governance.openstack.org/election/#election-officials

1 0

[election] [tc] Running for a second year on the TC
by Jay Faulkner 05 Sep '23

05 Sep '23

Hey all, Around a year ago, I announced my candidacy for the TC after a call for more contributors from a diverse employer base was made. I think a year later, it's more obvious than ever how important it is to have a diverse, independent technical committee. We have made huge progress on this front. We have six candidates for the four open TC positions, representing different independent companies utilizing OpenStack. This is a major improvement year over year. If you've been in the community a while, you've probably worked with me and have an idea if you want to vote for me or not in this position. Whether elected or not, I'll continue my vocal support of OpenStack technologies in private meetings and on social media, I'll continue to invite people I meet interested in cloud to join our community -- not because it's my job (it is), but also because I'm proud of what we've built. When I have those conversations with potential users or contributors, I highlight two things I admire about the OpenStack community: longevity -- there are not many large, distributed open source projects that have survived as long as we have, and the four opens -- putting openness out front as our face to the world is why I'll continue to be proud to represent OpenStack in the future -- as a TC member or not (you decide). If someone does want to talk about a particular policy or thought process of mine, I'm happy to respond to comments here or via private email/IRC. Thanks, Jay Faulkner

1 0

[QA][release] Cycle With Intermediary without recent deliverables
by Herve Beraud 05 Sep '23

05 Sep '23

Quick reminder that for deliverables following the cycle-with-intermediary model, the release team will use the latest $series release available on release week. The following deliverables have done a 2023.2 bobcat release, but it was not refreshed in the last two months: tempest You should consider making a new one very soon, so that we don't use an outdated version for the final release. Thanks -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

2 1

Host aggregates and placement aggregates
by Gk Gk 05 Sep '23

05 Sep '23

Hi All, We have a yoga OSA setup. We have upgraded the setup from Xena to Yoga. We have found that some of the host aggregates have not synced to the placement aggregates. As a result , the vm creation on those aggregate nodes is failing even though they have the full capacity. I have tried the placement sync aggregates command but it did not help. Is this a known bug in Yoga ? Thanks Y.G

1 0

[openstack-ansible] Recommended branch for a production environment?
by Roger Rivera 05 Sep '23

05 Sep '23

Hello, We have deployed Openstack-Ansible in a test environment and we were wondering what the recommended repository branch is to deploy in a production environment that will be integrated with a separate Ceph cluster. 1. We are thinking about pinning to the stable/zed. Is that recommended over the master branch? 2. Are bugs ironed out on both master and stable/zed with the same cadence? 3. Additionally, is Debian 11 a better alternative than Ubuntu 22 for target hosts? We noticed Ubuntu 22.04 support was added recently, whereas Debian 11 has been supported for quite some time now which leads us to believe its stability/integration could be more mature at this point? Any suggestions will be appreciated. Thank you, -- *Roger *

3 7

What is the difference between 4 and 8 virtual sockets to physical sockets?
by Jorge Visentini 05 Sep '23

05 Sep '23

Hello Team, What is the difference between creating an instance with *4* or *8 virtual sockets*, since the hypervisor has only *4 physical sockets*. My question is where do sockets, cores and virtual threads fit into the physical hardware. I think this question is not just related to Openstack, but with any virtualization. My hypervisor configuration is as follows: CPU(s): 192 Online CPU(s) list: 0-191 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 4 NUMA node(s): 4 Do you have any documentation that I can read and understand better? That we have a nice week! -- Att, Jorge Visentini +55 55 98432-9868

2 3

[tc] Technical Committee next weekly meeting Today on September 5
by Nikolla, Kristi 05 Sep '23

05 Sep '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, September 5, 2023 at 1800 UTC on Zoom https://us06web.zoom.us/j/87108541765?pwd=emlXVXg4QUxrUTlLNDZ2TTllWUM3Zz09 The agenda can be found at the link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggesti… and attached below: • Roll call • Follow up on past action items • No action items • Gate health check • OpenStack Elections • #link https://governance.openstack.org/election/ • Ballots go out tomorrow and remain open until September 20. • We have 6 candidates out of 4 seats for TC :) • Open Discussion and Reviews • Register for the PTG • #link https://openinfra.dev/ptg/ • #link https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla

1 0

RE: ***UNCHECKED*** openstack-discuss Digest, Vol 59, Issue 10 | Openstack Web Interface Issue
by Asma Naz Shariq 04 Sep '23

04 Sep '23

Hi Team, I have deployed Openstack through manual installation-Yoga release. I encountered the attached error and can't have access to Openstack dashboard. When I entered the login credentials it displays something went wrong and you don't have permission to use /horizon/. Can anyone have a solution for this error. -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Monday, September 4, 2023 8:27 PM To: openstack-discuss(a)lists.openstack.org Subject: ***UNCHECKED*** openstack-discuss Digest, Vol 59, Issue 10 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss or, via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: Live migration error (Karera Tony) 2. What is the difference between 4 and 8 virtual sockets to physical sockets? (Jorge Visentini) 3. [all][stable][ptl] Propose to EOL Stein series (El?d Ill?s) ---------------------------------------------------------------------- Message: 1 Date: Mon, 4 Sep 2023 14:11:37 +0200 From: Karera Tony <tonykarera(a)gmail.com> To: smooney(a)redhat.com Cc: Danny Webb <Danny.Webb(a)thehutgroup.com>, openstack-discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: Live migration error Message-ID: <CA+69TL3=7GwfgRgAjfxvx5q4aZCstXxtBeB2mGb9ofa0JCZmbA(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello Smooney, I can confirm that it is disabled on all the servers. Regards Tony Karera On Mon, Sep 4, 2023 at 1:25?PM <smooney(a)redhat.com> wrote: > On Mon, 2023-09-04 at 12:59 +0200, Karera Tony wrote: > > Hello Danny, > > > > I am actually using Broadwell-noTSX-IBRS on all the servers but I > > have > seen > > that some flags are not there in the older servers. > > > > The flags below to be specific. > > > > hwp hwp_act_window hwp_pkg_req > so hwp is apprentlly intels pstate contol based on > https://unix.stackexchange.com/a/43540 > > i dont belive this is normally exposed to a guest but if you are > seeign differnce between hosts then that likely means you have > disabled pstates also know as "intel speedstep" on some of the host > and not others. > > > > > Regards > > > > Tony Karera > > > > > > > > > > On Fri, Sep 1, 2023 at 4:05?PM <smooney(a)redhat.com> wrote: > > > > > On Fri, 2023-09-01 at 14:56 +0200, Karera Tony wrote: > > > > Hello Danny, > > > > > > > > Thanks for the feedback. > > > > > > > > use a lowest common denominator cpu_model : Does this mean > > > > changing > the > > > > servers ? > > > > > > it means tha tif you have a mix of cpus in the deployment you > > > shoudl hardcod to the older of the diffent cpu models i.e skylake > > > or whatever it may > be > > > in your case. > > > > > > > > > > https://docs.openstack.org/nova/latest/configuration/config.html#libvi > rt.cpu_models > > > > > > [libvirt] > > > cpu_mode=custom > > > cpu_models=skylake > > > > > > in more recent release we replaced the older cpu_model with > > > cpu_models with is a comma sperated list of models in prefernce > > > order. i.e. nova will use the first > cpu_modle > > > in the list that work > > > on the current host. not that while this give extra flexiblity it > > > vs > just > > > using the oldest cpu model > > > it does limit the set of hosts you can migrate too but means you > > > can > have > > > better perfromance so its > > > a tradeoff. performance vs portablity. > > > > > > the error you mentioned can also be caused by micorocode updates. > > > intel remove TSX and MPX i beilve in the last 2 years form come of > there > > > cpus > > > that breaks live migration if a vm was create with access to that > > > cpu instruction > > > > > > the only way to resolve that is to cold migrate the guest. > > > i.e. if a vm is currently booted with cpu_model X it cannot be > > > modifed while the guest is running so you either need to update > > > the config option and reboot all the vms > or > > > more particlaly update the config > > > and then cold migrate them which will allow you to move the > instnace(your > > > orginal goal) while allowing > > > the new cpu model to take effect. > > > > > > novas default for the cpu model when not set is with our default > cpu_mode > > > is host_model > > > meaning whatever model best matches the current hosts cpu. > > > > > > that is the correct default in general but if you have mixed cpu > > > generatiosn in your cloud its not ideal. > > > > > > hopefuly that helps > > > > > > > > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > On Fri, Sep 1, 2023 at 12:37?PM Danny Webb < > Danny.Webb(a)thehutgroup.com> > > > > wrote: > > > > > > > > > It means that you have CPUs with incompatible flags or you've > > > > > got differing kernel versions that expose different flags or > > > > > you've got differing libvirt versions that expose different > > > > > flags. You either > > > need to > > > > > use a lowest common denominator cpu_model or do a cold migration. > > > > > ------------------------------ > > > > > *From:* Karera Tony <tonykarera(a)gmail.com> > > > > > *Sent:* 01 September 2023 10:13 > > > > > *To:* openstack-discuss > > > > > <openstack-discuss(a)lists.openstack.org> > > > > > *Subject:* Live migration error > > > > > > > > > > > > > > > * CAUTION: This email originates from outside THG * > > > > > ------------------------------ Hello Team, > > > > > > > > > > I am trying to migrate instances from one host to another but > > > > > I am > > > getting > > > > > this error. > > > > > > > > > > *Error: *Failed to live migrate instance to host "compute1". > Details > > > > > <http://10.10.13.31/admin/instances/#message_details> > > > > > Migration pre-check error: Unacceptable CPU info: CPU doesn't > > > > > have compatibility. 0 Refer to > > > > > > http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult > > > > > (HTTP 400) (Request-ID: > > > > > req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da) > > > > > > > > > > Any idea on how to fix this? > > > > > > > > > > Regards > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > >

1 0

Need assistance
by Gk Gk 04 Sep '23

04 Sep '23

Hi, We have Yoga version setup of OSA. When a user is trying to use a certain flavor which has some extra specs, same as that which is mentioned for an aggregate, the vm creation is failing for the user. But I created a vm using that same flavor and it succeeded for a test user account. When I analyzed the logs, the nova scheduler is not filtering or checking the individual hosts from that target aggregate in the case of that user. But it is validating the target aggregate hosts in the case of a test user and able to select a host for vm creation. So what could be the reason that the scheduler is unable to find the hosts from that aggregate in the case of that user as opposed to that of the test user ? Is there any connection between a project and an aggregate ? Thanks Kumar

2 1

[all][qa] Devstack dropping support for Ubuntu Focal
by Ghanshyam Mann 04 Sep '23

04 Sep '23

Hello Everyone, As you know, this release 2023.2 (bobcat) does not mandate testing the ubuntu Focal[1] and with Nova bumped the libvirt version, focal jobs are failing. Devstack was planning to drop the Focal support in the next cycle but we have to do it now only[2]. This email is a heads-up if you are still wondering about the Focal job failure. If your project still running any of the jobs on focal, please plan to move to Jammy. [1] https://governance.openstack.org/tc/reference/runtimes/2023.2.html [2] https://review.opendev.org/c/openstack/devstack/+/885468 -gmann

1 0

[all][stable][ptl] Propose to EOL Stein series
by Elõd Illés 04 Sep '23

04 Sep '23

Hi, Core projects announced and deleted their stable/stein branches for some time already. Furthermore, most of the projects, where stable/stein is still open, have broken gates. So, the same way as we transitioned Rocky series to End of Life [1][2], now is the time to do the same with Stein. I'll propose the stein-eol transition patches for the rest of the open projects. I ask again the teams to signal their decision (with a +1 if the team is ready for the transition). Thanks in advance! (Please note, that the Extended Maintenance process is phasing out / transforming soon, see details in Technical Committee's resolution [3], which was accepted and merged in August.) Thanks, Előd Illés irc: elodilles @ #openstack-stable / #openstack-release [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031922… [2] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033386.h… [3] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc…

1 0

Live migration error
by Karera Tony 04 Sep '23

04 Sep '23

Hello Team, I am trying to migrate instances from one host to another but I am getting this error. *Error: *Failed to live migrate instance to host "compute1". Details <http://10.10.13.31/admin/instances/#message_details> Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da) Any idea on how to fix this? Regards Tony Karera

3 6

[PTL][release] 2023.2 Bobcat Cycle Highlights
by Herve Beraud 04 Sep '23

04 Sep '23

Hi, This is a reminder that Cycle highlights need to be added to deliverable yamls so that they can be included in release marketing preparations. (See the details about how to add them at the project team guide [1]) Note that the deadline was the Feature Freeze date in previous cycles, but due to requests of PTLs it was decided to postpone one week in the future (to R-4 week). This is better for PTLs as they are usually very busy around Feature Freeze, on the other hand gives less time to marketing team to process the highlights. [1] https://docs.openstack.org/project-team-guide/release-management.html# cycle-highlights Thanks, -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[neutron] Bug Deputy Report August 28 - September 3
by Elvira Garcia Ruiz 03 Sep '23

03 Sep '23

Hi! I was the bug deputy last week. Find the summary below: High ------ - https://bugs.launchpad.net/neutron/+bug/2033508 - [HA] "neutron-dynamic-routing" HA unittest failing Unassigned - https://bugs.launchpad.net/neutron/+bug/2033493 - [ndr] Unit tests failing due to a missing patch, still unreleased, in Neutron Fix proposed: https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/893189 Assigned to Rodolfo Alonso Hernandez - https://bugs.launchpad.net/neutron/+bug/2033305 - OVN metadata agent keeps restarting Assigned to Miro Tomaska Fix proposed: https://review.opendev.org/c/openstack/neutron/+/890986 - https://bugs.launchpad.net/neutron/+bug/2033556 - Documentation for DNS integration is incomplete Assigned to Dr. Jens Harbott - https://bugs.launchpad.net/neutron/+bug/2033887 - [OVN][Trunk] The cold migration process is broken since patch 882581 Assigned to Rodolfo Fix proposed: https://review.opendev.org/c/openstack/neutron/+/893447 + backports - https://bugs.launchpad.net/neutron/+bug/2033752 - test_reboot_server_hard fails with AssertionError: time.struct_time() not greater than time.struct_time() This is affecting us but the failure is on Nova. Yatin will monitor this. - https://bugs.launchpad.net/neutron/+bug/2033932 - Add support for OVN MAC_Binding aging Fix proposed: https://review.opendev.org/c/openstack/neutron/+/893575 Assigned to Terry Medium ---------- - https://bugs.launchpad.net/neutron/+bug/2033281 - [OVN] ovn_hash_ring table cleanup In progress. Fix merged on master: https://review.opendev.org/c/openstack/neutron/+/893030 Assigned to Lucas Low ------ - https://bugs.launchpad.net/neutron/+bug/2033651 - [fullstack] Reduce the CI job time Unassigned Incomplete --------------- - https://bugs.launchpad.net/neutron/+bug/2033293 - dns integration saying plugin does not match requirements Unassigned Left a comment since I think this might not be a bug. Also contacted mlavalle to get more information. Undecided --------------- https://bugs.launchpad.net/neutron/+bug/2033683 - openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n'] I’m not very sure about how to confirm this one. Unassigned Kind regards, Elvira

1 0

[SDK][CLI][API][OpenAPI] Machine readable OpenStack API spec - time to do a next step?
by Artem Goncharov 01 Sep '23

01 Sep '23

Hey, Some of you may know that for certain time I was doing experiments on the sdk/cli/ansible_mod side with regards to the code generation. Reason for that is simple: huge codebases and too few resources. While not being the primary goal of this initiative, but still a pointer into the same direction is that most of the tools we need to maintain do the same stuff with some adaptations necessary for the framework in question. I did a lot of diverse attempts trying to take current OpenStackSDK codebase as a “source of truth” regarding what exists in which service and use it to generate code (OSC, AnsibleModule, etc). Pretty quickly it became clear that we do not have enough information to cover all the different needs: - when query parameter has been added into or dropped from the API - what is the type of the query parameter and whether in the case it is list it should be rendered as "?tags=a,b" or "?tags=a&tags=b” - which fields are necessary to be send in the request, which are optional and how to deal when you send name but get back UUID in the same field - what are the parameter/resource fields types in detail with respect to strongly typed programming languages (“int” is not good to differentiate between u8 and f64) - how to determine list of supported “actions” on the resource which are above the plain CRUD Firstly I tried to extend our current model of things in SDK, but even that was not so helpful. And then I decided to give another look at OpenAPI and our desires since multiple years to move into that direction. First of all it provides all the required information to do what is necessary. And so I tried to gather all the things we said about OpenAPI at OpenStack and which challenges we have and compare it against the OpenAPI 3.1. Problems: 1) OpenAPI does not support model of microversions. To be more precise on that you can’t describe different request/response body based on the header differentiator 2) URL + Method are representing unique combination making it impossible to represent different actions based on the content 3) All relevant Headers are supposed to be described in the spec. For example Swift is using X-Object-Meta-* prefix to allow user customisations what is not currently supported by the spec Actually that’s it, everything else was doable. Have I missed something? So what can we do about it: 1) OpenAPI 3.1 is now based upon jsonschema, what allows huge flexibility in describing object schema including inheritance and polymorphism (with composition, discriminator, etc). So we can describe the necessary Microversion Header and schema using “oneOf” whenever necessary. Moreover it is possible to extend type schema with “x-“ prefixed custom properties. So I went and started adding “x-openstack-“ prefixed properties where necessary 2) I personally agree with OpenAPI that URL+method should be enough to uniquely identify what is expected (while not a big deal for the API consumer, it would have been easier on the server side to have URL uniquely tied with the data and operation instead of dynamically looking at the request to figure out what to do. This is actually a most complex thing to deal with and the only solution that comes to my mind would be not to try to put everything into single spec, but rather have a dedicated specs for different operations. It makes it anyway easier to maintain such spec. Maybe at some next OpenAPI iteration this gets resolved and we can adopt changes. 3) A tricky thing, but there is nothing preventing us specifying headers as “x-object-meta-*” and defining custom processors for things like that to for example compose/decompose this into dictionary on the client side OpenAPI now even allows defining spec extensions if necessary, but I think at the moment we can cover all issues without that. Demo time: - server live migration (in my eyes one of the trickiest things I have seen so far from the OpenStack APIs). Here you have an action requiring a dedicated spec, multiple revisions with micro versions and even parameter type bool which can be a string as well: [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ More real-life examples in [4] - image create and upload: [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ Here it is possible to put few methods into the single spec. You can also see here samples of different media-types, capture headers, readOnly/writeOnly properties (You can actually take those specs and paste them into https://editor-next.swagger.io/ and see how it looks like) Closing notes: I have implemented a WIP change [1] in SDK that consumes list of specs and performs requested operation. Series of SDK commands are converted to the new way dealing with APIs. This is backwards compatible, but opens a road to heavily reduce amount of code in SDK (precisely by dropping all those customisations that are necessary now). Honestly speaking my drafts even show performance improvement on SDK side in the area of 100-300ms for diverse API operations, but I am still not really able to explain why. Maybe because of avoiding most black magic of SDK. I assume after completion we could get another performance boost by dropping other unnecessary logic and abstractions. The work is not completed as the change title states. Once we have specs for the commands code generators can be updated to consume this data and produce much better code. BTW, I have now also a prototype of the new CLI written with Rust and this is a hypersonic bullet compared to the current OSC. But please do not push me more on that, it is still in a very early stages and depends heavily on the available specs. I can only say that it is now designed in the way that every API call can have a thin CLI coverage just by providing a spec, when additional logic is desired - surely will require human implementation. Code generators in the pipe: OSC, AnsibleModules, RustSDK (sync/async), RustCLI. Next thing that are on the radar: gopher cloud, terraform modules, async python sdk, JS SDK(?) If all of that gets executed properly and with some community traction we can all have following things covered: - improve standardisation of OpenStack internals and externals: glance and nova (at least those 2) are already using jsonschema internally in different areas to describe requests/responses. Why not to make this standard reaching the service consumers? - getting rid of api-ref work by updating our sphinx machinery to consume our customised specs and produce nice docs matching the reality - sharing specs between teams to improve interface (not like currently we need to read the api-ref with tons of bugs plus source code to understand how to cover new feature in service X). Maybe even a central repo with the specs per release. - there are plenty of code generators and server bindings for OpenAPI specs so that we can potentially align frameworks used by different teams to maintain less - less work for all of us who needs services talking to each other (not immediately right now, but once the code is switched on consuming specs) - request verification already on the client side not waiting for the response - finally show something to customers often annoying asking “where are your openapi specs” (no offence here ;-))? I know it is a long message. But I am pretty excited with the progress and would like to hear community opinions. For the more detailed discussion consider this as a pre-announcement of the topic for PTG in sdk/cli slots. Huge invest but huge outcome P.S. it can result in a good chunk of relatively easy work for students Regards, Artem [1] https://review.opendev.org/c/openstack/openstacksdk/+/892161 [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ [4] https://review.opendev.org/c/openstack/openstacksdk/+/893365

4 5

[release] Release countdown for week R-4, Sep 04-08
by Elõd Illés 01 Sep '23

01 Sep '23

Development Focus ----------------- We just passed feature freeze! Until release branches are cut, you should stop accepting featureful changes to deliverables following the cycle-with-rc release model, or to libraries. Exceptions should be discussed on separate threads on the mailing-list, and feature freeze exceptions approved by the team's PTL. Focus should be on finding and fixing release-critical bugs, so that release candidates and final versions of the 2023.2 Bobcat deliverables can be proposed, well ahead of the final 2023.2 Bobcat release date. General Information ------------------- We are still finishing up processing a few release requests, but the 2023.2 Bobcat release requirements are now frozen. If new library releases are needed to fix release-critical bugs in 2023.2 Bobcat, you must request a Requirements Freeze Exception (RFE) from the requirements team before we can do a new release to avoid having something released in 2023.2 Bobcat that is not actually usable. This is done by posting to the openstack-discuss mailing list with a subject line similar to: [$PROJECT][requirements] RFE requested for $PROJECT_LIB Include justification/reasoning for why a RFE is needed for this lib. If/when the requirements team OKs the post-freeze update, we can then process a new release. A soft String freeze is now in effect, in order to let the I18N team do the translation work in good conditions. In Horizon and the various dashboard plugins, you should stop accepting changes that modify user-visible strings. Exceptions should be discussed on the mailing-list. By September 28th, 2023 this will become a hard string freeze, with no changes in user-visible strings allowed. Actions ------- stable/2023.2 branches should be created soon for all not-already-branched libraries. You should expect 2-3 changes to be proposed for each: a .gitreview update, a reno update (skipped for projects not using reno), and a tox.ini constraints URL update. Please review those in priority so that the branch can be functional ASAP. The Prelude section of reno release notes is rendered as the top level overview for the release. Any important overall messaging for 2023.2 Bobcat changes should be added there to make sure the consumers of your release notes see them. Upcoming Deadlines & Dates -------------------------- RC1 deadline: September 14th, 2023 (R-3 week) Final RC deadline: September 28th, 2023 (R-1 week) Final 2023.2 Bobcat release: October 4th, 2023 2024.1 Caracal Virtual PTG - October 23-27, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[cinder][FFE] Granting FFE to features (8th September)
by Rajat Dhasmana 01 Sep '23

01 Sep '23

Hi All, We had 6 feature requests during the 2023.2 (Bobcat) development cycle[1]. Few of the features had other patch dependencies amounting to significantly more patches. With limited review bandwidth, we were not able to review all the features on time, hence I would like to grant FFE to the following listed features. The reason so many changes are granted FFE are due to the following reasons: * It has gone through iterations of updates and most of them has +2s so it should just require another core to merge * The response time for addressing review comments is fast which is a good thing * The changes are driver specific and isolated to a particular driver that have been tested by the vendor team and third party CI 1. Fujitsu Driver: Add QoS support Patch: https://review.opendev.org/c/openstack/cinder/+/847730 2. NetApp ONTAP: Added support to Active/Active mode in NFS driver Patch: https://review.opendev.org/c/openstack/cinder/+/886869 Dependent patches: https://review.opendev.org/c/openstack/cinder/+/798384 https://review.opendev.org/c/openstack/cinder/+/889826 3. [NetApp] LUN space-allocation support for iSCSI Patch: https://review.opendev.org/c/openstack/cinder/+/893106 4. [Pure Storage] Replication-Enabled and Snapshot Consistency Groups Patch: https://review.opendev.org/c/openstack/cinder/+/891234 5. [HPE XP] Support HA and data deduplication Patch: https://review.opendev.org/c/openstack/cinder/+/892608 Dependent patches: https://review.opendev.org/c/openstack/cinder/+/885986 https://review.opendev.org/c/openstack/cinder/+/879830 https://review.opendev.org/c/openstack/cinder/+/877672 https://review.opendev.org/c/openstack/cinder/+/891937 [1] https://etherpad.opendev.org/p/cinder-2023.2-bobcat-features The deadline after FFE is going to be 8th September, 2023 but we will try to get the changes reviewed and merged at the earliest. Let me know if there are any queries or concerns. Thanks Rajat Dhasmana

1 0

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 01 Sep '23

01 Sep '23

Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Have a nice weekend. [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[cinder] Cinder Midcycle - 2 (R-6) for 2023.2 (Bobcat) Summary
by Rajat Dhasmana 01 Sep '23

01 Sep '23

Hello Argonauts, The summary for mid cycle -2 (R-6) held on 23rd August, 2023 from 1400 to 1600 UTC is available at the wiki page[1]. Please go through the etherpad and the recording for the entire discussion. [1] https://wiki.openstack.org/wiki/CinderBobcatMidCycleSummary Thanks Rajat Dhasmana

1 0

[all] osprofiler requirements on opentelemetry: is this reasonable?
by Thomas Goirand 01 Sep '23

01 Sep '23

Hi, I just saw that the last version of osprofiler requires opentelemetry-exporter-otlp and opentelemetry-sdk. Since osprofiler is used in almost all projects, and that opentelemetry is really HUGE, I wonder if this is all reasonable. This forces downstream distros to do a lot of packaging only for that. Is this avoidable? I started doing the packaging, and then stopped to write this message, seeing how much work that would represent. Your thoughts? Cheers, Thomas Goirand (zigo)

3 3

[Telemetry] Adding Erno Kuvaja (jokke_) to telemetry cores
by Matthias Runge 31 Aug '23

31 Aug '23

Hi there, a bit premature, but in-line with[1] I've added Erno to telemetry cores. Erno is not a stranger in the OpenStack world and has been deeply involved in glance for quite a while. Thank you Erno for stepping up and for helping with Telemetry! Matthias [1] https://governance.openstack.org/election/#ptl-candidates -- Matthias Runge <mrunge(a)matthias-runge.de>

1 0