- openstack-discuss - lists.openstack.org

Re: [openstack][neutron][openvswitch] Openvswitch Packet loss when high throughput (pps)
by Ha Noi 10 Sep '23

10 Sep '23

Hi Satish, Actually, the guess interface is not using tap anymore. <interface type='vhostuser'> <mac address='fa:16:3e:76:77:dd'/> <source type='unix' path='/var/run/openvswitch/vhu3766ee8a-86' mode='server'/> <target dev='vhu3766ee8a-86'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> It's totally bypass the kernel stack ? On Fri, Sep 8, 2023 at 5:02 AM Satish Patel <satish.txt(a)gmail.com> wrote: > I did test OVS-DPDK and it helps offload the packet process on compute > nodes, But what about VMs it will still use a tap interface to attach from > compute to vm and bottleneck will be in vm. I strongly believe that we have > to run DPDK based guest to pass through the kernel stack. > > I love to hear from other people if I am missing something here. > > On Thu, Sep 7, 2023 at 5:27 PM Ha Noi <hanoi952022(a)gmail.com> wrote: > >> Oh. I heard from someone on the reddit said that Ovs-dpdk is transparent >> with user? >> >> So It’s not correct? >> >> On Thu, 7 Sep 2023 at 22:13 Satish Patel <satish.txt(a)gmail.com> wrote: >> >>> Because DPDK required DPDK support inside guest VM. It's not >>> suitable for general purpose workload. You need your guest VM network to >>> support DPDK to get 100% throughput. >>> >>> On Thu, Sep 7, 2023 at 8:06 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>> >>>> Hi Satish, >>>> >>>> Why dont you use DPDK? >>>> >>>> Thanks >>>> >>>> On Thu, 7 Sep 2023 at 19:03 Satish Patel <satish.txt(a)gmail.com> wrote: >>>> >>>>> I totally agreed with Sean on all his points but trust me, I have >>>>> tried everything possible to tune OS, Network stack, multi-queue, NUMA, CPU >>>>> pinning and name it.. but I didn't get any significant improvement. You may >>>>> gain 2 to 5% gain with all those tweek. I am running the entire workload on >>>>> sriov and life is happy except no LACP bonding. >>>>> >>>>> I am very interesting is this project >>>>> https://docs.openvswitch.org/en/latest/intro/install/afxdp/ >>>>> >>>>> On Thu, Sep 7, 2023 at 6:07 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>>>> >>>>>> Dear Smoney, >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 7, 2023 at 12:41 AM <smooney(a)redhat.com> wrote: >>>>>> >>>>>>> On Wed, 2023-09-06 at 11:43 -0400, Satish Patel wrote: >>>>>>> > Damn! We have noticed the same issue around 40k to 55k PPS. Trust >>>>>>> me >>>>>>> > nothing is wrong in your config. This is just a limitation of the >>>>>>> software >>>>>>> > stack and kernel itself. >>>>>>> its partly determined by your cpu frequency. >>>>>>> kernel ovs of yesteryear could handel about 1mpps total on a ~4GHZ >>>>>>> cpu. with per port troughpuyt being lower dependin on what >>>>>>> qos/firewall >>>>>>> rules that were apllied. >>>>>>> >>>>>>> >>>>>> >>>>>> My CPU frequency is 3Ghz and using CPU Intel Gold 2nd generation. I >>>>>> think the problem is tuning in the compute node inside. But I cannot find >>>>>> any guide or best practices for it. >>>>>> >>>>>> >>>>>> >>>>>>> moving form iptables firewall to ovs firewall can help to some degree >>>>>>> but your partly trading connection setup time for statead state >>>>>>> troughput >>>>>>> with the overhead of the connection tracker in ovs. >>>>>>> >>>>>>> using stateless security groups can help >>>>>>> >>>>>>> we also recently fixed a regression cause by changes in newer >>>>>>> versions of ovs. >>>>>>> this was notable in goign form rhel 8 to rhel 9 where litrally it >>>>>>> reduced >>>>>>> small packet performce to 1/10th and jumboframes to about 1/2 >>>>>>> on master we have a config option that will set the default qos on a >>>>>>> port to linux-noop >>>>>>> >>>>>>> https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L106-L1… >>>>>>> >>>>>>> the backports are propsoed upstream >>>>>>> https://review.opendev.org/q/Id9ef7074634a0f23d67a4401fa8fca363b51bb43 >>>>>>> and we have backported this downstream to adress that performance >>>>>>> regression. >>>>>>> the upstram backport is semi stalled just ebcasue we wanted to >>>>>>> disucss if we shoudl make ti opt in >>>>>>> by default upstream while backporting but it might be helpful for >>>>>>> you if this is related to yoru current >>>>>>> issues. >>>>>>> >>>>>>> 40-55 kpps is kind of low for kernel ovs but if you have a low >>>>>>> clockrate cpu, hybrid_plug + incorrect qos >>>>>>> then i could see you hitting such a bottelneck. >>>>>>> >>>>>>> one workaround by the way without the os-vif workaround backported >>>>>>> is to set >>>>>>> /proc/sys/net/core/default_qdisc to not apply any qos or a low >>>>>>> overhead qos type >>>>>>> i.e. sudo sysctl -w net.core.default_qdisc=pfifo_fast >>>>>>> >>>>>>> >>>>>> >>>>>>> that may or may not help but i would ensure that your are not usign >>>>>>> somting like fqdel or cake >>>>>>> for net.core.default_qdisc and if you are try changing it to >>>>>>> pfifo_fast and see if that helps. >>>>>>> >>>>>>> there isnet much you can do about the cpu clock rate but ^ is >>>>>>> somethign you can try for free >>>>>>> note it wont actully take effect on an exsitng vm if you jsut change >>>>>>> the default but you can use >>>>>>> tc to also chagne the qdisk for testing. hard rebooting the vm >>>>>>> shoudl also make the default take effect. >>>>>>> >>>>>>> the only other advice i can give assuming kernel ovs is the only >>>>>>> option you have is >>>>>>> >>>>>>> to look at >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rx… >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.tx… >>>>>>> and >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/extra-specs.html#hw:vi… >>>>>>> >>>>>>> if the bottelneck is actully in qemu or the guest kernel rather then >>>>>>> ovs adjusting the rx/tx queue size and >>>>>>> using multi queue can help. it will have no effect if ovs is the >>>>>>> bottel neck. >>>>>>> >>>>>>> >>>>>>> >>>>>> I have set this option to 1024, and enable multiqueue as well. But it >>>>>> did not help. >>>>>> >>>>>> >>>>>>> > >>>>>>> > On Wed, Sep 6, 2023 at 9:21 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > > Hi Satish, >>>>>>> > > >>>>>>> > > Actually, our customer get this issue when the tx/rx above only >>>>>>> 40k pps. >>>>>>> > > So what is the threshold of this throughput for OvS? >>>>>>> > > >>>>>>> > > >>>>>>> > > Thanks and regards >>>>>>> > > >>>>>>> > > On Wed, 6 Sep 2023 at 20:19 Satish Patel <satish.txt(a)gmail.com> >>>>>>> wrote: >>>>>>> > > >>>>>>> > > > Hi, >>>>>>> > > > >>>>>>> > > > This is normal because OVS or LinuxBridge wire up VMs using >>>>>>> TAP interface >>>>>>> > > > which runs on kernel space and that drives higher interrupt >>>>>>> and that makes >>>>>>> > > > the kernel so busy working on handling packets. Standard >>>>>>> OVS/LinuxBridge >>>>>>> > > > are not meant for higher PPS. >>>>>>> > > > >>>>>>> > > > If you want to handle higher PPS then look for DPDK or SRIOV >>>>>>> deployment. >>>>>>> > > > ( We are running everything in SRIOV because of high PPS >>>>>>> requirement) >>>>>>> > > > >>>>>>> > > > On Tue, Sep 5, 2023 at 11:11 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > > > >>>>>>> > > > > Hi everyone, >>>>>>> > > > > >>>>>>> > > > > I'm using Openstack Train and Openvswitch for ML2 driver and >>>>>>> GRE for >>>>>>> > > > > tunnel type. I tested our network performance between two >>>>>>> VMs and suffer >>>>>>> > > > > packet loss as below. >>>>>>> > > > > >>>>>>> > > > > VM1: IP: 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > VM2: IP: 10.20.1.154 <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > VM3: IP: 10.20.1.72 >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using iperf3 to testing performance between VM1 and VM2. >>>>>>> > > > > >>>>>>> > > > > Run iperf3 client and server on both VMs. >>>>>>> > > > > >>>>>>> > > > > On VM2: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > On VM1: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.154 >>>>>>> > > > > <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using VM3 ping into VM1, then the packet is lost and the >>>>>>> latency is >>>>>>> > > > > quite high. >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > ping -i 0.1 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > PING 10.20.1.206 (10.20.1.206) 56(84) bytes of data. >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=1 ttl=64 time=7.70 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=2 ttl=64 time=6.90 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=3 ttl=64 time=7.71 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=4 ttl=64 time=7.98 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=6 ttl=64 time=8.58 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=7 ttl=64 time=8.34 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=8 ttl=64 time=8.09 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=10 ttl=64 time=4.57 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=11 ttl=64 time=8.74 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=12 ttl=64 time=9.37 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=14 ttl=64 time=9.59 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=15 ttl=64 time=7.97 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=16 ttl=64 time=8.72 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=17 ttl=64 time=9.23 ms >>>>>>> > > > > >>>>>>> > > > > ^C >>>>>>> > > > > >>>>>>> > > > > --- 10.20.1.206 ping statistics --- >>>>>>> > > > > >>>>>>> > > > > 34 packets transmitted, 28 received, 17.6471% packet loss, >>>>>>> time 3328ms >>>>>>> > > > > >>>>>>> > > > > rtt min/avg/max/mdev = 1.396/6.266/9.590/2.805 ms >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Does any one get this issue ? >>>>>>> > > > > >>>>>>> > > > > Please help me. Thanks >>>>>>> > > > > >>>>>>> > > > >>>>>>> >>>>>>>

3 5

[release] Release countdown for week R-3, Sep 11-15
by Herve Beraud 08 Sep '23

08 Sep '23

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, September 14. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/2023.2 branch will be cut. This branch will track the 2023.2 Bobcat release. Once stable/2023.2 has been created, the master branch will be ready to switch to 2024.1 Caracal development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing 2023.2 Bobcat plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/2023.2 branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/2023.2 branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be 2024.1 Bobcat) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any 2023.2 Bobcat release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: September 14 (R-3 week) Final RC deadline: September 28 (R-1 week) Final 2023.2 Bobcat release: October 4 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 08 Sep '23

08 Sep '23

Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Have a nice weekend. [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[openstack][neutron][openvswitch] Openvswitch Packet loss when high throughput (pps)
by Ha Noi 08 Sep '23

08 Sep '23

Hi everyone, I'm using Openstack Train and Openvswitch for ML2 driver and GRE for tunnel type. I tested our network performance between two VMs and suffer packet loss as below. VM1: IP: 10.20.1.206 VM2: IP: 10.20.1.154 <https://10.20.1.154/24> VM3: IP: 10.20.1.72 Using iperf3 to testing performance between VM1 and VM2. Run iperf3 client and server on both VMs. On VM2: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.206 On VM1: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.154 <https://10.20.1.154/24> Using VM3 ping into VM1, then the packet is lost and the latency is quite high. ping -i 0.1 10.20.1.206 PING 10.20.1.206 (10.20.1.206) 56(84) bytes of data. 64 bytes from 10.20.1.206: icmp_seq=1 ttl=64 time=7.70 ms 64 bytes from 10.20.1.206: icmp_seq=2 ttl=64 time=6.90 ms 64 bytes from 10.20.1.206: icmp_seq=3 ttl=64 time=7.71 ms 64 bytes from 10.20.1.206: icmp_seq=4 ttl=64 time=7.98 ms 64 bytes from 10.20.1.206: icmp_seq=6 ttl=64 time=8.58 ms 64 bytes from 10.20.1.206: icmp_seq=7 ttl=64 time=8.34 ms 64 bytes from 10.20.1.206: icmp_seq=8 ttl=64 time=8.09 ms 64 bytes from 10.20.1.206: icmp_seq=10 ttl=64 time=4.57 ms 64 bytes from 10.20.1.206: icmp_seq=11 ttl=64 time=8.74 ms 64 bytes from 10.20.1.206: icmp_seq=12 ttl=64 time=9.37 ms 64 bytes from 10.20.1.206: icmp_seq=14 ttl=64 time=9.59 ms 64 bytes from 10.20.1.206: icmp_seq=15 ttl=64 time=7.97 ms 64 bytes from 10.20.1.206: icmp_seq=16 ttl=64 time=8.72 ms 64 bytes from 10.20.1.206: icmp_seq=17 ttl=64 time=9.23 ms ^C --- 10.20.1.206 ping statistics --- 34 packets transmitted, 28 received, 17.6471% packet loss, time 3328ms rtt min/avg/max/mdev = 1.396/6.266/9.590/2.805 ms Does any one get this issue ? Please help me. Thanks

3 9

[ops]
by Chris Morgan 08 Sep '23

08 Sep '23

I'm trying to reanimate the openstack ops meetups. I moved the social media account used for notifications from X to mastodon (using the fosstodon instance). Please follow there if interested (link below). I also posted a similar announcement to the openstack group on linkedin. One of the first tasks (besides reforming the team) is finding which communications channels actually work. https://fosstodon.org/@osopsmeetup The old account was here https://twitter.com/osopsmeetup now "sepia-tinted" because it's just historical. Cheers, Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[i18n] New I18n SIG Core
by Ian Y. Choi 08 Sep '23

08 Sep '23

Hello, I would like to happily announce a new i18n core: Seongsoo Cho, considering his notable contributions on I18n with volunteering: 1. PDF Docs contribution with I18n issues (font issues, latex with Asian languages) [1] 2. Zanata to Weblate migration volunteering effort [2] 3. Participation of I18n SIG PTG on March [3] and Forum on May [4] Also, he is active on #openstack-i18n IRC channel (nick name: "seongsoocho"). All, please welcome Seongsoo as I18n-core. I am looking forward to more active contributions from him and interactions with other wonderful I18n members and OpenStack team! Thank you, /Ian [1] https://review.opendev.org/q/topic:bp%252Fbuild-pdf-from-rst-guides [2] https://lists.openstack.org/pipermail/openstack-i18n/2022-October/003568.ht… [3] https://etherpad.opendev.org/p/march2023-ptg-i18n [4] https://etherpad.opendev.org/p/vancouver-2023-i18n-forum

4 3

[Glance] Cancelling Weekly Meeting 7th Sept
by Pranali Deore 07 Sep '23

07 Sep '23

Hello, I won't be able to chair today's meeting, my kid is not well, going to take him to the doctor. so cancelling the meeting for this week. Let's meet next week ! Thanks, Pranali D

1 0

Windows Image failed to access network
by Richard Buregeya 07 Sep '23

07 Sep '23

Hello Team, I created the windows image and added the N/w drivers, but it can't be able to get from DHCP. Any idea? Regards Richard.

2 1

[neutron][release] Proposing transition to EOL Train (Neutron and neutron-lib)
by Rodolfo Alonso Hernandez 07 Sep '23

07 Sep '23

Hello: I'm sending this mail in advance to propose transitioning the Neutron and the neutron-lib Train branch to EOL. This topic was proposed and approved during the last Neutron team meeting [1]. These are the only two Neutron related projects still in EM. The announcement is the first step [2] to transition a stable branch to EOL. The patch to mark these branches as EOL will be pushed in two weeks. If you have any inconvenience, please let me know in this mail chain or in IRC (ralonsoh, #openstack-neutron channel). You can also contact any Neutron core reviewer in the IRC channel. Regards. [2] https://meetings.opendev.org/meetings/networking/2023/networking.2023-09-05… [1] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l…

1 0

[all][elections][ptl][tc] Conbined PTL/TC 2024.1 cycle Election Voting Kickoff
by Ian Y. Choi 07 Sep '23

07 Sep '23

Polls for PTL and TC elections are now open and will remain open for you to cast your vote until Sep 20, 2023 23:45 UTC. We are selecting 4 TC members, and are having PTL elections for OpenStack_Helm. Please rank all candidates in your order of preference. You are eligible to vote in the TC election if you are a Foundation individual member[0] that also has committed to any official project team's deliverable repositories[1] over the Sep 16, 2022 00:00 UTC - Aug 30, 2023 00:00 UTC timeframe (2023.1 to 2023.2) or if you are in the list of extra-atcs[2] for any official project team. You are eligible to vote in a PTL election if you are a Foundation individual member[0] and had a commit in one of that team's deliverable repositories[1] over the Sep 16, 2022 00:00 UTC - Aug 30, 2023 00:00 UTC timeframe (2023.1 to 2023.2) or if you are in that team's list of extra-atcs[2]. If you are eligible to vote in an election, you should find your email with a link to the Condorcet Internet Voting Service (CIVS) page to cast your vote in the inbox of your Gerrit preferred email[3]. What to do if you don't see the email and have a commit in at least one of the projects having an election: * due to a new CIVS policy, to get the email from poll supervisors, you must opt in to email communication opt in with your Gerrit email address at: https://civs1.civs.us/cgi-bin/opt_in.pl * eligible voters who opt in to email communication late can see a pending poll invitation and vote until Sep 20, 2023 23:45 UTC. * check the trash or spam folders of your Gerrit Preferred Email address, in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from the project's deliverable repos[0] and email the election officials[4]. If we can confirm that you are entitled to vote, we will add you to the voters list for the appropriate election. Our democratic process is important to the health of OpenStack, please exercise your right to vote! Candidate statements/platforms can be found linked to Candidate names on this page: https://governance.openstack.org/election/ Happy voting, [0] https://www.openstack.org/community/members/ [1] The list of the repositories eligible for electoral status: https://opendev.org/openstack/governance/src/tag/0.15.0/reference/projects.… [2] Look for the extra-atcs element in [1] [3] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your preferred email. That is where the ballot has been sent. [4] https://governance.openstack.org/election/#election-officials

1 0

[manila] Cancelling Sep 7 weekly meeting
by Carlos Silva 07 Sep '23

07 Sep '23

Hello, Zorillas! As discussed in the previous meeting, we are cancelling tomorrow's weekly meeting, as a big part of the audience is on PTO tomorrow/this week. The next Manila weekly meeting will be on Sep 14th. If you have something urgent to bring up, please let me know on the #openstack-manila IRC channel. Regards, carloss

1 0

[largescale-sig] Next meeting: Sept 6, 15utc
by Thierry Carrez 07 Sep '23

07 Sep '23

Hi everyone, The Large Scale SIG is back after the northern hemisphere summer break and will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC, our EU+US-friendly time. I will be chairing. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230906T15 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[kolla-ansible][octavia] octavia management network setup for vlan provider
by Satish Patel 06 Sep '23

06 Sep '23

Folks, I have setup kolla-ansible and configured octavia using the o-hm0 interface with the tenant and it works. For production I think I should use VLAN based provider for octavia management network so this is what I did I have created a bond0.41 dedicated interface on all 3 controller nodes and created vlan 41 on all network switches. This is what my global.yml looks like ## Octivia enable_octavia: "yes" octavia_network_interface: "bond0.41" octavia_amp_flavor: name: "amphora" is_public: no vcpus: 2 ram: 2048 disk: 5 octavia_amp_network: name: lb-mgmt-net provider_network_type: vlan provider_segmentation_id: 41 provider_physical_network: physnet1 external: false shared: false subnet: name: lb-mgmt-subnet cidr: "192.168.41.0/24" allocation_pool_start: "192.168.41.100" allocation_pool_end: "192.168.41.200" enable_dhcp: yes After running the playbook all get setup as per document. When I create loadbalancer it just get stuck in PENDING status. [1] Document saying make sure your octavia_network_interface is connected to openvswitch. Do I need to connect manually or will kolla-ansible do that for me? If I am going to do that then on which bridge I should attach br-ex or br-int ? [1] https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavi…

3 3

Cinder Bug Report 2023-09-06
by Jon Bernard 06 Sep '23

06 Sep '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - Weak Cryptographic Algorithm (MD5) used <https://bugs.launchpad.net/cinder/+bug/2033612> - Status: New - group list doesn't parse 'all_tenants' parameter correctly <https://bugs.launchpad.net/cinder/+bug/2034248> - Status: In Progress - Documentation check and correction for PowerStore NFS driver <https://bugs.launchpad.net/cinder/+bug/2034482> - Status: New Thanks -- Jon

1 0

[all][elections][ptl][tc] Opt in to CIVS voting system
by Ian Y. Choi 06 Sep '23

06 Sep '23

The election uses the Condorcet Internet Voting Service (CIVS). Due to CIVS policy, to vote in private CIVS polls, you must opt in to email communication. To opt in, please enter your Gerrit email address in the following page before the start of the election Sep 06, 2023 23:45 UTC, and confirm with the code that will sent to you via email. https://civs1.civs.us/cgi-bin/opt_in.pl If you have any question, please contact the election officials. https://governance.openstack.org/election/#election-officials

1 0

[election] [tc] Running for a second year on the TC
by Jay Faulkner 06 Sep '23

06 Sep '23

Hey all, Around a year ago, I announced my candidacy for the TC after a call for more contributors from a diverse employer base was made. I think a year later, it's more obvious than ever how important it is to have a diverse, independent technical committee. We have made huge progress on this front. We have six candidates for the four open TC positions, representing different independent companies utilizing OpenStack. This is a major improvement year over year. If you've been in the community a while, you've probably worked with me and have an idea if you want to vote for me or not in this position. Whether elected or not, I'll continue my vocal support of OpenStack technologies in private meetings and on social media, I'll continue to invite people I meet interested in cloud to join our community -- not because it's my job (it is), but also because I'm proud of what we've built. When I have those conversations with potential users or contributors, I highlight two things I admire about the OpenStack community: longevity -- there are not many large, distributed open source projects that have survived as long as we have, and the four opens -- putting openness out front as our face to the world is why I'll continue to be proud to represent OpenStack in the future -- as a TC member or not (you decide). If someone does want to talk about a particular policy or thought process of mine, I'm happy to respond to comments here or via private email/IRC. Thanks, Jay Faulkner

1 0

[QA][release] Cycle With Intermediary without recent deliverables
by Herve Beraud 06 Sep '23

06 Sep '23

Quick reminder that for deliverables following the cycle-with-intermediary model, the release team will use the latest $series release available on release week. The following deliverables have done a 2023.2 bobcat release, but it was not refreshed in the last two months: tempest You should consider making a new one very soon, so that we don't use an outdated version for the final release. Thanks -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

2 1

Host aggregates and placement aggregates
by Gk Gk 05 Sep '23

05 Sep '23

Hi All, We have a yoga OSA setup. We have upgraded the setup from Xena to Yoga. We have found that some of the host aggregates have not synced to the placement aggregates. As a result , the vm creation on those aggregate nodes is failing even though they have the full capacity. I have tried the placement sync aggregates command but it did not help. Is this a known bug in Yoga ? Thanks Y.G

1 0

[openstack-ansible] Recommended branch for a production environment?
by Roger Rivera 05 Sep '23

05 Sep '23

Hello, We have deployed Openstack-Ansible in a test environment and we were wondering what the recommended repository branch is to deploy in a production environment that will be integrated with a separate Ceph cluster. 1. We are thinking about pinning to the stable/zed. Is that recommended over the master branch? 2. Are bugs ironed out on both master and stable/zed with the same cadence? 3. Additionally, is Debian 11 a better alternative than Ubuntu 22 for target hosts? We noticed Ubuntu 22.04 support was added recently, whereas Debian 11 has been supported for quite some time now which leads us to believe its stability/integration could be more mature at this point? Any suggestions will be appreciated. Thank you, -- *Roger *

3 7

What is the difference between 4 and 8 virtual sockets to physical sockets?
by Jorge Visentini 05 Sep '23

05 Sep '23

Hello Team, What is the difference between creating an instance with *4* or *8 virtual sockets*, since the hypervisor has only *4 physical sockets*. My question is where do sockets, cores and virtual threads fit into the physical hardware. I think this question is not just related to Openstack, but with any virtualization. My hypervisor configuration is as follows: CPU(s): 192 Online CPU(s) list: 0-191 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 4 NUMA node(s): 4 Do you have any documentation that I can read and understand better? That we have a nice week! -- Att, Jorge Visentini +55 55 98432-9868

2 3

[tc] Technical Committee next weekly meeting Today on September 5
by Nikolla, Kristi 05 Sep '23

05 Sep '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, September 5, 2023 at 1800 UTC on Zoom https://us06web.zoom.us/j/87108541765?pwd=emlXVXg4QUxrUTlLNDZ2TTllWUM3Zz09 The agenda can be found at the link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggesti… and attached below: • Roll call • Follow up on past action items • No action items • Gate health check • OpenStack Elections • #link https://governance.openstack.org/election/ • Ballots go out tomorrow and remain open until September 20. • We have 6 candidates out of 4 seats for TC :) • Open Discussion and Reviews • Register for the PTG • #link https://openinfra.dev/ptg/ • #link https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla

1 0

RE: ***UNCHECKED*** openstack-discuss Digest, Vol 59, Issue 10 | Openstack Web Interface Issue
by Asma Naz Shariq 05 Sep '23

05 Sep '23

Hi Team, I have deployed Openstack through manual installation-Yoga release. I encountered the attached error and can't have access to Openstack dashboard. When I entered the login credentials it displays something went wrong and you don't have permission to use /horizon/. Can anyone have a solution for this error. -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Monday, September 4, 2023 8:27 PM To: openstack-discuss(a)lists.openstack.org Subject: ***UNCHECKED*** openstack-discuss Digest, Vol 59, Issue 10 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss or, via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: Live migration error (Karera Tony) 2. What is the difference between 4 and 8 virtual sockets to physical sockets? (Jorge Visentini) 3. [all][stable][ptl] Propose to EOL Stein series (El?d Ill?s) ---------------------------------------------------------------------- Message: 1 Date: Mon, 4 Sep 2023 14:11:37 +0200 From: Karera Tony <tonykarera(a)gmail.com> To: smooney(a)redhat.com Cc: Danny Webb <Danny.Webb(a)thehutgroup.com>, openstack-discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: Live migration error Message-ID: <CA+69TL3=7GwfgRgAjfxvx5q4aZCstXxtBeB2mGb9ofa0JCZmbA(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello Smooney, I can confirm that it is disabled on all the servers. Regards Tony Karera On Mon, Sep 4, 2023 at 1:25?PM <smooney(a)redhat.com> wrote: > On Mon, 2023-09-04 at 12:59 +0200, Karera Tony wrote: > > Hello Danny, > > > > I am actually using Broadwell-noTSX-IBRS on all the servers but I > > have > seen > > that some flags are not there in the older servers. > > > > The flags below to be specific. > > > > hwp hwp_act_window hwp_pkg_req > so hwp is apprentlly intels pstate contol based on > https://unix.stackexchange.com/a/43540 > > i dont belive this is normally exposed to a guest but if you are > seeign differnce between hosts then that likely means you have > disabled pstates also know as "intel speedstep" on some of the host > and not others. > > > > > Regards > > > > Tony Karera > > > > > > > > > > On Fri, Sep 1, 2023 at 4:05?PM <smooney(a)redhat.com> wrote: > > > > > On Fri, 2023-09-01 at 14:56 +0200, Karera Tony wrote: > > > > Hello Danny, > > > > > > > > Thanks for the feedback. > > > > > > > > use a lowest common denominator cpu_model : Does this mean > > > > changing > the > > > > servers ? > > > > > > it means tha tif you have a mix of cpus in the deployment you > > > shoudl hardcod to the older of the diffent cpu models i.e skylake > > > or whatever it may > be > > > in your case. > > > > > > > > > > https://docs.openstack.org/nova/latest/configuration/config.html#libvi > rt.cpu_models > > > > > > [libvirt] > > > cpu_mode=custom > > > cpu_models=skylake > > > > > > in more recent release we replaced the older cpu_model with > > > cpu_models with is a comma sperated list of models in prefernce > > > order. i.e. nova will use the first > cpu_modle > > > in the list that work > > > on the current host. not that while this give extra flexiblity it > > > vs > just > > > using the oldest cpu model > > > it does limit the set of hosts you can migrate too but means you > > > can > have > > > better perfromance so its > > > a tradeoff. performance vs portablity. > > > > > > the error you mentioned can also be caused by micorocode updates. > > > intel remove TSX and MPX i beilve in the last 2 years form come of > there > > > cpus > > > that breaks live migration if a vm was create with access to that > > > cpu instruction > > > > > > the only way to resolve that is to cold migrate the guest. > > > i.e. if a vm is currently booted with cpu_model X it cannot be > > > modifed while the guest is running so you either need to update > > > the config option and reboot all the vms > or > > > more particlaly update the config > > > and then cold migrate them which will allow you to move the > instnace(your > > > orginal goal) while allowing > > > the new cpu model to take effect. > > > > > > novas default for the cpu model when not set is with our default > cpu_mode > > > is host_model > > > meaning whatever model best matches the current hosts cpu. > > > > > > that is the correct default in general but if you have mixed cpu > > > generatiosn in your cloud its not ideal. > > > > > > hopefuly that helps > > > > > > > > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > On Fri, Sep 1, 2023 at 12:37?PM Danny Webb < > Danny.Webb(a)thehutgroup.com> > > > > wrote: > > > > > > > > > It means that you have CPUs with incompatible flags or you've > > > > > got differing kernel versions that expose different flags or > > > > > you've got differing libvirt versions that expose different > > > > > flags. You either > > > need to > > > > > use a lowest common denominator cpu_model or do a cold migration. > > > > > ------------------------------ > > > > > *From:* Karera Tony <tonykarera(a)gmail.com> > > > > > *Sent:* 01 September 2023 10:13 > > > > > *To:* openstack-discuss > > > > > <openstack-discuss(a)lists.openstack.org> > > > > > *Subject:* Live migration error > > > > > > > > > > > > > > > * CAUTION: This email originates from outside THG * > > > > > ------------------------------ Hello Team, > > > > > > > > > > I am trying to migrate instances from one host to another but > > > > > I am > > > getting > > > > > this error. > > > > > > > > > > *Error: *Failed to live migrate instance to host "compute1". > Details > > > > > <http://10.10.13.31/admin/instances/#message_details> > > > > > Migration pre-check error: Unacceptable CPU info: CPU doesn't > > > > > have compatibility. 0 Refer to > > > > > > http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult > > > > > (HTTP 400) (Request-ID: > > > > > req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da) > > > > > > > > > > Any idea on how to fix this? > > > > > > > > > > Regards > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > >

1 0

Need assistance
by Gk Gk 05 Sep '23

05 Sep '23

Hi, We have Yoga version setup of OSA. When a user is trying to use a certain flavor which has some extra specs, same as that which is mentioned for an aggregate, the vm creation is failing for the user. But I created a vm using that same flavor and it succeeded for a test user account. When I analyzed the logs, the nova scheduler is not filtering or checking the individual hosts from that target aggregate in the case of that user. But it is validating the target aggregate hosts in the case of a test user and able to select a host for vm creation. So what could be the reason that the scheduler is unable to find the hosts from that aggregate in the case of that user as opposed to that of the test user ? Is there any connection between a project and an aggregate ? Thanks Kumar

2 1

[all][qa] Devstack dropping support for Ubuntu Focal
by Ghanshyam Mann 05 Sep '23

05 Sep '23

Hello Everyone, As you know, this release 2023.2 (bobcat) does not mandate testing the ubuntu Focal[1] and with Nova bumped the libvirt version, focal jobs are failing. Devstack was planning to drop the Focal support in the next cycle but we have to do it now only[2]. This email is a heads-up if you are still wondering about the Focal job failure. If your project still running any of the jobs on focal, please plan to move to Jammy. [1] https://governance.openstack.org/tc/reference/runtimes/2023.2.html [2] https://review.opendev.org/c/openstack/devstack/+/885468 -gmann

1 0

[all][stable][ptl] Propose to EOL Stein series
by Elõd Illés 05 Sep '23

05 Sep '23

Hi, Core projects announced and deleted their stable/stein branches for some time already. Furthermore, most of the projects, where stable/stein is still open, have broken gates. So, the same way as we transitioned Rocky series to End of Life [1][2], now is the time to do the same with Stein. I'll propose the stein-eol transition patches for the rest of the open projects. I ask again the teams to signal their decision (with a +1 if the team is ready for the transition). Thanks in advance! (Please note, that the Extended Maintenance process is phasing out / transforming soon, see details in Technical Committee's resolution [3], which was accepted and merged in August.) Thanks, Előd Illés irc: elodilles @ #openstack-stable / #openstack-release [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031922… [2] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033386.h… [3] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc…

1 0