September 2023 - openstack-discuss

[Skyline] error username or password incorrect
by Satish Patel 11 Sep '23

11 Sep '23

Folks, Try to install skyline UI to replace horizon using doc: https://docs.openstack.org/skyline-apiserver/latest/install/docker-install-… Everything went well and I got a login page on http://x.x.x.x:9999 also it pulled Region/Domains. When I am trying to login with my account, I get an error: Username or Password is incorrect. I am using sqlite DB for skyline as per documents. No errors in logs command $ docker logs skyline When I use Chrome Developer Tools then it was indicating an error in these URLs. http://openstack.example.com:9999/api/openstack/skyline/api/v1/profile http://openstack.example.com:9999/api/openstack/skyline/api/v1/policies 401 Unauthorized ( {"detail":"no such table: revoked_token"} )

2 16

No valid host was found. There are not enough hosts available
by - 10 Sep '23

10 Sep '23

I added a new node to the OpenStack cluster and when creating a new instance I have this error. Maybe someone can tell me what else I can check? No valid host was found. There are not enough hosts available. No valid host was found. There are not enough hosts available. Kod 500 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1548, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 908, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Utworzono

2 1

Re: [openstack][neutron][openvswitch] Openvswitch Packet loss when high throughput (pps)
by Ha Noi 10 Sep '23

10 Sep '23

Hi Satish, Actually, the guess interface is not using tap anymore. <interface type='vhostuser'> <mac address='fa:16:3e:76:77:dd'/> <source type='unix' path='/var/run/openvswitch/vhu3766ee8a-86' mode='server'/> <target dev='vhu3766ee8a-86'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> It's totally bypass the kernel stack ? On Fri, Sep 8, 2023 at 5:02 AM Satish Patel <satish.txt(a)gmail.com> wrote: > I did test OVS-DPDK and it helps offload the packet process on compute > nodes, But what about VMs it will still use a tap interface to attach from > compute to vm and bottleneck will be in vm. I strongly believe that we have > to run DPDK based guest to pass through the kernel stack. > > I love to hear from other people if I am missing something here. > > On Thu, Sep 7, 2023 at 5:27 PM Ha Noi <hanoi952022(a)gmail.com> wrote: > >> Oh. I heard from someone on the reddit said that Ovs-dpdk is transparent >> with user? >> >> So It’s not correct? >> >> On Thu, 7 Sep 2023 at 22:13 Satish Patel <satish.txt(a)gmail.com> wrote: >> >>> Because DPDK required DPDK support inside guest VM. It's not >>> suitable for general purpose workload. You need your guest VM network to >>> support DPDK to get 100% throughput. >>> >>> On Thu, Sep 7, 2023 at 8:06 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>> >>>> Hi Satish, >>>> >>>> Why dont you use DPDK? >>>> >>>> Thanks >>>> >>>> On Thu, 7 Sep 2023 at 19:03 Satish Patel <satish.txt(a)gmail.com> wrote: >>>> >>>>> I totally agreed with Sean on all his points but trust me, I have >>>>> tried everything possible to tune OS, Network stack, multi-queue, NUMA, CPU >>>>> pinning and name it.. but I didn't get any significant improvement. You may >>>>> gain 2 to 5% gain with all those tweek. I am running the entire workload on >>>>> sriov and life is happy except no LACP bonding. >>>>> >>>>> I am very interesting is this project >>>>> https://docs.openvswitch.org/en/latest/intro/install/afxdp/ >>>>> >>>>> On Thu, Sep 7, 2023 at 6:07 AM Ha Noi <hanoi952022(a)gmail.com> wrote: >>>>> >>>>>> Dear Smoney, >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 7, 2023 at 12:41 AM <smooney(a)redhat.com> wrote: >>>>>> >>>>>>> On Wed, 2023-09-06 at 11:43 -0400, Satish Patel wrote: >>>>>>> > Damn! We have noticed the same issue around 40k to 55k PPS. Trust >>>>>>> me >>>>>>> > nothing is wrong in your config. This is just a limitation of the >>>>>>> software >>>>>>> > stack and kernel itself. >>>>>>> its partly determined by your cpu frequency. >>>>>>> kernel ovs of yesteryear could handel about 1mpps total on a ~4GHZ >>>>>>> cpu. with per port troughpuyt being lower dependin on what >>>>>>> qos/firewall >>>>>>> rules that were apllied. >>>>>>> >>>>>>> >>>>>> >>>>>> My CPU frequency is 3Ghz and using CPU Intel Gold 2nd generation. I >>>>>> think the problem is tuning in the compute node inside. But I cannot find >>>>>> any guide or best practices for it. >>>>>> >>>>>> >>>>>> >>>>>>> moving form iptables firewall to ovs firewall can help to some degree >>>>>>> but your partly trading connection setup time for statead state >>>>>>> troughput >>>>>>> with the overhead of the connection tracker in ovs. >>>>>>> >>>>>>> using stateless security groups can help >>>>>>> >>>>>>> we also recently fixed a regression cause by changes in newer >>>>>>> versions of ovs. >>>>>>> this was notable in goign form rhel 8 to rhel 9 where litrally it >>>>>>> reduced >>>>>>> small packet performce to 1/10th and jumboframes to about 1/2 >>>>>>> on master we have a config option that will set the default qos on a >>>>>>> port to linux-noop >>>>>>> >>>>>>> https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L106-L1… >>>>>>> >>>>>>> the backports are propsoed upstream >>>>>>> https://review.opendev.org/q/Id9ef7074634a0f23d67a4401fa8fca363b51bb43 >>>>>>> and we have backported this downstream to adress that performance >>>>>>> regression. >>>>>>> the upstram backport is semi stalled just ebcasue we wanted to >>>>>>> disucss if we shoudl make ti opt in >>>>>>> by default upstream while backporting but it might be helpful for >>>>>>> you if this is related to yoru current >>>>>>> issues. >>>>>>> >>>>>>> 40-55 kpps is kind of low for kernel ovs but if you have a low >>>>>>> clockrate cpu, hybrid_plug + incorrect qos >>>>>>> then i could see you hitting such a bottelneck. >>>>>>> >>>>>>> one workaround by the way without the os-vif workaround backported >>>>>>> is to set >>>>>>> /proc/sys/net/core/default_qdisc to not apply any qos or a low >>>>>>> overhead qos type >>>>>>> i.e. sudo sysctl -w net.core.default_qdisc=pfifo_fast >>>>>>> >>>>>>> >>>>>> >>>>>>> that may or may not help but i would ensure that your are not usign >>>>>>> somting like fqdel or cake >>>>>>> for net.core.default_qdisc and if you are try changing it to >>>>>>> pfifo_fast and see if that helps. >>>>>>> >>>>>>> there isnet much you can do about the cpu clock rate but ^ is >>>>>>> somethign you can try for free >>>>>>> note it wont actully take effect on an exsitng vm if you jsut change >>>>>>> the default but you can use >>>>>>> tc to also chagne the qdisk for testing. hard rebooting the vm >>>>>>> shoudl also make the default take effect. >>>>>>> >>>>>>> the only other advice i can give assuming kernel ovs is the only >>>>>>> option you have is >>>>>>> >>>>>>> to look at >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rx… >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.tx… >>>>>>> and >>>>>>> >>>>>>> https://docs.openstack.org/nova/latest/configuration/extra-specs.html#hw:vi… >>>>>>> >>>>>>> if the bottelneck is actully in qemu or the guest kernel rather then >>>>>>> ovs adjusting the rx/tx queue size and >>>>>>> using multi queue can help. it will have no effect if ovs is the >>>>>>> bottel neck. >>>>>>> >>>>>>> >>>>>>> >>>>>> I have set this option to 1024, and enable multiqueue as well. But it >>>>>> did not help. >>>>>> >>>>>> >>>>>>> > >>>>>>> > On Wed, Sep 6, 2023 at 9:21 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > > Hi Satish, >>>>>>> > > >>>>>>> > > Actually, our customer get this issue when the tx/rx above only >>>>>>> 40k pps. >>>>>>> > > So what is the threshold of this throughput for OvS? >>>>>>> > > >>>>>>> > > >>>>>>> > > Thanks and regards >>>>>>> > > >>>>>>> > > On Wed, 6 Sep 2023 at 20:19 Satish Patel <satish.txt(a)gmail.com> >>>>>>> wrote: >>>>>>> > > >>>>>>> > > > Hi, >>>>>>> > > > >>>>>>> > > > This is normal because OVS or LinuxBridge wire up VMs using >>>>>>> TAP interface >>>>>>> > > > which runs on kernel space and that drives higher interrupt >>>>>>> and that makes >>>>>>> > > > the kernel so busy working on handling packets. Standard >>>>>>> OVS/LinuxBridge >>>>>>> > > > are not meant for higher PPS. >>>>>>> > > > >>>>>>> > > > If you want to handle higher PPS then look for DPDK or SRIOV >>>>>>> deployment. >>>>>>> > > > ( We are running everything in SRIOV because of high PPS >>>>>>> requirement) >>>>>>> > > > >>>>>>> > > > On Tue, Sep 5, 2023 at 11:11 AM Ha Noi <hanoi952022(a)gmail.com> >>>>>>> wrote: >>>>>>> > > > >>>>>>> > > > > Hi everyone, >>>>>>> > > > > >>>>>>> > > > > I'm using Openstack Train and Openvswitch for ML2 driver and >>>>>>> GRE for >>>>>>> > > > > tunnel type. I tested our network performance between two >>>>>>> VMs and suffer >>>>>>> > > > > packet loss as below. >>>>>>> > > > > >>>>>>> > > > > VM1: IP: 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > VM2: IP: 10.20.1.154 <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > VM3: IP: 10.20.1.72 >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using iperf3 to testing performance between VM1 and VM2. >>>>>>> > > > > >>>>>>> > > > > Run iperf3 client and server on both VMs. >>>>>>> > > > > >>>>>>> > > > > On VM2: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > On VM1: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.154 >>>>>>> > > > > <https://10.20.1.154/24> >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Using VM3 ping into VM1, then the packet is lost and the >>>>>>> latency is >>>>>>> > > > > quite high. >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > ping -i 0.1 10.20.1.206 >>>>>>> > > > > >>>>>>> > > > > PING 10.20.1.206 (10.20.1.206) 56(84) bytes of data. >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=1 ttl=64 time=7.70 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=2 ttl=64 time=6.90 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=3 ttl=64 time=7.71 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=4 ttl=64 time=7.98 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=6 ttl=64 time=8.58 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=7 ttl=64 time=8.34 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=8 ttl=64 time=8.09 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=10 ttl=64 time=4.57 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=11 ttl=64 time=8.74 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=12 ttl=64 time=9.37 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=14 ttl=64 time=9.59 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=15 ttl=64 time=7.97 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=16 ttl=64 time=8.72 ms >>>>>>> > > > > >>>>>>> > > > > 64 bytes from 10.20.1.206: icmp_seq=17 ttl=64 time=9.23 ms >>>>>>> > > > > >>>>>>> > > > > ^C >>>>>>> > > > > >>>>>>> > > > > --- 10.20.1.206 ping statistics --- >>>>>>> > > > > >>>>>>> > > > > 34 packets transmitted, 28 received, 17.6471% packet loss, >>>>>>> time 3328ms >>>>>>> > > > > >>>>>>> > > > > rtt min/avg/max/mdev = 1.396/6.266/9.590/2.805 ms >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > Does any one get this issue ? >>>>>>> > > > > >>>>>>> > > > > Please help me. Thanks >>>>>>> > > > > >>>>>>> > > > >>>>>>> >>>>>>>

3 5

[release] Release countdown for week R-3, Sep 11-15
by Herve Beraud 08 Sep '23

08 Sep '23

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, September 14. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/2023.2 branch will be cut. This branch will track the 2023.2 Bobcat release. Once stable/2023.2 has been created, the master branch will be ready to switch to 2024.1 Caracal development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing 2023.2 Bobcat plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/2023.2 branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/2023.2 branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be 2024.1 Bobcat) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any 2023.2 Bobcat release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: September 14 (R-3 week) Final RC deadline: September 28 (R-1 week) Final 2023.2 Bobcat release: October 4 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 08 Sep '23

08 Sep '23

Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Have a nice weekend. [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[openstack][neutron][openvswitch] Openvswitch Packet loss when high throughput (pps)
by Ha Noi 07 Sep '23

07 Sep '23

Hi everyone, I'm using Openstack Train and Openvswitch for ML2 driver and GRE for tunnel type. I tested our network performance between two VMs and suffer packet loss as below. VM1: IP: 10.20.1.206 VM2: IP: 10.20.1.154 <https://10.20.1.154/24> VM3: IP: 10.20.1.72 Using iperf3 to testing performance between VM1 and VM2. Run iperf3 client and server on both VMs. On VM2: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.206 On VM1: iperf3 -t 10000 -b 130M -l 442 -P 6 -u -c 10.20.1.154 <https://10.20.1.154/24> Using VM3 ping into VM1, then the packet is lost and the latency is quite high. ping -i 0.1 10.20.1.206 PING 10.20.1.206 (10.20.1.206) 56(84) bytes of data. 64 bytes from 10.20.1.206: icmp_seq=1 ttl=64 time=7.70 ms 64 bytes from 10.20.1.206: icmp_seq=2 ttl=64 time=6.90 ms 64 bytes from 10.20.1.206: icmp_seq=3 ttl=64 time=7.71 ms 64 bytes from 10.20.1.206: icmp_seq=4 ttl=64 time=7.98 ms 64 bytes from 10.20.1.206: icmp_seq=6 ttl=64 time=8.58 ms 64 bytes from 10.20.1.206: icmp_seq=7 ttl=64 time=8.34 ms 64 bytes from 10.20.1.206: icmp_seq=8 ttl=64 time=8.09 ms 64 bytes from 10.20.1.206: icmp_seq=10 ttl=64 time=4.57 ms 64 bytes from 10.20.1.206: icmp_seq=11 ttl=64 time=8.74 ms 64 bytes from 10.20.1.206: icmp_seq=12 ttl=64 time=9.37 ms 64 bytes from 10.20.1.206: icmp_seq=14 ttl=64 time=9.59 ms 64 bytes from 10.20.1.206: icmp_seq=15 ttl=64 time=7.97 ms 64 bytes from 10.20.1.206: icmp_seq=16 ttl=64 time=8.72 ms 64 bytes from 10.20.1.206: icmp_seq=17 ttl=64 time=9.23 ms ^C --- 10.20.1.206 ping statistics --- 34 packets transmitted, 28 received, 17.6471% packet loss, time 3328ms rtt min/avg/max/mdev = 1.396/6.266/9.590/2.805 ms Does any one get this issue ? Please help me. Thanks

3 9

[ops]
by Chris Morgan 07 Sep '23

07 Sep '23

I'm trying to reanimate the openstack ops meetups. I moved the social media account used for notifications from X to mastodon (using the fosstodon instance). Please follow there if interested (link below). I also posted a similar announcement to the openstack group on linkedin. One of the first tasks (besides reforming the team) is finding which communications channels actually work. https://fosstodon.org/@osopsmeetup The old account was here https://twitter.com/osopsmeetup now "sepia-tinted" because it's just historical. Cheers, Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[i18n] New I18n SIG Core
by Ian Y. Choi 07 Sep '23

07 Sep '23

Hello, I would like to happily announce a new i18n core: Seongsoo Cho, considering his notable contributions on I18n with volunteering: 1. PDF Docs contribution with I18n issues (font issues, latex with Asian languages) [1] 2. Zanata to Weblate migration volunteering effort [2] 3. Participation of I18n SIG PTG on March [3] and Forum on May [4] Also, he is active on #openstack-i18n IRC channel (nick name: "seongsoocho"). All, please welcome Seongsoo as I18n-core. I am looking forward to more active contributions from him and interactions with other wonderful I18n members and OpenStack team! Thank you, /Ian [1] https://review.opendev.org/q/topic:bp%252Fbuild-pdf-from-rst-guides [2] https://lists.openstack.org/pipermail/openstack-i18n/2022-October/003568.ht… [3] https://etherpad.opendev.org/p/march2023-ptg-i18n [4] https://etherpad.opendev.org/p/vancouver-2023-i18n-forum

4 3

[Glance] Cancelling Weekly Meeting 7th Sept
by Pranali Deore 07 Sep '23

07 Sep '23

Hello, I won't be able to chair today's meeting, my kid is not well, going to take him to the doctor. so cancelling the meeting for this week. Let's meet next week ! Thanks, Pranali D

1 0

Windows Image failed to access network
by Richard Buregeya 07 Sep '23

07 Sep '23

Hello Team, I created the windows image and added the N/w drivers, but it can't be able to get from DHCP. Any idea? Regards Richard.

2 1