- openstack-discuss - lists.openstack.org

[neutron][OpenStack-ansible] Performance issues with trunk ports
by John Bartelme 05 Apr '23

05 Apr '23

Hello, I'm currently experiencing some pretty severe performance issues with my openstack-ansible deployed cluster(yoga) while deploying trunk ports and I'm looking for some help determining what might be the cause of this poor performance. In my simplest case I'm deploying 2 servers each with one trunk port each. The first trunk has 2 subports and the second 6 subports. Both servers also have 3 other regular ports. When deploying the first trunk port its subports are often provisioned quickly and the second trunk port takes anywhere from 30 seconds to 18 minutes. This happens even when I isolate neutron-server to a single physical machine with 44(88 threads) and 256GB ram. Further diagnosis has shown me some things i didn't quite understand. My deployment with OpenStack-ansible deploys neutron-server with 16 uWSGI processes and neutron-rpc-server with 16 rpc workers. However the way that the trunk RPC server is implemented it is only run on the parent RPC thread and instead runs in all of the uWSGI processes as well. This means that most of my trunk RPC calls are being handled by the uWSGI instead of the RPC workers. When the parent RPC thread handles the trunk port creation calls I constantly see creation times of 1-1.5 seconds. I've isolated it so that this thread does all of the trunk RPC calls and this works quite well but this doesn't seem ideal. What could be causing such poor performance in the uWSGI side of the house? I'm having a really hard time getting a good feeling for what might be slowing it down so much. I'm wondering if it could be green thread preemption but I really don't know. I've tried setting 'enable-threads' false for uWSGI but I don't think that is improving performance. Putting the profiled decorator on update_subport_bindings shows different places taking longer every time, but in general a lot of time(tottime, i.e. not subfunction time) spent in webob/dec.py(__call__), paste/urlmap.py(__call__), webob/request.py(call_application),webob/request.py(send). What else can I do to try and look for why this is taking so long? As a side question it seems counterintuitive that the uWSGI handles most of the trunk RPC calls and not the RPC workers? A couple other notes about my environment that could indicate my challenges: I had to disable rabbitmq heartbeats for neutron as they kept not getting sent reliably and connections were terminated. I tried with heartbeat_in_pthread both true and false but still had issues. It looks like nova also sometimes experiences this but not near as often. I was overzealous with my vxlan ranges in my first configuration and gave it a range of 10,000,000 not realizing that would create that many rows in the database. Looking into that I saw that pymysql in my cluster takes 3.5 minutes to retrieve those rows. mysql CLI only takes 4 seconds. Perhaps that is just the overhead of pymysql? I've greatly scaled down the vxlan range now. I'm provisioning the 2 servers with a heat template that contains around 200 custom resources. For 198 of the resources they are set to conditionally not create any OpenStack native resources. Deploying this template of mostly no-op resources still takes about 3 minutes. Horizon works but almost every page load take a few seconds to load. I'm not sure if that is normal or not. Thanks for any help anyone can provide. john

4 8

Does Openstack support the notion of cinder backend in maintenance mode?
by wodel youchi 05 Apr '23

05 Apr '23

Hi, When you have multiple cinder backends, and you need to take down one of them for maintenance purposes, what is the best way to proceed? Does Openstack have the notion of : cinder backend in maintenance mode? Regards.

3 6

[cinder] 2023.2 Bobcat Meeting Etherpad
by Rajat Dhasmana 05 Apr '23

05 Apr '23

Hello Argonauts, As we're done with the 2023.2 Bobcat PTG, we will be moving to a new etherpad for cinder meetings for the 2023.2 Bobcat development cycle. Please add topics (if you have any) to the etherpad for the meeting on 05 April, 2023 (Today). Etherpad: https://etherpad.opendev.org/p/cinder-bobcat-meetings Thanks Rajat Dhasmana

1 0

[manila] Bobcat vPTG summary
by Carlos Silva 04 Apr '23

04 Apr '23

Hello, Zorillas and interested stackers! Thank you for attending last week's PTG. We had productive discussions and came up with good plans for this cycle! Here is a summary of the topics we spoke about during the PTG. The etherpad with all the notes is available in [0]. If you would like to see the recordings for those discussions, they are available on the Manila Youtube channel [1]. - *Share Transfer Future development* - Share transfers feature was introduced in Antelope but it has a couple of possible enhancements, such as: transferring shares that have snapshots, replicated shares, share networks alongside its share servers and shares. - Snapshots shouldn't be very difficult to cover but we would need to keep in mind quota allocations while doing so. - Transferring shares with replicas is the bigger challenge, considering we need to work with modifying the access rules. - The next step would be to work on transferring share networks. - AIs: Haixin plans to work on this feature during Bobcat. - *S-RBAC changes in Bobcat* - For the Bobcat cycle, we will focus on: - Ensure we are landing even more tests and enabling the oslo policy defaults. - Modify the testing we are doing to use the reader role to test read-only API, opposed to the member role. - Audit all cross service interactions and isolate actions performed by the "service" user. - Push for the completion of unit tests for Secure RBAC - AIs: Manila team will continue promoting events to get tests reviewed and implemented - *Cross project discussion with Nova for VirtIO FS * - Manila and Nova team discussed approaches for avoiding shares to be deleted while they are attached to instances. - We have agreed that an admin/service level API for locking/unlocking shares deletion would be the best solution to avoid shares being deleted while mounted. This API would ideally be a 'locked by <consumer>' action instead of a simple boolean - As part of this cross-project effort, the Manila team has also been focusing on getting more APIs available in OpenStack SDK. We have been making great progress with it and we plan to have all APIs nova is using merged in the OpenStack SDK by the bobcat cycle. - AIs: - Manila team to write a spec describing the behavior of the locking/unlocking API for share. - Manila team to focus on having all APIs nova is using in OpenStack SDK by the Bobcat cycle. - *Scalable NFS Ganesha for CephFS - state of the kitchen* - We have shared a status of what was our focus during the previous release and some important bugs that have been fixed. - Updates on moving from a standalone ganesha to a cephadm deployed: - There are three changes being worked on at the moment, after past discussions: - Ensure that the shares access rules will be applied when moving from the old to the new server - Ensure that the share exists in both old and new ganesha servers and to reapply all access rules in both - Change to represent export locations from both old and new export locations while both cephfs_nfs_cluster_id and ganesha_server_ips are present in the backend. - With these three changes we would cover all the scenarios considering release upgrades. - The changes are being tested and need feedback. - A job for automated testing is in progress. We are targeting to have a multi-node CI job that will be capable of deploying ganesha using cephadm and run tests using it. - AI: carloss and gouthamr will pursue closure for the three changes related to the upgrades and ashrodri will continue to work on the multi-node testing job. - *SQLAlchemy 2.0 and other DB challenges/changes* - DB testing is failing sporadically in the CI and we were discussing possible changes to avoid rechecks caused by sporadic failures. - We intend to work on missing indexes for queries, so we can gain performance in the database. - Implementing __repr__ to manila models - Other people are already doing this and this could be of huge help for debugging, as one of the operators suggested in the session - We will investigate and apply this in Manila - AIs: gouthamr will open bugs for tracking the work on the joinedloads and a blueprint for the missing indexes. - Sqlalchemy 2.0 changes have already started and we have a list of changes yet to merge. stephenfin is doing a great work on the changes. - AIs: manila team to review the changes proposed by stephenfin - *Create share from replicated snapshot / keep source share on replica promote* - Users of this feature are looking for a way to test the inactive replicas before actually promoting them and if everything is correct, they continue with the promotion. - One suggestion to get this working would be having a replication pause API, which would make the inactive instance able to be mounted - We do think this feature could be relevant and some suggestions from the Manila team are: - Letting users know if they can pause/unpause replication even before they create their share replicas - Making users able to know about errors in the API (something similar to the check mechanism on security-service-update) - AIs: carthaca and team will work on a spec and will discuss it further with the Manila team. - *Tech debt:* - We brought up the current issues with our CI, mostly caused by tests being very resource intensive. - We discussed approaches for scenario testing and thought it is worth it trying to create containers to mount share during scenario tests instead of spawning VMs. - Also, the audience has mentioned that it is difficult to keep track of all of these issues and it would be better if we had them documented somewhere, and that we could bring these up more often. - Migration to Ubuntu 22 is yet to complete for some jobs due to failures in package installation and the usage of quagga. - AIs: - carloss will work on a wiki with tech debt items and will bring this up more often in the manila weekly meetings, so if people are available in that time frame to work on them, they will have more chances to raise their hands. - carloss and gouthamr will work to fix the issues with jobs yet to migrate to Ubuntu 22 - *Scheduler data placement based on vendor specific tool* - NetApp has a tool called Active IQ that uses machine learning to identify what is the best pool of disks to place a request based on custom parameters (latency, iops and so on) - They are thinking of trying to have this tool interact with Manila, so it would also benefit from their tool when NetApp is the only storage vendor in the cloud. - The suggestion is to have a new weigher that will make calls to the storage based on the request and use Active IQ response to identify where the share should be placed. - Testing this feature could be a challenge, considering that it's not straightforward to foresee what a weigher would return in a given moment - AIs: felipe_rodriguess alongside the NetApp team will work on a spec so this can be further discussed. [0] https://etherpad.opendev.org/p/manila-bobcat-ptg [1] https://www.youtube.com/watch?v=gefbYrzRD04&list=PLnpzT0InFrqDW_djitgCiFzwh… Thank you! carloss

1 0

[neutron] bug deputy report March 27th - April 2nd
by Miguel Lavalle 04 Apr '23

04 Apr '23

Here's last week's bugs deputy report: Medium ====== https://bugs.launchpad.net/neutron/+bug/2012944 neutron-remove-duplicated-port-bindings does not write to stdout. Valid bug. Assigned to mlavalle https://bugs.launchpad.net/neutron/+bug/2013326 Trying to activate port binding as regular user causes error 500. Assigned to slaweq Low: ========== https://bugs.launchpad.net/neutron/+bug/2012912 [neutron-lib][stable] Wallaby CI n-t-p, test "test_qos_dscp_create_and_update" failing. This bug is addressed by https://review.opendev.org/c/openstack/neutron-lib/+/879441. No further action seems likely RFEs ==== https://bugs.launchpad.net/neutron/+bug/2013228 [rfe] Non-admin users unable to create SR-IOV switch dev ports. Discussed during the PTG during the Neutron / Nova session, pending further discussion in drivers meeting https://bugs.launchpad.net/neutron/+bug/2013540 [RFE] Add support for Napatech LinkVirt SmartNICs. Discussed during the PTG, pending further discussion in drivers meeting

1 0

OpenInfra Live - April 6 at 9 a.m. CT / 14:00 UTC
by Kristin Barrientos 04 Apr '23

04 Apr '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack Large Scale SIG. Episode: Large Scale Ops Deep Dive: Société Générale In the "Large Scale Ops Deep Dive" series, a panel of OpenStack operators invites special guests to talk about their deployment and discuss their operations. For this episode, our guest will be Guillaume Allard from Société Générale, one of France’s largest banks. Speakers: Guillaume Allard, Arnaud Morin, Thierry Carrez, Belmiro Moreira, Mohammed Naser and Kristin Barrientos Date and time: April 6, 2023, at 9 a.m. CT (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/live/ZIjpEZSu1gw?feature=share LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:7043641363350908928/ WeChat: recording will be posted on OpenStack WeChat after the live stream Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[horizon][domain][yoga][kolla-ansible] horizon display failed to display domain tab
by James Leong 04 Apr '23

04 Apr '23

Hi all, I am playing around with the domain. I have used kolla-ansible as a deployment tool for deploying the yoga version of OpenStack. However, after deploying a fresh OpenStack, I was not able to view the domain tab on my dashboard. Every time when I want to create a new domain, I have to go to my terminal and create it manually without any GUI. Is there any other configuration I have to enable in the globals.yml file other than setting "horizon_keystone_multidomain: "True"? Thanks for your help James

3 4

[policy][rbac] RBAC 2023.2 Bobcat cycle vPTG discussions summary
by Ghanshyam Mann 04 Apr '23

04 Apr '23

Hello Everyone, We discussed the RBAC goal on Tuesday. I am summarizing the discussion here. Goal document: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… Tracking: https://etherpad.opendev.org/p/rbac-goal-tracking Current progress: ============= Phase-1 (project personas and drop system scope): ------------------------------------------------------------ Projects completed: * Nova * Neutron * Glance * Manila * Ironic (no change needed) * Octavia * Placement * Service that completed Phase 1 in Zed and enabled scope and new defaults by default (enforce_scope=True & enforce_new_defaults=True by default): ** Nova ** Glance Projects in progress: * Cinder (almost completed. not using system scope, but have not added scope=['project'] to the default rules (currently, no scope is specified) * Magnum ** https://review.opendev.org/c/openstack/magnum/+/874945 * Tacker ** https://review.opendev.org/q/topic:bp%252Fimplement-project-personas Pending work (for phase-2|3): * Keystone implements a new default role called manager: ** https://review.opendev.org/c/openstack/keystone/+/822601 * Keystone implements a new default role called service: ** https://review.opendev.org/c/openstack/keystone/+/863420 Phase-2 (service role): -------------------------- In-progress: * Keystone: ** bootstrap support for servicerole: *** https://review.opendev.org/c/openstack/keystone/+/863420 ** bootstrap support for manager role: *** https://review.opendev.org/c/openstack/keystone/+/822601 *Nova ** https://review.opendev.org/c/openstack/nova/+/864594 Other discussion: ============= *Service role We discussed the service role and how the policy will add the service role. It is correct to add service as well as user role as default if that API is supposed to be called by the service as well as the user role. For example, if Manila is talking to Nova, Cinder, or Neutron via APIs, it needs to use a service role to interact, and Nova, Cinder, and Neutron can update such API rules to allow for service roles also. * Update the goal timeline for removing deprecated rules as per the SLURP release. ** Need at least 1 SLURP release between enabling the new default and removal. ** Action: gmann to update this in the goal document. *Testing: ** Manila has a lot of tests and running on the stable release ** Tempest and devstack ready to implement the test ** The current job with nova, cinder, neutron, glance with the scope and new default enable *** https://zuul.openstack.org/builds?job_name=tempest-full-enforce-scope-new-d… * Tempest now has a project reader/member same project_id * Related sessions: ** Tacker *** Title: Secure RBAC: Implement support of project-personas in Tacker [Continue from Antelope release] (manpreetk) *** Etherpad: https://etherpad.opendev.org/p/tacker-bobcat-ptg#L136 ** Glance *** Title: Secure RBAC *** Etherpad: https://etherpad.opendev.org/p/glance-bobcat-ptg#L53 ** Neutron *** Title: (slaweq) Secure RBAC - phase 2 description and review of the existing API calls *** Etherpad: https://etherpad.opendev.org/p/neutron-bobcat-ptg#L358 <feel free to add here if I missed any related sessions> I will continue holding the biweekly meeting to discuss progress and any query on RBAC. - https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[keystone] Weekly Meeting Cancelled (04-Apr 2023)
by Dave Wilde 04 Apr '23

04 Apr '23

Hi folks, I'm cancelling the Keystone weekly meeting this week (04-Apr 2023) due to a conflict that I have. Please reach out to me on IRC or via email if you need anything. Thanks! /Dave

1 0

[neutron][ovn-bgp-agent] Make ovn-bgp-agent a Neutron stadium project
by Rodolfo Alonso Hernandez 04 Apr '23

04 Apr '23

Hello all: During the last PTG, it was discussed and accepted to migrate the ovn-bgp-agent project [1] to the Neutron stadium umbrella. This mail is a heads-up for anyone using it. In the next days (maybe today), Lucas will send the corresponding patches for "governance" and "project-config" repositories in order to start this migration. If you have any questions, please ask them in this email or via IRC in the #openstack-neutron channel. Regards. [1]https://opendev.org/x/ovn-bgp-agent

1 0