August 2021 - openstack-discuss - lists.openstack.org

RabbitMQ performance improvement for single-control node
by Eugen Block 30 Aug '21

30 Aug '21

Hi *, I would greatly appreciate if anyone could help identifying some tweaks, if possible at all. This is a single control node environment with 19 compute nodes which are heavily used. It's still running on Pike and we won't be able to update until complete redeployment. I was able to improve the performance a lot by increasing the number of workers etc., but to me it seems the next bottleneck is rabbitmq. The rabbit process shows heavy CPU usage, almost constantly around 150% according to 'top' and I see heartbeat errors in the logs (e.g. cinder). Since there are so many config options I haven't touched them yet. Reading some of the tuning docs often refer to HA environments which is not the case here. I can paste some of the config snippets, maybe that already helps identifying anything: # nova.conf [oslo_messaging_rabbit] pool_max_size = 1600 pool_max_overflow = 1600 pool_timeout = 120 rpc_response_timeout = 120 rpc_conn_pool_size = 600

3 4

[glance][interop] Yoga PTG request
by Arkady Kanevsky 29 Aug '21

29 Aug '21

Glance, team, we would like to add a 20 minutes chat between Glance and Interop to discuss latest interop guidelines, and any changes in glance testing. Please, provide a pointer to etherpad and I will add it to the agenda. Thanks, Arkady -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456

2 1

[neutron] Bug deputy report - week of Aug 23th
by Hongbin Lu 29 Aug '21

29 Aug '21

Hi, I was Neutron bug deputy last week. Below is the report: Critical: * https://bugs.launchpad.net/neutron/+bug/1940905 openstack-tox-py36-with-neutron-lib-master is broken Medium: * https://bugs.launchpad.net/neutron/+bug/1941537 Neutron Policy Engine issues with PUT/Update * https://bugs.launchpad.net/neutron/+bug/1941902 is_lsp_router_port() doesn't consider all router device owners Triaging In Progress: * https://bugs.launchpad.net/neutron/+bug/1940950 [ovn] neutron api worker gets overloaded processing chassis_private updates Best regards, Hongbin

1 0

[kolla][operators] Critical regression in Wallaby
by Radosław Piliszek 28 Aug '21

28 Aug '21

Dear Operators of Kolla-based deployments, There is a critical regression in current Kolla Ansible Wallaby code that results in an environment that shuts down VMs on each libvirtd container stop or restart on non-cgroupsv2 distros (so CentOS, Ubuntu and Debian Buster but not Debian Bullseye). [1] The fix is already available. [2] Please apply it to your Kolla Ansible installation if you are using Wallaby. Do note the fix only applies after redeploying which means redeployment action will still trigger the buggy behaviour that once! What to do if you have already deployed Wallaby? First of all, make sure you don't accidentally take an action that stops nova_libvirt (including restarts: both manual and those applied by Kolla Ansible due to user-requested changes). Please apply the patch above but don't rush with redeploying! Redeploy each compute node separately (or in batches if you prefer) - using --limit commandline parameter - and always make sure you have first migrated relevant VMs out of the nodes that are going to get nova_libvirt restarted. This way you can safely fix an existing deployment. We will be working on improving the testing to avoid such issues in the future. Acknowledgements Thanks to Ignazio Cassano for noticing and reporting the issue. I have triaged and analysed it, proposing a fix afterwards. [1] https://bugs.launchpad.net/kolla-ansible/+bug/1941706 [2] https://review.opendev.org/c/openstack/kolla-ansible/+/806476 -yoctozepto

1 0

Why the mariadb container always restart ?
by Tommy Sway 28 Aug '21

28 Aug '21

Hi: The system is broken down for the power, and I restart the whole system, but the mariadb container always restart, about one minite restart once. Ant this is the log : 2021-08-28 0:34:51 0 [Note] WSREP: (a8e9005b, 'tcp://10.10.10.63:4567') turning message relay requesting off 2021-08-28 0:35:03 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():160 2021-08-28 0:35:03 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -110 (Connection timed out) 2021-08-28 0:35:03 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1475: Failed to open channel 'openstack' at 'gcomm://10.10.10.61:4567,10.10.10.62:4567,10.10.10.63:4567': -110 (Connection timed out) 2021-08-28 0:35:03 0 [ERROR] WSREP: gcs connect failed: Connection timed out 2021-08-28 0:35:03 0 [ERROR] WSREP: wsrep::connect(gcomm://10.10.10.61:4567,10.10.10.62:4567,10.10.10.63:4567) failed: 7 2021-08-28 0:35:03 0 [ERROR] Aborting 210828 00:35:05 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210828 00:35:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210828 00:35:08 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//control03-recover.pid' 210828 00:35:12 mysqld_safe WSREP: Recovered position 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:140403 2021-08-28 0:35:12 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-08-28 0:35:12 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 2021-08-28 0:35:12 0 [Note] /usr/libexec/mysqld (mysqld 10.3.28-MariaDB-log) starting as process 258 ... 2021-08-28 0:35:12 0 [Note] WSREP: wsrep_load(): Galera 3.32(rXXXX) by Codership Oy info(a)codership.com <mailto:info@codership.com> loaded successfully. 2021-08-28 0:35:12 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-08-28 0:35:12 0 [Note] WSREP: Found saved state: 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:-1, safe_to_bootstrap: 1 2021-08-28 0:35:12 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.10.10.63; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.10.10.63:4567; gmcast.segment = 0; gmc 2021-08-28 0:35:12 0 [Note] WSREP: GCache history reset: 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:0 -> 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:140403 2021-08-28 0:35:12 0 [Note] WSREP: Assign initial position for certification: 140403, protocol version: -1 2021-08-28 0:35:12 0 [Note] WSREP: wsrep_sst_grab() 2021-08-28 0:35:12 0 [Note] WSREP: Start replication 2021-08-28 0:35:12 0 [Note] WSREP: Setting initial position to 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:140403 2021-08-28 0:35:12 0 [Note] WSREP: protonet asio version 0 2021-08-28 0:35:12 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-08-28 0:35:12 0 [Note] WSREP: backend: asio 2021-08-28 0:35:12 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-08-28 0:35:12 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2021-08-28 0:35:12 0 [Note] WSREP: restore pc from disk failed 2021-08-28 0:35:12 0 [Note] WSREP: GMCast version 0 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') listening at tcp://10.10.10.63:4567 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') multicast: , ttl: 1 2021-08-28 0:35:12 0 [Note] WSREP: EVS version 0 2021-08-28 0:35:12 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.10.10.61:4567,10.10.10.62:4567,10.10.10.63:4567' 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') connection established to b1ae23e3 tcp://10.10.10.62:4567 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') turning message relay requesting on, nonlive peers: 2021-08-28 0:35:16 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') turning message relay requesting off 2021-08-28 0:35:16 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') connection established to c296912b tcp://10.10.10.61:4567 2021-08-28 0:35:16 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') turning message relay requesting on, nonlive peers: 2021-08-28 0:35:18 0 [Note] WSREP: declaring b1ae23e3 at tcp://10.10.10.62:4567 stable 2021-08-28 0:35:18 0 [Note] WSREP: declaring c296912b at tcp://10.10.10.61:4567 stable 2021-08-28 0:35:19 0 [Warning] WSREP: no nodes coming from prim view, prim not possible What's matter with it ? Thanks!

2 5

[horizon][dev] Multiple Login Sessions
by Soares Sarto, Ricardo 27 Aug '21

27 Aug '21

Hi Everyone, Is there any feature to prevent multiple login sessions for the same user in Horizon? That feature could be used to block a second login attempt for the same user when another session already exists or invalidate the the first session right after a second session is opened by the user. Any reference documentation would be very helpful. Kind regards, rsoaress

2 1

[tempest] Tempest gate is blocked by tempest plugins sanity check job
by Ghanshyam Mann 27 Aug '21

27 Aug '21

Hello Everyone, Tempest gate is blocked by tempest plugin sanity job which is failing 100% for x/vmware-nsx-tempest-plugin issue. - https://zuul.opendev.org/t/openstack/builds?job_name=tempest-tox-plugin-san… Please avoid recheck until the below patch to blacklist failing plugin is merged - https://review.opendev.org/c/openstack/tempest/+/806411 -gmann

1 1

[all][tc] What's happening in Technical Committee: summary 27th Aug, 21: Reading: 5 min
by Ghanshyam Mann 27 Aug '21

27 Aug '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC held this week's IRC meeting on Aug 26th Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the full log @ - https://meetings.opendev.org/meetings/tc/2021/tc.2021-08-26-15.00.log.html * We will have next week's meeting (Video call) on Sept 2nd, Thursday 15:00 UTC, feel free the topic in agenda [1] by Sept 1st. 2. What we completed this week: ========================= * "Secure RBAC" is selected as one of the community wide goal for Yoga cycle[2]. * TC discussed and agreed on having monthly (1st meeting of every month) Video meeting. First Video meeting on 2nd Sept (next week meeting I will send the joining detail next week). The rest of other weeks meeting will be on IRC. 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[3] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Four open reviews for ongoing activities[4]. Combined PTL/TC Election Season for Yoga ---------------------------------------------------- * TC election results are out (no election this time)[5]. * PTL election are going on, Cyborg project will have election. New project application: 'Venus' -------------------------------------- * There is a new project application called 'Venus': Log management service[6]. * Governance patch is also up with all the details[7] and discussion going on. * If you have any queries on this, this is the right time to ask on the ML thread or on Gerrit. Collecting Pain points from projects/SIGs/pop-up ----------------------------------------------------------- * We have lot of pain points from different projects as listed in etherpad[8]. * As next step, we are going to identify the common pain point which can be targeted as community-wide goal. * Join us in discussion in next week meeting and do not miss to read Rico detail email on this[9]. Project updates ------------------- * Retiring js-openstack-lib [10] * Retire puppet-monasca[11] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [12]. * Current status: "Secure RBAC" is selected for Yoga cycle[13]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[14], feel free to add any topic you would like to discuss. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[15], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[16]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [17] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [18] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/803783 [3] https://etherpad.opendev.org/p/tc-xena-tracker [4] https://review.opendev.org/q/project:openstack/governance+status:open [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024403.h… [6] http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019748.… [7] https://review.opendev.org/c/openstack/governance/+/804824 [8] https://etherpad.opendev.org/p/pain-point-elimination [9] http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024487.h… [9] https://review.opendev.org/c/openstack/governance/+/798540 [10] https://review.opendev.org/c/openstack/governance/+/805105 [12] https://etherpad.opendev.org/p/y-series-goals [13] https://review.opendev.org/c/openstack/governance/+/803783 [14] https://etherpad.opendev.org/p/tc-yoga-ptg [15] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [16] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [17] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [18] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

1 0

[release] Release countdown for week R-5, Aug 30 - Sep 03
by Herve Beraud 27 Aug '21

27 Aug '21

Development Focus ----------------- We are getting close to the end of the Xena cycle! Next week on 02 September, 2021 is the Xena-3 milestone, also known as feature freeze. It's time to wrap up feature work in the services and their client libraries, and defer features that won't make it to the Yoga cycle. General Information ------------------- This coming week is the deadline for client libraries: their last feature release needs to happen before "Client library freeze" on 02 September, 2021. Only bugfix releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/xena branching request with the review. As an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/… 02 September, 2021 is also the deadline for feature work in all OpenStack deliverables following the cycle-with-rc model. To help those projects produce a first release candidate in time, only bugfixes should be allowed in the master branch beyond this point. Any feature work past that deadline has to be raised as a Feature Freeze Exception (FFE) and approved by the team PTL. Finally, feature freeze is also the deadline for submitting a first version of your cycle-highlights. Cycle highlights are the raw data that helps shape what is communicated in press releases and other release activity at the end of the cycle, avoiding direct contacts from marketing folks. See https://docs.openstack.org/project-team-guide/release-management.html#cycle… for more details. Upcoming Deadlines & Dates -------------------------- Xena-3 milestone (feature freeze): 02 September, 2021 (R-5 week) RC1 deadline: The week of 13 September, 2021 (R-3 week) Final RC deadline: The week of 27 September, 2021 (R-1 week) Final Xena release: 06 October, 2021 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

Support for ONOS
by Hrishikesh Karanjikar 27 Aug '21

27 Aug '21

Hi, Does Openstack latest release support ONOS? -- Regards, Hrishikesh Karanjikar

3 10