August 2021 - openstack-discuss - lists.openstack.org

[PTLs][release] Xena Cycle Highlights
by helena＠openstack.org 31 Aug '21

31 Aug '21

Hello Everyone! The deadline for cycle highlights is just a few days away, on September 3 [1]. Please be sure to get those in. I look forward to seeing what all you all have accomplished this release! How To Reminder: ------------------------- Simply add them to the deliverables/$RELEASE/$PROJECT.yaml in the openstack/releases repo like this: cycle-highlights: - Introduced new service to use unused host to mine bitcoin. The formatting options for this tag are the same as what you are probably used to with Reno release notes. Also, you can check on the formatting of the output by either running locally: tox -e docs And then checking the resulting doc/build/html/$RELEASE/highlights.html file or the output of the build-openstack-sphinx-docs job under html/$RELEASE/highlights.html. Feel free to add Kendall Nelson (diablo_rojo) as a reviewer on your patches. Thanks :) -Helena Spease (hspease) [1] [ https://releases.openstack.org/xena/schedule.html ]( https://releases.openstack.org/xena/schedule.html )

1 0

[kolla][wallaby] hacluster does not work anymore after update
by Ignazio Cassano 30 Aug '21

30 Aug '21

Hello All, I have just update kolla wallaby with images released 27 hours ago but hacluster stopped working with them. TASK [hacluster : Ensure stonith is disabled] fails On the first controller pacemaker.log reports: https://paste.openstack.org/show/808400/ I have not changed configurations bevort updating . Thanks Ignazio

2 2

[nova][dev] Improved scheduling error messages
by Brito, Hugo Nicodemos 30 Aug '21

30 Aug '21

Hi, In a prototype, we have improved Nova's scheduling error messages. This helps both developers and end users better understand the scheduler problems that occur on creation of an instance. When a scheduler error happens during instance creation via the nova upstream, we get the following message on the Overview tab (Horizon dashboard): "No valid host was found." This doesn't give us enough information about what really happened, so our solution was to add more details on the instance's overview page, e.g.: **Fault:Message** attribute provides a summary of why each host can not satisfy the instance’s resource requirements, e.g. for controller-0, it indicates “No valid host was found. Not enough host cell CPUs to fit instance cell” (where cell is a numa-node or socket). **Fault:Details** attribute provides even more detail for each individual host, for example it shows that the instance “required” 2 CPU cores and shows the “actual” CPU cores available on each “numa” node: “actual:0, numa:1” and “actual:1, numa:0”. These details are also present using the OpenStack CLI, in the _fault_ attribute: - openstack server show <instance> With that in mind, we'd like to know if you are open to consider such a change. We are willing to submit a spec and upstream that implementation. Regards, - nicodemos

3 2

RabbitMQ performance improvement for single-control node
by Eugen Block 30 Aug '21

30 Aug '21

Hi *, I would greatly appreciate if anyone could help identifying some tweaks, if possible at all. This is a single control node environment with 19 compute nodes which are heavily used. It's still running on Pike and we won't be able to update until complete redeployment. I was able to improve the performance a lot by increasing the number of workers etc., but to me it seems the next bottleneck is rabbitmq. The rabbit process shows heavy CPU usage, almost constantly around 150% according to 'top' and I see heartbeat errors in the logs (e.g. cinder). Since there are so many config options I haven't touched them yet. Reading some of the tuning docs often refer to HA environments which is not the case here. I can paste some of the config snippets, maybe that already helps identifying anything: # nova.conf [oslo_messaging_rabbit] pool_max_size = 1600 pool_max_overflow = 1600 pool_timeout = 120 rpc_response_timeout = 120 rpc_conn_pool_size = 600

3 4

[glance][interop] Yoga PTG request
by Arkady Kanevsky 30 Aug '21

30 Aug '21

Glance, team, we would like to add a 20 minutes chat between Glance and Interop to discuss latest interop guidelines, and any changes in glance testing. Please, provide a pointer to etherpad and I will add it to the agenda. Thanks, Arkady -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456

2 1

[neutron] Bug deputy report - week of Aug 23th
by Hongbin Lu 29 Aug '21

29 Aug '21

Hi, I was Neutron bug deputy last week. Below is the report: Critical: * https://bugs.launchpad.net/neutron/+bug/1940905 openstack-tox-py36-with-neutron-lib-master is broken Medium: * https://bugs.launchpad.net/neutron/+bug/1941537 Neutron Policy Engine issues with PUT/Update * https://bugs.launchpad.net/neutron/+bug/1941902 is_lsp_router_port() doesn't consider all router device owners Triaging In Progress: * https://bugs.launchpad.net/neutron/+bug/1940950 [ovn] neutron api worker gets overloaded processing chassis_private updates Best regards, Hongbin

1 0

[kolla][operators] Critical regression in Wallaby
by Radosław Piliszek 29 Aug '21

29 Aug '21

Dear Operators of Kolla-based deployments, There is a critical regression in current Kolla Ansible Wallaby code that results in an environment that shuts down VMs on each libvirtd container stop or restart on non-cgroupsv2 distros (so CentOS, Ubuntu and Debian Buster but not Debian Bullseye). [1] The fix is already available. [2] Please apply it to your Kolla Ansible installation if you are using Wallaby. Do note the fix only applies after redeploying which means redeployment action will still trigger the buggy behaviour that once! What to do if you have already deployed Wallaby? First of all, make sure you don't accidentally take an action that stops nova_libvirt (including restarts: both manual and those applied by Kolla Ansible due to user-requested changes). Please apply the patch above but don't rush with redeploying! Redeploy each compute node separately (or in batches if you prefer) - using --limit commandline parameter - and always make sure you have first migrated relevant VMs out of the nodes that are going to get nova_libvirt restarted. This way you can safely fix an existing deployment. We will be working on improving the testing to avoid such issues in the future. Acknowledgements Thanks to Ignazio Cassano for noticing and reporting the issue. I have triaged and analysed it, proposing a fix afterwards. [1] https://bugs.launchpad.net/kolla-ansible/+bug/1941706 [2] https://review.opendev.org/c/openstack/kolla-ansible/+/806476 -yoctozepto

1 0

Why the mariadb container always restart ?
by Tommy Sway 28 Aug '21

28 Aug '21

Hi: The system is broken down for the power, and I restart the whole system, but the mariadb container always restart, about one minite restart once. Ant this is the log : 2021-08-28 0:34:51 0 [Note] WSREP: (a8e9005b, 'tcp://10.10.10.63:4567') turning message relay requesting off 2021-08-28 0:35:03 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():160 2021-08-28 0:35:03 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -110 (Connection timed out) 2021-08-28 0:35:03 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1475: Failed to open channel 'openstack' at 'gcomm://10.10.10.61:4567,10.10.10.62:4567,10.10.10.63:4567': -110 (Connection timed out) 2021-08-28 0:35:03 0 [ERROR] WSREP: gcs connect failed: Connection timed out 2021-08-28 0:35:03 0 [ERROR] WSREP: wsrep::connect(gcomm://10.10.10.61:4567,10.10.10.62:4567,10.10.10.63:4567) failed: 7 2021-08-28 0:35:03 0 [ERROR] Aborting 210828 00:35:05 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended 210828 00:35:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ 210828 00:35:08 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//control03-recover.pid' 210828 00:35:12 mysqld_safe WSREP: Recovered position 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:140403 2021-08-28 0:35:12 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2021-08-28 0:35:12 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 2021-08-28 0:35:12 0 [Note] /usr/libexec/mysqld (mysqld 10.3.28-MariaDB-log) starting as process 258 ... 2021-08-28 0:35:12 0 [Note] WSREP: wsrep_load(): Galera 3.32(rXXXX) by Codership Oy info(a)codership.com <mailto:info@codership.com> loaded successfully. 2021-08-28 0:35:12 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-08-28 0:35:12 0 [Note] WSREP: Found saved state: 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:-1, safe_to_bootstrap: 1 2021-08-28 0:35:12 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.10.10.63; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.10.10.63:4567; gmcast.segment = 0; gmc 2021-08-28 0:35:12 0 [Note] WSREP: GCache history reset: 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:0 -> 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:140403 2021-08-28 0:35:12 0 [Note] WSREP: Assign initial position for certification: 140403, protocol version: -1 2021-08-28 0:35:12 0 [Note] WSREP: wsrep_sst_grab() 2021-08-28 0:35:12 0 [Note] WSREP: Start replication 2021-08-28 0:35:12 0 [Note] WSREP: Setting initial position to 5bdd8d83-05a4-11ec-adfd-ae6e7e26deb9:140403 2021-08-28 0:35:12 0 [Note] WSREP: protonet asio version 0 2021-08-28 0:35:12 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-08-28 0:35:12 0 [Note] WSREP: backend: asio 2021-08-28 0:35:12 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-08-28 0:35:12 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2021-08-28 0:35:12 0 [Note] WSREP: restore pc from disk failed 2021-08-28 0:35:12 0 [Note] WSREP: GMCast version 0 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') listening at tcp://10.10.10.63:4567 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') multicast: , ttl: 1 2021-08-28 0:35:12 0 [Note] WSREP: EVS version 0 2021-08-28 0:35:12 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.10.10.61:4567,10.10.10.62:4567,10.10.10.63:4567' 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') connection established to b1ae23e3 tcp://10.10.10.62:4567 2021-08-28 0:35:12 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') turning message relay requesting on, nonlive peers: 2021-08-28 0:35:16 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') turning message relay requesting off 2021-08-28 0:35:16 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') connection established to c296912b tcp://10.10.10.61:4567 2021-08-28 0:35:16 0 [Note] WSREP: (c05f5c52, 'tcp://10.10.10.63:4567') turning message relay requesting on, nonlive peers: 2021-08-28 0:35:18 0 [Note] WSREP: declaring b1ae23e3 at tcp://10.10.10.62:4567 stable 2021-08-28 0:35:18 0 [Note] WSREP: declaring c296912b at tcp://10.10.10.61:4567 stable 2021-08-28 0:35:19 0 [Warning] WSREP: no nodes coming from prim view, prim not possible What's matter with it ? Thanks!

2 5

[horizon][dev] Multiple Login Sessions
by Soares Sarto, Ricardo 28 Aug '21

28 Aug '21

Hi Everyone, Is there any feature to prevent multiple login sessions for the same user in Horizon? That feature could be used to block a second login attempt for the same user when another session already exists or invalidate the the first session right after a second session is opened by the user. Any reference documentation would be very helpful. Kind regards, rsoaress

2 1

[tempest] Tempest gate is blocked by tempest plugins sanity check job
by Ghanshyam Mann 28 Aug '21

28 Aug '21

Hello Everyone, Tempest gate is blocked by tempest plugin sanity job which is failing 100% for x/vmware-nsx-tempest-plugin issue. - https://zuul.opendev.org/t/openstack/builds?job_name=tempest-tox-plugin-san… Please avoid recheck until the below patch to blacklist failing plugin is merged - https://review.opendev.org/c/openstack/tempest/+/806411 -gmann

1 1