The timing result is 1.6 seconds (vs 0.2 on a non-rebuilt region), but the elapsed time is 2.7 seconds (vs 1.2):
Rebuilt region:
$ time openstack user show --timing user +---------------------+------------------------------------------------------------------+ | Field | Value | +---------------------+------------------------------------------------------------------+ | domain_id | <DID> | | email | user@ourdomain.com | | enabled | True | | id | <ID> | | name | user | | options | {} | | password_expires_at | None | +---------------------+------------------------------------------------------------------+
+------------------------------------------------------------------------------------------------------------------+--------------------+ | URL | Seconds | +------------------------------------------------------------------------------------------------------------------+--------------------+ | GET https://api-ext.region.ourdomain.com:5000/v3 | 0.030608 | | POST https://api-ext.region.ourdomain.com:5000/v3/auth/tokens | 0.550521 | | GET https://api-ext.region.ourdomain.com:5000/ | 0.006306 | | GET https://api-ext.region.ourdomain.com:5000/v3/auth/tokens | 0.608377 | | GET https://api-ext.region.ourdomain.com:5000/v3/users/<ID> | 0.458892 | | Total | 1.6547040000000002 | +------------------------------------------------------------------------------------------------------------------+--------------------+
real 0m2.733s user 0m0.982s sys 0m0.105s
Not rebuilt:
$ time openstack user show --timing user +---------------------+------------------------------------------------------------------+ | Field | Value | +---------------------+------------------------------------------------------------------+ | domain_id | <DID> | | email | user@ourdomain.com | | enabled | True | | id | <ID> | | name | user | | options | {} | | password_expires_at | None | +---------------------+------------------------------------------------------------------+
+------------------------------------------------------------------------------------------------------------------+----------+ | URL | Seconds | +------------------------------------------------------------------------------------------------------------------+----------+ | GET https://api-ext.oldregion.ourdomain.com:5000/v3 | 0.026787 | | POST https://api-ext.oldregion.ourdomain.com:5000/v3/auth/tokens | 0.042511 | | GET https://api-ext.oldregion.ourdomain.com:5000/ | 0.00767 | | GET https://api-ext.oldregion.ourdomain.com:5000/v3/auth/tokens | 0.049546 | | GET https://api-ext.oldregion.ourdomain.com:5000/v3/users/<ID> | 0.044296 | | Total | 0.17081 | +------------------------------------------------------------------------------------------------------------------+----------+
real 0m1.176s user 0m0.892s sys 0m0.118s
-----Original Message----- From: Eugen Block eblock@nde.ag Sent: Tuesday, April 2, 2024 12:52 PM To: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: [kolla] [Wallaby] Openstack operations 2x slower after rebuilding from RHEL8/Train to RHEL9/Wallaby
Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi, could you please share the details of (or any other command that takes longer than usual):
openstack user show --timing
Thanks, Eugen
Zitat von Albert Braden ozzzo@yahoo.com:
After rebuilding a couple of our clusters, moving from Train on RHEL8 to Wallaby on RHEL9, we see all openstack operations taking 2x longer. Even simple database reads such as "openstack user show" average 1.5 seconds on old regions and 3 seconds on rebuilt regions. I enabled debug and poked around in the logs but nothing is jumping out at me. I was thinking maybe mariadb needed to be tuned differently from RHEL8->9 but if I log into the mariadb container and query the database it seems equally fast; I can do 100 simple queries in a second. Where should I be looking for the source of this delay?