That's plausible; we did have some trouble with memcached/mcrouter after the rebuild and we had to tweak our mcrouter config, but I'm having trouble getting good logs. I set designate_logging_debug: "True" and did a full deploy but I don't see any logs for memcached and mcrouter doesn't log anything interesting.
On Tuesday, April 2, 2024 at 03:50:53 PM EDT, Eugen Block <eblock@nde.ag> wrote:


Hm, we’ve seen something similar in the past when our memcached config 
was not correct, although I’m not sure if keystone was the culprit or 
if it was a different service. Anyway, I would probably start to 
compare memcached configs.

Zitat von "Braden, Albert" <abraden@verisign.com>:

> The timing result is 1.6 seconds (vs 0.2 on a non-rebuilt region), 
> but the elapsed time is 2.7 seconds (vs 1.2):
>
> Rebuilt region:
>
> $ time openstack user show --timing user
> +---------------------+------------------------------------------------------------------+
> | Field              | Value                                       
>                    |
> +---------------------+------------------------------------------------------------------+
> | domain_id          | <DID>                                |
> | email              | user@ourdomain.com                           
>                  |
> | enabled            | True                                         
>                    |
> | id                  | <ID> |
> | name                | user                                         
>                  |
> | options            | {}                                           
>                    |
> | password_expires_at | None                                         
>                    |
> +---------------------+------------------------------------------------------------------+
>
> +------------------------------------------------------------------------------------------------------------------+--------------------+
> | URL                                                               
>                                              |            Seconds |
> +------------------------------------------------------------------------------------------------------------------+--------------------+
> | GET https://api-ext.region.ourdomain.com:5000/v3                   
>                                                      |           
> 0.030608 |
> | POST https://api-ext.region.ourdomain.com:5000/v3/auth/tokens     
>                                                      |           
> 0.550521 |
> | GET https://api-ext.region.ourdomain.com:5000/                     
>                                                      |           
> 0.006306 |
> | GET https://api-ext.region.ourdomain.com:5000/v3/auth/tokens       
>                                                      |           
> 0.608377 |
> | GET https://api-ext.region.ourdomain.com:5000/v3/users/<ID> |     
>      0.458892 |
> | Total                                                             
>                                              | 1.6547040000000002 |
> +------------------------------------------------------------------------------------------------------------------+--------------------+
>
> real    0m2.733s
> user    0m0.982s
> sys    0m0.105s
>
> Not rebuilt:
>
> $ time openstack user show --timing user
> +---------------------+------------------------------------------------------------------+
> | Field              | Value                                       
>                    |
> +---------------------+------------------------------------------------------------------+
> | domain_id          | <DID>                                |
> | email              | user@ourdomain.com                           
>                  |
> | enabled            | True                                         
>                    |
> | id                  | <ID> |
> | name                | user                                         
>                  |
> | options            | {}                                           
>                    |
> | password_expires_at | None                                         
>                    |
> +---------------------+------------------------------------------------------------------+
>
> +------------------------------------------------------------------------------------------------------------------+----------+
> | URL                                                               
>                                              |  Seconds |
> +------------------------------------------------------------------------------------------------------------------+----------+
> | GET https://api-ext.oldregion.ourdomain.com:5000/v3               
>                                                        | 0.026787 |
> | POST https://api-ext.oldregion.ourdomain.com:5000/v3/auth/tokens   
>                                                        | 0.042511 |
> | GET https://api-ext.oldregion.ourdomain.com:5000/                 
>                                                        |  0.00767 |
> | GET https://api-ext.oldregion.ourdomain.com:5000/v3/auth/tokens   
>                                                        | 0.049546 |
> | GET https://api-ext.oldregion.ourdomain.com:5000/v3/users/<ID> | 0.044296 |
> | Total                                                             
>                                              |  0.17081 |
> +------------------------------------------------------------------------------------------------------------------+----------+
>
> real    0m1.176s
> user    0m0.892s
> sys    0m0.118s
>
> -----Original Message-----
> From: Eugen Block <eblock@nde.ag>
> Sent: Tuesday, April 2, 2024 12:52 PM
> To: openstack-discuss@lists.openstack.org
> Subject: [EXTERNAL] Re: [kolla] [Wallaby] Openstack operations 2x 
> slower after rebuilding from RHEL8/Train to RHEL9/Wallaby
>
> Caution: This email originated from outside the organization. Do not 
> click links or open attachments unless you recognize the sender and 
> know the content is safe.
>
> Hi, could you please share the details of (or any other command that 
> takes longer than usual):
>
> openstack user show --timing
>
> Thanks,
> Eugen
>
> Zitat von Albert Braden <ozzzo@yahoo.com>:
>
>> After rebuilding a couple of our clusters, moving from Train on
>> RHEL8 to Wallaby on RHEL9, we see all openstack operations taking 2x
>> longer. Even simple database reads such as "openstack user show"
>> average 1.5 seconds on old regions and 3 seconds on rebuilt regions.
>> I enabled debug and poked around in the logs but nothing is jumping
>> out at me. I was thinking maybe mariadb needed to be tuned differently
>> from RHEL8->9 but if I log into the mariadb container and query the
>> database it seems equally fast; I can do 100 simple queries in a
>> second. Where should I be looking for the source of this delay?