slow server list

Eugen Block eblock at nde.ag
Wed Feb 8 10:25:00 UTC 2023


Hi,

the mysql approach is much faster, of course. But it also could be a  
memcached issue if the requests take too long. In a customer cluster  
with more than 400 VMs I get a response in under 7 seconds, 6 seconds  
for the long nova request:

| GET https://control:8774/v2.1/servers/detail?all_tenants=True     |  
6.023134 |
| Total                                                             |  
6.978378 |

In a different cluster with around 200 VMs the request comes back  
after less than 4 seconds:

| GET http://controller:8774/v2.1/servers/detail?all_tenants=True   |   
          2.263082 |
| Total                                                             |  
3.6819349999999997 |

It appears to be proportional, so in a cluster with more than 1000 VMs  
it seems reasonable to assume that it could take that long.
The worst thing we had was a few years back when listing all VMs from  
all projects via dashboard timed out after one minute. That's when we  
took the time to analyze and noticed a misconfiguration of memcached.  
But I'm not sure how much room for improvement there is with so many  
VMs, maybe other operators can report their experiences as well.

Regards,
Eugen

Zitat von Satish Patel <satish.txt at gmail.com>:

> I am experiencing the same behavior on all of my clouds. It's normal I
> believe because behind the command so many api calls go out to construct
> data. We have 500 vms in single projects so I wrote my own wrapper script
> to talk to mariadb and fetch that information directly from DB. This is not
> the best solution but works for me and now i can pull out any records
> within microseconds.
>
> $ mysql -h 172.28.15.79 -u user1 --password=<PASSWORD> nova -e "select
> display_name as
> hostname,vm_state,node,availability_zone,vcpus,memory_mb,root_gb from
> instances WHERE (vm_state='active' OR vm_state='stopped');"
>
>
>
> On Tue, Feb 7, 2023 at 2:07 PM Parsa Aminian <p.aminian.server at gmail.com>
> wrote:
>
>> hello
>> On openstack wallaby version openstack server list result too slow :
>> | GET http://example.com:35357/v3
>>     |  0.040385 |
>> | POST http://example.com:35357/v3/auth/tokens
>>      |   0.49567 |
>> | GET http://example.com:9292/
>>            |  0.011694 |
>> | GET http://example.com:8774/v2.1/servers/detail?all_tenants=True
>>            | 16.032512 |
>> | GET http://example.com:9292/v2/images
>>           |  0.241679 |
>> | GET
>> http://example.com:9292/v2/images?marker=b17469eb-4548-4fbb-b871-5880323a7a76
>> |  0.205339 |
>> | GET
>> http://example.com:9292/v2/images?marker=eb00ee00-4401-4e59-bf92-602be2e45c4a
>> |  0.204854 |
>> | GET
>> http://example.com:9292/v2/images?marker=edf7d33a-286a-4a00-a8d8-4b0055d06093
>> |  0.203524 |
>> | GET
>> http://example.com:9292/v2/images?marker=75ab289d-90ce-496c-97a4-6fdd247c862a
>> |  0.215708 |
>> | GET
>> http://example.com:9292/v2/images?marker=b4eaf24c-94dc-4f2f-bd54-3eecbf290d75
>> |   0.28073 |
>> | GET http://example.com:8774/v2.1/flavors/detail?is_public=None
>>            |  0.641211 |
>> | Total
>>             | 18.573306
>>
>> as you see nova api takes 16 seconds to load . I have one project with
>> 1100 instance on it
>>






More information about the openstack-discuss mailing list