Open Stack

Hi all,

I am facing an issue with Kilo on Ubuntu Trusty.

I have a tenant (used for benchmarking/stress test) that started ~100k
instances in the last 2 weeks. When running "nova usage" for this
tenant, the following happens:

* nova-api is stuck at 100% for a long time
* as a consequence, nova CLI returns "ERROR (ConnectionRefused):
Unable to establish connection to ..."
* on MySQL slow query log I see there is a query like:

SELECT instance_system_metadata.created_at AS
instance_system_metadata_created_at,
instance_system_metadata.updated_at AS
instance_system_metadata_updated_at,
instance_system_metadata.deleted_at AS
instance_system_metadata_deleted_at, instance_system_metadata.deleted
AS instance_system_metadata_deleted, instance_system_metadata.id AS
instance_system_metadata_id, instance_system_metadata.`key` AS
instance_system_metadata_key, instance_system_metadata.value AS
instance_system_metadata_value, instance_system_metadata.instance_uuid
AS instance_system_metadata_instance_uuid
FROM instance_system_metadata
WHERE instance_system_metadata.deleted = 0 AND
 instance_system_metadata.instance_uuid IN (<list of ~100k UUID>)

which took 1.8 seconds.

Should I open a bug on this?

Additional information:

I'm using a 3-nodes galera cluster 1x active/2x passive, HAProxy,
3-nodes rabbitmq, 3 controller nodes running all services but neutron.

When logging in from Horizon, login is very slow, and I get an error
"Error: Unable to retrieve usage information.".

On a side note: I've noticed this because logging in through HAProxy
was not working, probably because haproxy timeout is lower than the
time took by horizon to realize that nova-api was not responding.

.a.

-- 
antonio.s.messina at gmail.com
antonio.messina at uzh.ch                     +41 (0)44 635 42 22
S3IT: Service and Support for Science IT   http://www.s3it.uzh.ch/
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland