[openstack-dev] [nova] [cinder] Timeout problems
tobias.engelbert at ericsson.com
Wed Nov 26 10:17:00 UTC 2014
When testing high load scenarios, e.g. issuing 100 volume attachments, we are running into timeout problems between nova, cinder and the centralized storage backend.
Has anybody experienced similar problems?
From: Tobias Engelbert [mailto:tobias.engelbert at ericsson.com]
Sent: Monday, November 03, 2014 11:33 PM
To: openstack-dev at lists.openstack.org
Subject: [openstack-dev] [cinder][nova] Timeout problems
Parallel volume operations lead to inconsistencies between the OpenStack database and the deployed view on a centralized storage backend.
When performing multiple volume operation on a centralized storage backend it can come to timeouts on the OpenStack side. These timeouts can be the RPC timeout or e.g. in high availability scenarios, the HA proxy timeout.
When nova wants to attach a volume, it triggers the status change from available to attaching and sends initialize_connection via cinderclient to cinder API via the REST API. Cinder API performs a synchronous CALL to cinder volume, then via the driver the centralized storage backend is contacted. When now a timeout occurs, nova triggers the database to change the volume status from attaching to available. Meanwhile the centralized storage backend performs, what was originally requested. Here we can have a mismatch between database and the real view of the centralized storage backend.
I would be curious if this behavior is also seen by others and would like to discuss possible solution.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev