Open Stack

Hi,
When testing high load scenarios, e.g. issuing 100 volume attachments, we are running into timeout problems between nova, cinder and the centralized storage backend.
Has anybody experienced similar problems?
/Tobi


https://blueprints.launchpad.net/nova/+spec/volume-status-polling
https://blueprints.launchpad.net/cinder/+spec/volume-status-polling


From: Tobias Engelbert [mailto:tobias.engelbert at ericsson.com]
Sent: Monday, November 03, 2014 11:33 PM
To: openstack-dev at lists.openstack.org
Subject: [openstack-dev] [cinder][nova] Timeout problems

Hi,
Parallel volume operations lead to inconsistencies between the OpenStack database and the deployed view on a centralized storage backend.

When performing multiple volume operation on a centralized storage backend it can come to timeouts on the OpenStack side. These timeouts can be the RPC timeout or e.g. in high availability scenarios, the HA proxy timeout.
When nova wants to attach a volume, it triggers the status change from available to attaching and sends initialize_connection via cinderclient to cinder API via the REST API. Cinder API performs a synchronous CALL to cinder volume, then via the driver the centralized storage backend is contacted. When now a timeout occurs, nova triggers the database to change the volume status from attaching to available. Meanwhile the centralized storage backend performs, what was originally requested. Here we can have a mismatch between database and the real view of the centralized storage backend.

I would be curious if this behavior is also seen by others and would like to discuss possible solution.
/Tobi

See also
https://blueprints.launchpad.net/cinder/+spec/volume-status-polling

11<javascript:;>


https://review.openstack.org/#/c/132225/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141126/071963cc/attachment.html>