Open Stack

Hi all,
During the analysis of the following bug:
https://bugs.launchpad.net/heat/+bug/1418878
i figured out that orchestration engine doesn't work properly in some cases.
The case is the following:
trying to delete the same stack with resources n times in series.
It might happen if the stack deleting takes much time and a user is sending
the second delete request again.
Orchestration engine behavior is the following:
1) When first stack-delete command comes to heat service
it acquires the stack lock and sends delete request for resources
to other clients.
Unfortunately, the command does not start to delete resources from heat db.
2) At that time second stack-delete command for the same stack
comes to heat engine. It steals the stack lock, waits 0.2 (hard-coded
constant!)
sec to allow previous stack-delete command finish the operations (of
course,
the first didn't manage to finish deleting on time). After that engine
service starts
the deleting again:
     - Request resources from heat DB (They exist!)
     - Send requests for delete to other clients (They do not exist because
of
        point 1).
Finally, we have stack in DELETE_FAILED state because the clients raise
exceptions during stack delete.
I have some proposals how to fix it:
p1) Make waiting time (0.2 sec) configurable. It allows to finish
stack-delete ops
before the second command starts deleting. From my point of view, it is
just
workaround because different stacks (and operations) took different time.
p2) Try to deny lock stealing if the current thread executes deleting. As
an option,
we can wait for the other thread if stack is deleting but it seems that it
is not possible
to analyze with the current solution.
p3) Just leave it as it is. IMO, the last solution.
Do you have any other proposals how to manage such kind of cases?
Perhaps there is exists more proper solution.

Thank You,
Kairat Kushaev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150210/772df18d/attachment.html>