Arne, I saw the messages like ''moving volume to trash" in the cinder-volume logs and the peridic task also reports like "Deleted <vol-uuid> from trash for backend '<backends-name>'" The patch worked well when clearing a small number of volumes. This happens only when I am deleting a large number of volumes. I will try to adjust the number of thread pools by adjusting the environment variables with your advices Do you know why the cinder-volume hang does not occur when create a volume, but only when delete a volume? Thanks. 2019년 2월 11일 (월) 오후 6:14, Arne Wiebalck <Arne.Wiebalck@cern.ch>님이 작성:
Jae,
To make sure deferred deletion is properly working: when you delete individual large volumes with data in them, do you see that - the volume is fully “deleted" within a few seconds, ie. not staying in ‘deleting’ for a long time? - that the volume shows up in trash (with “rbd trash ls”)? - the periodic task reports it is deleting volumes from the trash?
Another option to look at is “backend_native_threads_pool_size": this will increase the number of threads to work on deleting volumes. It is independent from deferred deletion, but can also help with situations where Cinder has more work to do than it can cope with at the moment.
Cheers, Arne
On 11 Feb 2019, at 09:47, Jae Sang Lee <hyangii@gmail.com> wrote:
Yes, I added your code to pike release manually.
2019년 2월 11일 (월) 오후 4:39에 Arne Wiebalck <Arne.Wiebalck@cern.ch>님이 작성:
Hi Jae,
You back ported the deferred deletion patch to Pike?
Cheers, Arne
On 11 Feb 2019, at 07:54, Jae Sang Lee <hyangii@gmail.com> wrote:
Hello,
I recently ran a volume deletion test with deferred deletion enabled on the pike release.
We experienced a cinder-volume hung when we were deleting a large amount of the volume in which the data was actually written(I make 15GB file in every volumes), and we thought deferred deletion would solve it.
However, while deleting 200 volumes, after 50 volumes, the cinder-volume downed as before. In my opinion, the trash_move api does not seem to work properly when removing multiple volumes, just like remove api.
If these test results are my fault, please let me know the correct test method.
-- Arne Wiebalck CERN IT
-- Arne Wiebalck CERN IT