On 2019/09/28 1:44, Ben Nemec wrote:
>
>
> On 9/23/19 11:42 PM, Rikimaru Honjo wrote:
>> Hi Eric,
>>
>> On 2019/09/20 23:10, Eric Harney wrote:
>>> On 9/20/19 1:52 AM, Rikimaru Honjo wrote:
>>>> Hi,
>>>>
>>>> I'm using Queens cinder with the following setting.
>>>>
>>>> ---------------------------------
>>>> [coordination]
>>>> backend_url = file://$state_path
>>>> ---------------------------------
>>>>
>>>> As a result, the files like the following were remained under the state path after some operations.[1]
>>>>
>>>> cinder-63dacb3d-bd4d-42bb-88fe-6e4180164765-delete_volume
>>>> cinder-32c426af-82b4-41de-b637-7d76fed69e83-delete_snapshot
>>>>
>>>> In my understanding, these are lock-files created for synchronization by tooz.
>>>> But, these lock-files were not deleted after finishing operations.
>>>> Is this behaviour correct?
>>>>
>>>> [1]
>>>> e.g. Delete volume, Delete snapshot
>>>
>>> This is a known bug that's described here:
>>>
>>> https://github.com/harlowja/fasteners/issues/26
>>>
>>> (The fasteners library is used by tooz, which is used by Cinder for managing these lock files.)
>>>
>>> There's an old Cinder bug for it here:
>>> https://bugs.launchpad.net/cinder/+bug/1432387
>>>
>>> but that's marked as "Won't Fix" because Cinder needs it to be fixed in the underlying libraries.
>> Thank you for your explanation.
>> I understood the state.
>>
>> But, I have one more question.
>> Can I think this bug doesn't affect synchronization?
>
> It does not. In fact, it's important to not remove lock files while a service is running or you can end up with synchronization issues.
>
> To clean up the leftover lock files, we generally recommend clearing the lock_path for each service on reboot before the services have started.
Thank you for your information.
I think that I understood this issue completely.
Best Regards,
>>
>> Best regards,
>>
>>> Thanks,
>>> Eric
>>>
>>
>
--
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Rikimaru Honjo
E-mail:honjo.rikimaru@ntt-tx.co.jp
I proposed some patches through heat templates and puppet-cinder to remove lock files older than 1 week and avoid file system growing.
This is a solution based on a cron job, to fix that on stable branches, in a second time I'll help the fasteners project to fix the root cause by reviewing and testing the proposed patch (lock based on file offset). In next versions I hope we will use a patched fasteners and so we could drop the cron based solution.
Please can you give /reviews/feedbacks:
- https://review.opendev.org/688413
- https://review.opendev.org/688414
- https://review.opendev.org/688415
Thanks