Open Stack

On 10/15/19 7:48 AM, Herve Beraud wrote:
> I proposed some patches through heat templates and puppet-cinder to 
> remove lock files older than 1 week and avoid file system growing.
> 
> This is a solution based on a cron job, to fix that on stable branches, 
> in a second time I'll help the fasteners project to fix the root cause 
> by reviewing and testing the proposed patch (lock based on file offset). 
> In next versions I hope we will use a patched fasteners and so we could 
> drop the cron based solution.
> 
> Please can you give /reviews/feedbacks:
> - https://review.opendev.org/688413
> - https://review.opendev.org/688414
> - https://review.opendev.org/688415

I'm rather hesitant to recommend this. It looks like the change is only 
removing the -delete lock files, which are a fraction of the total lock 
files created by Cinder, and I don't particularly want to encourage 
people to start monkeying with the lock files while a service is 
running. Even with this limited set of deletions, I think we need a 
Cinder person to look and verify that we aren't making bad assumptions 
about how the locks are used.

In essence, I don't think this is going to meaningfully reduce the 
amount of leftover lock files and it sets a bad precedent for how to 
handle them.

Personally, I'd rather see a boot-time service added for each OpenStack 
service that goes out and wipes the lock file directory before starting 
the service.

On a more general note, I'm going to challenge the assertion that 
"Customer file system growing slowly and so customer risk to facing some
issues to file system usage after a long period." I have yet to hear an 
actual bug report from the leftover lock files. Every time this comes up 
it's because someone noticed a lot of lock files and thought we were 
leaking them. I've never heard anyone report an actual functional or 
performance problem as a result of the lock files. I don't think we 
should "fix" this until someone reports that it's actually broken. 
Especially because previous attempts have all resulted in very real bugs 
that did break people.

Maybe we should have oslo.concurrency drop a file named _README (or 
something else likely to sort first in the file listing) into the 
configured lock_path that explains why the files are there and the 
proper way to deal with them.

> 
> Thanks
> 
> 
> Le lun. 30 sept. 2019 à 03:35, Rikimaru Honjo 
> <honjo.rikimaru at ntt-tx.co.jp <mailto:honjo.rikimaru at ntt-tx.co.jp>> a écrit :
> 
>     On 2019/09/28 1:44, Ben Nemec wrote:
>      >
>      >
>      > On 9/23/19 11:42 PM, Rikimaru Honjo wrote:
>      >> Hi Eric,
>      >>
>      >> On 2019/09/20 23:10, Eric Harney wrote:
>      >>> On 9/20/19 1:52 AM, Rikimaru Honjo wrote:
>      >>>> Hi,
>      >>>>
>      >>>> I'm using Queens cinder with the following setting.
>      >>>>
>      >>>> ---------------------------------
>      >>>> [coordination]
>      >>>> backend_url = file://$state_path
>      >>>> ---------------------------------
>      >>>>
>      >>>> As a result, the files like the following were remained under
>     the state path after some operations.[1]
>      >>>>
>      >>>> cinder-63dacb3d-bd4d-42bb-88fe-6e4180164765-delete_volume
>      >>>> cinder-32c426af-82b4-41de-b637-7d76fed69e83-delete_snapshot
>      >>>>
>      >>>> In my understanding, these are lock-files created for
>     synchronization by tooz.
>      >>>> But, these lock-files were not deleted after finishing operations.
>      >>>> Is this behaviour correct?
>      >>>>
>      >>>> [1]
>      >>>> e.g. Delete volume, Delete snapshot
>      >>>
>      >>> This is a known bug that's described here:
>      >>>
>      >>> https://github.com/harlowja/fasteners/issues/26
>      >>>
>      >>> (The fasteners library is used by tooz, which is used by Cinder
>     for managing these lock files.)
>      >>>
>      >>> There's an old Cinder bug for it here:
>      >>> https://bugs.launchpad.net/cinder/+bug/1432387
>      >>>
>      >>> but that's marked as "Won't Fix" because Cinder needs it to be
>     fixed in the underlying libraries.
>      >> Thank you for your explanation.
>      >> I understood the state.
>      >>
>      >> But, I have one more question.
>      >> Can I think this bug doesn't affect synchronization?
>      >
>      > It does not. In fact, it's important to not remove lock files
>     while a service is running or you can end up with synchronization
>     issues.
>      >
>      > To clean up the leftover lock files, we generally recommend
>     clearing the lock_path for each service on reboot before the
>     services have started.
> 
>     Thank you for your information.
>     I think that I understood this issue completely.
> 
>     Best Regards,
> 
> 
>      >>
>      >> Best regards,
>      >>
>      >>> Thanks,
>      >>> Eric
>      >>>
>      >>
>      >
> 
>     -- 
>     _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
>     Rikimaru Honjo
>     E-mail:honjo.rikimaru at ntt-tx.co.jp
>     <mailto:E-mail%3Ahonjo.rikimaru at ntt-tx.co.jp>
> 
> 
> 
> 
> -- 
> Hervé Beraud
> Senior Software Engineer
> Red Hat - Openstack Oslo
> irc: hberaud
> -----BEGIN PGP SIGNATURE-----
> 
> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
> v6rDpkeNksZ9fFSyoY2o
> =ECSj
> -----END PGP SIGNATURE-----
>