<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 15 oct. 2019 à 16:48, Ben Nemec <<a href="mailto:openstack@nemebean.com">openstack@nemebean.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
On 10/15/19 7:48 AM, Herve Beraud wrote:<br>
> I proposed some patches through heat templates and puppet-cinder to <br>
> remove lock files older than 1 week and avoid file system growing.<br>
> <br>
> This is a solution based on a cron job, to fix that on stable branches, <br>
> in a second time I'll help the fasteners project to fix the root cause <br>
> by reviewing and testing the proposed patch (lock based on file offset). <br>
> In next versions I hope we will use a patched fasteners and so we could <br>
> drop the cron based solution.<br>
> <br>
> Please can you give /reviews/feedbacks:<br>
> - <a href="https://review.opendev.org/688413" rel="noreferrer" target="_blank">https://review.opendev.org/688413</a><br>
> - <a href="https://review.opendev.org/688414" rel="noreferrer" target="_blank">https://review.opendev.org/688414</a><br>
> - <a href="https://review.opendev.org/688415" rel="noreferrer" target="_blank">https://review.opendev.org/688415</a><br>
<br>
I'm rather hesitant to recommend this. It looks like the change is only <br>
removing the -delete lock files, which are a fraction of the total lock <br>
files created by Cinder, and I don't particularly want to encourage <br>
people to start monkeying with the lock files while a service is <br>
running. Even with this limited set of deletions, I think we need a <br>
Cinder person to look and verify that we aren't making bad assumptions <br>
about how the locks are used.<br></blockquote><div><br></div><div>Yes these changes should be validated by the cinder team.</div><div>I chosen this approach to allow use to fix that on stable branches too, and to avoid to introduce a non backportable new feature.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
In essence, I don't think this is going to meaningfully reduce the <br>
amount of leftover lock files and it sets a bad precedent for how to <br>
handle them.<br>
<br>
Personally, I'd rather see a boot-time service added for each OpenStack <br>
service that goes out and wipes the lock file directory before starting <br>
the service.<br></blockquote><div><br></div><div>I agree it can be an alternative to the proposed changes. <br></div><div>I guess it's related to some sort of puppet code too, I'm right? (the boot-time service)<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
On a more general note, I'm going to challenge the assertion that <br>
"Customer file system growing slowly and so customer risk to facing some<br>
issues to file system usage after a long period." I have yet to hear an <br>
actual bug report from the leftover lock files. Every time this comes up <br>
it's because someone noticed a lot of lock files and thought we were <br>
leaking them. I've never heard anyone report an actual functional or <br>
performance problem as a result of the lock files. I don't think we <br>
should "fix" this until someone reports that it's actually broken. <br>
Especially because previous attempts have all resulted in very real bugs <br>
that did break people.<br></blockquote><div><br></div><div>Yes I agreee it's more an assumption than a reality, I never seen anybody report a disk usage issue or things like this due to leftover lock files. <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Maybe we should have oslo.concurrency drop a file named _README (or <br>
something else likely to sort first in the file listing) into the <br>
configured lock_path that explains why the files are there and the <br>
proper way to deal with them.<br></blockquote><div><br></div><div>Good idea. <br></div><div><br></div><div><div>Anyway, even if nobody facing a file system issue related to files leftover, I think it's not a good thing to lets grow a FS, and we need to try to address it to prevent potential file system issues related to disk usage and lock files, but in a secure way to avoid to introduce race conditions with cinder. <br></div><div><br></div><div>Cinder people need to confirm that my proposed changes can fit well with cinder's mechanismes or choose a better approach.</div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> <br>
> Thanks<br>
> <br>
> <br>
> Le lun. 30 sept. 2019 à 03:35, Rikimaru Honjo <br>
> <<a href="mailto:honjo.rikimaru@ntt-tx.co.jp" target="_blank">honjo.rikimaru@ntt-tx.co.jp</a> <mailto:<a href="mailto:honjo.rikimaru@ntt-tx.co.jp" target="_blank">honjo.rikimaru@ntt-tx.co.jp</a>>> a écrit :<br>
> <br>
> On 2019/09/28 1:44, Ben Nemec wrote:<br>
> ><br>
> ><br>
> > On 9/23/19 11:42 PM, Rikimaru Honjo wrote:<br>
> >> Hi Eric,<br>
> >><br>
> >> On 2019/09/20 23:10, Eric Harney wrote:<br>
> >>> On 9/20/19 1:52 AM, Rikimaru Honjo wrote:<br>
> >>>> Hi,<br>
> >>>><br>
> >>>> I'm using Queens cinder with the following setting.<br>
> >>>><br>
> >>>> ---------------------------------<br>
> >>>> [coordination]<br>
> >>>> backend_url = file://$state_path<br>
> >>>> ---------------------------------<br>
> >>>><br>
> >>>> As a result, the files like the following were remained under<br>
> the state path after some operations.[1]<br>
> >>>><br>
> >>>> cinder-63dacb3d-bd4d-42bb-88fe-6e4180164765-delete_volume<br>
> >>>> cinder-32c426af-82b4-41de-b637-7d76fed69e83-delete_snapshot<br>
> >>>><br>
> >>>> In my understanding, these are lock-files created for<br>
> synchronization by tooz.<br>
> >>>> But, these lock-files were not deleted after finishing operations.<br>
> >>>> Is this behaviour correct?<br>
> >>>><br>
> >>>> [1]<br>
> >>>> e.g. Delete volume, Delete snapshot<br>
> >>><br>
> >>> This is a known bug that's described here:<br>
> >>><br>
> >>> <a href="https://github.com/harlowja/fasteners/issues/26" rel="noreferrer" target="_blank">https://github.com/harlowja/fasteners/issues/26</a><br>
> >>><br>
> >>> (The fasteners library is used by tooz, which is used by Cinder<br>
> for managing these lock files.)<br>
> >>><br>
> >>> There's an old Cinder bug for it here:<br>
> >>> <a href="https://bugs.launchpad.net/cinder/+bug/1432387" rel="noreferrer" target="_blank">https://bugs.launchpad.net/cinder/+bug/1432387</a><br>
> >>><br>
> >>> but that's marked as "Won't Fix" because Cinder needs it to be<br>
> fixed in the underlying libraries.<br>
> >> Thank you for your explanation.<br>
> >> I understood the state.<br>
> >><br>
> >> But, I have one more question.<br>
> >> Can I think this bug doesn't affect synchronization?<br>
> ><br>
> > It does not. In fact, it's important to not remove lock files<br>
> while a service is running or you can end up with synchronization<br>
> issues.<br>
> ><br>
> > To clean up the leftover lock files, we generally recommend<br>
> clearing the lock_path for each service on reboot before the<br>
> services have started.<br>
> <br>
> Thank you for your information.<br>
> I think that I understood this issue completely.<br>
> <br>
> Best Regards,<br>
> <br>
> <br>
> >><br>
> >> Best regards,<br>
> >><br>
> >>> Thanks,<br>
> >>> Eric<br>
> >>><br>
> >><br>
> ><br>
> <br>
> -- <br>
> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/<br>
> Rikimaru Honjo<br>
> <a href="mailto:E-mail%3Ahonjo.rikimaru@ntt-tx.co.jp" target="_blank">E-mail:honjo.rikimaru@ntt-tx.co.jp</a><br>
> <mailto:<a href="mailto:E-mail%253Ahonjo.rikimaru@ntt-tx.co.jp" target="_blank">E-mail%3Ahonjo.rikimaru@ntt-tx.co.jp</a>><br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> Hervé Beraud<br>
> Senior Software Engineer<br>
> Red Hat - Openstack Oslo<br>
> irc: hberaud<br>
> -----BEGIN PGP SIGNATURE-----<br>
> <br>
> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>
> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>
> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>
> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>
> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>
> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>
> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>
> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>
> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>
> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>
> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>
> v6rDpkeNksZ9fFSyoY2o<br>
> =ECSj<br>
> -----END PGP SIGNATURE-----<br>
> <br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer<br></div><div>Red Hat - Openstack Oslo</div><div>irc: hberaud</div><div>-----BEGIN PGP SIGNATURE-----<br><br>wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>v6rDpkeNksZ9fFSyoY2o<br>=ECSj<br>-----END PGP SIGNATURE-----<br><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>