<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>That has done the trick! Thank you so much for your help.</p>
<p>Regards,</p>
<p>Grant<br>
</p>
<div class="moz-cite-prefix">On 21/07/2020 14:45, CHANU ROMAIN
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:02f5e9fb1d6df3179e7ae856df7326c8a8499cb3.camel@univ-lyon1.fr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div>Hello,</div>
<div><br>
</div>
<div>I do not use CentOS and XFS but I had a simillar issue after
an outrage. Ceph didnt release the lock on rados block device.
You can check if you are facing the same issue than I did. You
have to shutdown your instance then type this command:</div>
<div><br>
</div>
<div>rbd -p your-pool-name lock list instance-volume-id</div>
<div><br>
</div>
<div>The command should not return any output if your instance is
shut. If you got an output about 1 exclusive lock just remove
it:</div>
<div><br>
</div>
<div>rbd -p your-pool-name lock remove instance-volume-id</div>
<div><br>
</div>
<div>Best Regards,</div>
<div>Romain</div>
<div><br>
</div>
<div>On Tue, 2020-07-21 at 14:04 +0100, Grant Morley wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px
#729fcf solid;padding-left:1ex">
<p>Hi all,</p>
<p>We recently had an issue with our ceph cluster which ended up
going into "Error" status after some drive failures. The
system stopped allowing writes for a while whilst it
recovered. The ceph cluster is healthy again but we seem to
have a few instances that have corrupt filesystems on them.
They are all CentOS 7 instances. We have got them into rescue
mode to try and repair the FS with "xfs_repair -L" However
when we do that we get this:</p>
<p><span data-intercom-target="last-user-comment" id="ember1512"
class="m__conversation-stream__compact
m__conversation-stream__compact-last ember-view">
973.026283] XFS (vdb1): Mounting V5 Filesystem<br>
[ 973.203261] blk_update_request: I/O error, dev vdb, sector
8389693<br>
[ 973.204746] blk_update_request: I/O error, dev vdb, sector
8390717<br>
[ 973.206136] blk_update_request: I/O error, dev vdb, sector
8391741<br>
[ 973.207608] blk_update_request: I/O error, dev vdb, sector
8392765<br>
[ 973.209544] XFS (vdb1): xfs_do_force_shutdown(0x1) called
from line 1236 of file fs/xfs/xfs_buf.c. Return address =
0xffffffffc017a50c<br>
[ 973.212137] XFS (vdb1): I/O Error Detected. Shutting down
filesystem<br>
[ 973.213429] XFS (vdb1): Please umount the filesystem and
rectify the problem(s)<br>
[ 973.215036] XFS (vdb1): metadata I/O error: block 0x7ffc3d
("xlog_bwrite") error 5 numblks 8192<br>
[ 973.217201] XFS (vdb1): failed to locate log tail<br>
[ 973.218239] XFS (vdb1): log mount/recovery failed: error
-5<br>
[ 973.219865] XFS (vdb1): log mount failed<br>
[ 973.233792] blk_update_request: I/O error, dev vdb, sector
0</span></p>
<p><span data-intercom-target="last-user-comment" id="ember1512"
class="m__conversation-stream__compact
m__conversation-stream__compact-last ember-view">Interestingly
any debian based instances we could recover. It just seems
to be CentOS and having XFS on CentOS and ceph the instances
don't seem happy. This seems more low level to me in ceph
rather than a corrupt FS on a guest.</span></p>
<p><span data-intercom-target="last-user-comment" id="ember1512"
class="m__conversation-stream__compact
m__conversation-stream__compact-last ember-view">Does anyone
know of any "ceph tricks" that we can use to try and at
least get an "xfs_repair" running?</span></p>
<p><span data-intercom-target="last-user-comment" id="ember1512"
class="m__conversation-stream__compact
m__conversation-stream__compact-last ember-view">Many
thanks,</span></p>
<p><span data-intercom-target="last-user-comment" id="ember1512"
class="m__conversation-stream__compact
m__conversation-stream__compact-last ember-view"><br>
</span></p>
<div class="moz-signature">-- <br>
<div style="font-family: helvetica, Arial; font-size: 13px;"><font
face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"><span style="font-size: 11px; line-height:
15px; float: left; margin-right: 10px;"><img
src="https://www.civo.com/images/email-logo.jpg"
style="width: 70px; color: #a8d8ff; font-family:
'Lucida Grande', Verdana, Arial, sans-serif;
font-size: 11px; line-height: 15px; float: left;
margin-top: 0px; margin-right: 10px"
moz-do-not-send="true"></span></font>
<div style="margin-left: 90px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"><span
style="font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; font-size: 18px; color:
rgb(35, 157, 255);">Grant Morley</span> <font
face="Hind, Lucida Grande, Verdana, Arial, sans-serif"><span
style="font-size: 15px;"></span></font></div>
<div style="margin-left: 90px; line-height: 15px; color:
#888888; margin-bottom: 10px;"><font face="Hind, Lucida
Grande, Verdana, Arial, sans-serif"><span
style="font-size: 15px;">Cloud Lead, Civo Ltd</span></font></div>
<div style="font-family: Hind, helvetica, arial;
margin-left: 90px;"><a href="https://www.civo.com/"
style="color: rgb(35, 157, 255); font-family: Hind,
"Lucida Grande", Verdana, Arial, sans-serif;
line-height: 15px; text-decoration: none;"
moz-do-not-send="true">www.civo.com</a><font
color="#888888"> </font><a <="" div=""
moz-do-not-send="true"> </a></div>
<a <="" div="" moz-do-not-send="true"> </a></div>
<a <="" div="" moz-do-not-send="true"> </a></div>
</blockquote>
</blockquote>
<div class="moz-signature">-- <br>
<div style="font-family: helvetica, Arial; font-size: 13px;"><font
face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"><span style="font-size: 11px; line-height:
15px; float: left; margin-right: 10px;"><img
src="https://www.civo.com/images/email-logo.jpg"
style="width: 70px; color: #a8d8ff; font-family: 'Lucida
Grande', Verdana, Arial, sans-serif; font-size: 11px;
line-height: 15px; float: left; margin-top: 0px;
margin-right: 10px"></span></font>
<div style="margin-left: 90px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"><span
style="font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; font-size: 18px; color: rgb(35,
157, 255);">Grant Morley</span> <font face="Hind, Lucida
Grande, Verdana, Arial, sans-serif"><span style="font-size:
15px;"></span></font></div>
<div style="margin-left: 90px; line-height: 15px; color:
#888888; margin-bottom: 10px;"><font face="Hind, Lucida
Grande, Verdana, Arial, sans-serif"><span style="font-size:
15px;">Cloud Lead, Civo Ltd</span></font></div>
<div style="font-family: Hind, helvetica, arial; margin-left:
90px;"><a href="https://www.civo.com/" style="color: rgb(35,
157, 255); font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; line-height: 15px;
text-decoration: none;">www.civo.com</a><font
color="#888888"> </font><a <="" div=""> </a></div>
<a <="" div=""> </a></div>
<a <="" div=""> </a></div>
</body>
</html>