<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>Thanks for the email. We have managed to fix this by upgrading to
the latest minor version patch of Ceph Jewel and restarting the
OSDs. Seems like there might have been some write lock issues that
were not being reported by ceph.</p>
<p>Many Thanks,<br>
</p>
<div class="moz-cite-prefix">On 21/01/2019 20:49, melanie witt
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:367a4705-468c-f73a-ba4e-450e36af7609@gmail.com">On Mon,
21 Jan 2019 17:40:52 +0000, Grant Morley
<a class="moz-txt-link-rfc2396E" href="mailto:grant@absolutedevops.io"><grant@absolutedevops.io></a> wrote:
<br>
<blockquote type="cite">Hi all,
<br>
<br>
We are in the process of retiring one of our old platforms and
last night our ceph cluster went into an "Error" state briefly
because 1 of the OSDs went close to full. The data got
re-balanced fine and the health of ceph is now "OK" - however we
have about 40% of our instances that now have corrupt disks
which is a bit odd.
<br>
<br>
Even more strange is that we cannot get them into rescue mode.
As soon as we try we the instances seem to hang during the
bootup process when they are trying to mount "/dev/vdb1" and we
eventually get a kernel timeout error as below:
<br>
<br>
Warning: fsck not present, so skipping root file system
<br>
[ 5.644526] EXT4-fs (vdb1): INFO: recovery required on
readonly filesystem
<br>
[ 5.645583] EXT4-fs (vdb1): write access will be enabled
during recovery
<br>
[ 240.504873] INFO: task exe:332 blocked for more than 120
seconds.
<br>
[ 240.506986] Not tainted 4.4.0-66-generic #87-Ubuntu
<br>
[ 240.508782] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 240.511438] exe D ffff88003714b878 0
332 1 0x00000000
<br>
[ 240.513809] ffff88003714b878 ffff88007c18e358
ffffffff81e11500 ffff88007be81c00
<br>
[ 240.516665] ffff88003714c000 ffff88007fc16dc0
7fffffffffffffff ffffffff81838cd0
<br>
[ 240.519546] ffff88003714b9d0 ffff88003714b890
ffffffff818384d5 0000000000000000
<br>
[ 240.522399] Call Trace:
<br>
<br>
I have even tried using a different image for nova rescue and we
are getting the same results. Has anyone come across this
before?
<br>
<br>
This system is running OpenStack Mitaka with Ceph Jewel.
<br>
<br>
Any help or suggestions will be much appreciated.
<br>
</blockquote>
<br>
I don't know whether this is related, but what you describe
reminded me of issues I have seen before in the past:
<br>
<br>
<a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/nova/+bug/1781878">https://bugs.launchpad.net/nova/+bug/1781878</a>
<br>
<br>
See my comment #1 on the bug ^ for links to additional information
on the same root issue.
<br>
<br>
Hope this helps in some way,
<br>
-melanie
<br>
</blockquote>
<div class="moz-signature">-- <br>
<!--?xml version="1.0" encoding="UTF-8"?-->
<title></title>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"> <span style="font-size: 11px; line-height:
15px; float: left; margin-right: 10px;"> <img
src="https://www.absolutedevops.io/ado-email.png" s=""
tyle="color: #a8d8ff; font-family: 'Lucida Grande',
Verdana, Arial, sans-serif; font-size: 11px; line-height:
15px; float: left; margin-top: 0px; margin-right: 10px"> </span>
</font>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"> <span
style="font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif; font-size: 18px; font-weight: 200; color:
rgb(35, 157, 255);">Grant Morley</span> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span
style="font-size: 15px;"> </span> </font> </div>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 10px;"> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span
style="font-size: 15px;">Cloud Lead</span> </font> </div>
<div style="font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif; font-size: 13px; margin-left: 80px; line-height:
15px; color: rgb(153, 153, 153);"> <span style="color:
#616161">Absolute</span> <span style="color: #98a5b0">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP
</div>
<div style="font-family: Hind, helvetica, arial; font-size:
13px; margin-left: 80px;"> <a
href="http://www.absolutedevops.io/" style="color: #239dff;
font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;">www.absolutedevops.io</a> <a
href="mailto:grant@absolutedevops.io" style="color: #239dff;
font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;">grant@absolutedevops.io</a> <a
style="color: #239dff; font-family: Hind, 'Lucida Grande',
Verdana, Arial, sans-serif;line-height: 15px; margin-right:
6px;">0845 874 0580</a> </div>
</div>
</div>
</body>
</html>