<!DOCTYPE html>
<html><head>
<meta charset="UTF-8">
<style type="text/css">.mceResizeHandle {position: absolute;border: 1px solid black;background: #FFF;width: 5px;height: 5px;z-index: 10000}.mceResizeHandle:hover {background: #000}img[data-mce-selected] {outline: 1px solid black}img.mceClonedResizable, table.mceClonedResizable {position: absolute;outline: 1px dashed black;opacity: .5;z-index: 10000}
</style></head><body style=""><div>Hi all,</div>
<div> </div>
<div>we got a strange problem on our new mitaka installation. We have this messages in the syslog on the block storage node:</div>
<div> </div>
<div>
<p>Jul 25 09:10:33 block1 tgtd: device_mgmt(246) sz:69 params:path=/dev/cinder-volumes/volume-41d6c674-1d0d-471d-ad7d-07e9fab5c90d<br>Jul 25 09:10:33 block1 tgtd: bs_thread_open(412) 16<br>Jul 25 09:10:55 block1 kernel: [1471887.006569] sd 5:0:0:0: [sdc] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE<br>Jul 25 09:10:55 block1 kernel: [1471887.006585] sd 5:0:0:0: [sdc] Sense Key : Illegal Request [current]<br>Jul 25 09:10:55 block1 kernel: [1471887.006589] sd 5:0:0:0: [sdc] Add. Sense: Invalid field in cdb<br>Jul 25 09:10:55 block1 kernel: [1471887.006590] sd 5:0:0:0: [sdc] CDB:<br>Jul 25 09:10:55 block1 kernel: [1471887.006593] Write(16): 8a 00 00 00 00 00 00 1c d0 00 00 00 40 00 00 00<br>Jul 25 09:10:55 block1 kernel: [1471887.006603] blk_update_request: critical target error, dev sdc, sector 1888256<br>Jul 25 09:10:55 block1 kernel: [1471887.025141] blk_update_request: critical target error, dev dm-0, sector 1888256<br>Jul 25 09:10:55 block1 kernel: [1471887.043979] buffer_io_error: 6695 callbacks suppressed<br>Jul 25 09:10:55 block1 kernel: [1471887.043981] Buffer I/O error on dev dm-1, logical block 235776, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.063894] Buffer I/O error on dev dm-1, logical block 235777, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.082592] Buffer I/O error on dev dm-1, logical block 235778, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.100903] Buffer I/O error on dev dm-1, logical block 235779, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.119625] Buffer I/O error on dev dm-1, logical block 235780, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.138360] Buffer I/O error on dev dm-1, logical block 235781, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.157247] Buffer I/O error on dev dm-1, logical block 235782, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.175086] Buffer I/O error on dev dm-1, logical block 235783, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.193637] Buffer I/O error on dev dm-1, logical block 235784, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.212358] Buffer I/O error on dev dm-1, logical block 235785, lost async page write<br>Jul 25 09:10:55 block1 kernel: [1471887.232830] sd 5:0:0:0: [sdc] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE<br>Jul 25 09:10:55 block1 kernel: [1471887.232833] sd 5:0:0:0: [sdc] Sense Key : Illegal Request [current]<br>Jul 25 09:10:55 block1 kernel: [1471887.232836] sd 5:0:0:0: [sdc] Add. Sense: Invalid field in cdb<br>Jul 25 09:10:55 block1 kernel: [1471887.232837] sd 5:0:0:0: [sdc] CDB:<br>Jul 25 09:10:55 block1 kernel: [1471887.232839] Write(16): 8a 00 00 00 00 00 00 1d 10 00 00 00 40 00 00 00<br>Jul 25 09:10:55 block1 kernel: [1471887.232847] blk_update_request: critical target error, dev sdc, sector 1904640<br>Jul 25 09:10:55 block1 kernel: [1471887.251046] sd 5:0:0:0: [sdc] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE<br>Jul 25 09:10:55 block1 kernel: [1471887.251049] sd 5:0:0:0: [sdc] Sense Key : Illegal Request [current]<br>Jul 25 09:10:55 block1 kernel: [1471887.251052] sd 5:0:0:0: [sdc] Add. Sense: Invalid field in cdb<br>Jul 25 09:10:55 block1 kernel: [1471887.251053] sd 5:0:0:0: [sdc] CDB:<br>Jul 25 09:10:55 block1 kernel: [1471887.251054] Write(16): 8a 00 00 00 00 00 00 1d 50 00 00 00 40 00 00 00<br>Jul 25 09:10:55 block1 kernel: [1471887.251062] blk_update_request: critical target error, dev sdc, sector 1921024<br>Jul 25 09:10:55 block1 kernel: [1471887.269726] sd 5:0:0:0: [sdc] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE<br>Jul 25 09:10:55 block1 kernel: [1471887.269729] sd 5:0:0:0: [sdc] Sense Key : Illegal Request [current]<br>Jul 25 09:10:55 block1 kernel: [1471887.269732] sd 5:0:0:0: [sdc] Add. Sense: Invalid field in cdb<br>Jul 25 09:10:55 block1 kernel: [1471887.269733] sd 5:0:0:0: [sdc] CDB:<br>Jul 25 09:10:55 block1 kernel: [1471887.269735] Write(16): 8a 00 00 00 00 00 00 1d 90 00 00 00 11 88 00 00<br>Jul 25 09:10:55 block1 kernel: [1471887.269744] blk_update_request: critical target error, dev sdc, sector 1937408<br>Jul 25 09:10:55 block1 kernel: [1471887.287739] blk_update_request: critical target error, dev dm-0, sector 1904640<br>Jul 25 09:10:55 block1 kernel: [1471887.309002] blk_update_request: critical target error, dev dm-0, sector 1921024<br>Jul 25 09:10:55 block1 kernel: [1471887.330162] blk_update_request: critical target error, dev dm-0, sector 1937408<br>Jul 25 09:10:55 block1 tgtd: bs_rdwr_request(370) io error 0x9a87c0 35 0 0 0, Input/output error<br>Jul 25 09:11:50 block1 tgtd: conn_close(103) connection closed, 0x9a7dc0 1<br>Jul 25 09:11:50 block1 tgtd: conn_close(109) session 0x9a72c0 1</p>
<p> </p>
<p>bit before we had this messages also on teh compute and the object nodes.</p>
<p>We use iscsi storage (HP MSA 2040) over multipathd (4 paths, sdb,sdc,sdd, sde) on alle nodes on the compute nodes we have ocfs2 for the instance store, to solve this problem we changed the blocksize of the ocfs2 from 4k to 1k then the error disappeared.</p>
<p>On the objectstore nodes we had xfs but changign the blocksize did not help so we used ther also ocfs2 with 1k blocksize so the error was going also away.</p>
<p>Now we have the same problem on the blockstorage node with lvm. When we create a new volume from an image ther is no error, even the new instance is booting fine from the volume. But when we log into the instance than this errors pop up in the syslog of the blockstorege node and in the instance we get errors that the filesystem is damaged an mounted read only.</p>
<p><span style="font-size: 9pt;">I tried different multipath.conf configurations also different images, and formats (raw/qcow2) but the error stays the same.</span></p>
<p>Did anyone encounter this error so far or maybe know whats might be the problem? </p>
<p>Any idea is highly appreciated :-)</p>
<p><span style="font-size: 9pt;">Thanks and kind regards,</span></p>
<p>Michael</p>
<p> </p>
<p> </p>
</div></body></html>