Open Stack

Hi all,
 
we got a strange problem on our new mitaka installation. We have this messages
in the syslog on the block storage node:
 

Jul 25 09:10:33 block1 tgtd: device_mgmt(246) sz:69
params:path=/dev/cinder-volumes/volume-41d6c674-1d0d-471d-ad7d-07e9fab5c90d
Jul 25 09:10:33 block1 tgtd: bs_thread_open(412) 16
Jul 25 09:10:55 block1 kernel: [1471887.006569] sd 5:0:0:0: [sdc] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 25 09:10:55 block1 kernel: [1471887.006585] sd 5:0:0:0: [sdc] Sense Key :
Illegal Request [current]
Jul 25 09:10:55 block1 kernel: [1471887.006589] sd 5:0:0:0: [sdc] Add. Sense:
Invalid field in cdb
Jul 25 09:10:55 block1 kernel: [1471887.006590] sd 5:0:0:0: [sdc] CDB:
Jul 25 09:10:55 block1 kernel: [1471887.006593] Write(16): 8a 00 00 00 00 00 00
1c d0 00 00 00 40 00 00 00
Jul 25 09:10:55 block1 kernel: [1471887.006603] blk_update_request: critical
target error, dev sdc, sector 1888256
Jul 25 09:10:55 block1 kernel: [1471887.025141] blk_update_request: critical
target error, dev dm-0, sector 1888256
Jul 25 09:10:55 block1 kernel: [1471887.043979] buffer_io_error: 6695 callbacks
suppressed
Jul 25 09:10:55 block1 kernel: [1471887.043981] Buffer I/O error on dev dm-1,
logical block 235776, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.063894] Buffer I/O error on dev dm-1,
logical block 235777, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.082592] Buffer I/O error on dev dm-1,
logical block 235778, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.100903] Buffer I/O error on dev dm-1,
logical block 235779, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.119625] Buffer I/O error on dev dm-1,
logical block 235780, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.138360] Buffer I/O error on dev dm-1,
logical block 235781, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.157247] Buffer I/O error on dev dm-1,
logical block 235782, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.175086] Buffer I/O error on dev dm-1,
logical block 235783, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.193637] Buffer I/O error on dev dm-1,
logical block 235784, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.212358] Buffer I/O error on dev dm-1,
logical block 235785, lost async page write
Jul 25 09:10:55 block1 kernel: [1471887.232830] sd 5:0:0:0: [sdc] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 25 09:10:55 block1 kernel: [1471887.232833] sd 5:0:0:0: [sdc] Sense Key :
Illegal Request [current]
Jul 25 09:10:55 block1 kernel: [1471887.232836] sd 5:0:0:0: [sdc] Add. Sense:
Invalid field in cdb
Jul 25 09:10:55 block1 kernel: [1471887.232837] sd 5:0:0:0: [sdc] CDB:
Jul 25 09:10:55 block1 kernel: [1471887.232839] Write(16): 8a 00 00 00 00 00 00
1d 10 00 00 00 40 00 00 00
Jul 25 09:10:55 block1 kernel: [1471887.232847] blk_update_request: critical
target error, dev sdc, sector 1904640
Jul 25 09:10:55 block1 kernel: [1471887.251046] sd 5:0:0:0: [sdc] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 25 09:10:55 block1 kernel: [1471887.251049] sd 5:0:0:0: [sdc] Sense Key :
Illegal Request [current]
Jul 25 09:10:55 block1 kernel: [1471887.251052] sd 5:0:0:0: [sdc] Add. Sense:
Invalid field in cdb
Jul 25 09:10:55 block1 kernel: [1471887.251053] sd 5:0:0:0: [sdc] CDB:
Jul 25 09:10:55 block1 kernel: [1471887.251054] Write(16): 8a 00 00 00 00 00 00
1d 50 00 00 00 40 00 00 00
Jul 25 09:10:55 block1 kernel: [1471887.251062] blk_update_request: critical
target error, dev sdc, sector 1921024
Jul 25 09:10:55 block1 kernel: [1471887.269726] sd 5:0:0:0: [sdc] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 25 09:10:55 block1 kernel: [1471887.269729] sd 5:0:0:0: [sdc] Sense Key :
Illegal Request [current]
Jul 25 09:10:55 block1 kernel: [1471887.269732] sd 5:0:0:0: [sdc] Add. Sense:
Invalid field in cdb
Jul 25 09:10:55 block1 kernel: [1471887.269733] sd 5:0:0:0: [sdc] CDB:
Jul 25 09:10:55 block1 kernel: [1471887.269735] Write(16): 8a 00 00 00 00 00 00
1d 90 00 00 00 11 88 00 00
Jul 25 09:10:55 block1 kernel: [1471887.269744] blk_update_request: critical
target error, dev sdc, sector 1937408
Jul 25 09:10:55 block1 kernel: [1471887.287739] blk_update_request: critical
target error, dev dm-0, sector 1904640
Jul 25 09:10:55 block1 kernel: [1471887.309002] blk_update_request: critical
target error, dev dm-0, sector 1921024
Jul 25 09:10:55 block1 kernel: [1471887.330162] blk_update_request: critical
target error, dev dm-0, sector 1937408
Jul 25 09:10:55 block1 tgtd: bs_rdwr_request(370) io error 0x9a87c0 35 0 0 0,
Input/output error
Jul 25 09:11:50 block1 tgtd: conn_close(103) connection closed, 0x9a7dc0 1
Jul 25 09:11:50 block1 tgtd: conn_close(109) session 0x9a72c0 1

 

bit before we had this messages also on teh compute and the object nodes.

We use iscsi storage (HP MSA 2040) over multipathd (4 paths, sdb,sdc,sdd, sde)
on alle nodes on the compute nodes we have ocfs2 for the instance store, to
solve this problem we changed the blocksize of the ocfs2 from 4k to 1k then the
error disappeared.

On the objectstore nodes we had xfs but changign the blocksize did not help so
we used ther also ocfs2 with 1k blocksize so the error was going also away.

Now we have the same problem on the blockstorage node with lvm. When we create a
new volume from an image ther is no error, even the new instance is booting fine
from the volume. But when we log into the instance than this errors pop up in
the syslog of the blockstorege node and in the instance we get errors that the
filesystem is damaged an mounted read only.

I tried different multipath.conf configurations also different images, and
formats (raw/qcow2) but the error stays the same.

Did anyone encounter this error so far or maybe know whats might be the problem?


Any idea is highly appreciated :-)

Thanks and kind regards,

Michael

 

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160726/3ba4d420/attachment.html>