debugging iSCSI

Post summary:  Get wireshark.  Use it.  It might not solve your problem, but it will tell you who to blame.

I’ve been trying to mount an OpenSolaris ZFS on a diskless Ubuntu 10.04 box via iSCSI.  The goal is to share one large storage space among several hosts, with better performance than NFS.  Ubuntu recognizes the iSCSI target, logs in, and reports success.  But no device nodes appear, so I can’t newfs or mount the drive.  OpenSolaris reports no errors.  Ubuntu reports no errors.  But the device node isn’t there.

iSCSI isn’t encrypted.  This means that you can open up your packet sniffer and see the actual iSCSI discussions.  In my case, I saw the iSCSI login requests and acknowledgement of successful logins.  The initiator then asks for the status on a drive, and OpenSolaris responds with:

.111 0000 = SNS Error Type: Current Error (0x70)
Filemark: 0, EOM: 0, ILI: 0
…. 0100 = Sense Key: Hardware Error (0x04)

A hardware error.  From a 40GB virtual disk on a 1TB ZFS mirror.  One of the underlying disks is reporting intermittent errors, but the mirror as a whole is OK.  I asked the OpenSolaris folks about this, and got silence.  Either this problem is so common that nobody will even bother with a “check the archives,” or the OpenSolaris iSCSI community is dispirited by Oracle gutting them.

I’d call this another case of “fault-oblivious computing.”  One side or the other should report this error is a more visible way.

If only there was another open-source operating system with ZFS dedup support and iSCSI.  Hmmm…