Blog

I’m working on OpenZFS Mastery’s discussion of self-healing with a completely plugged ear, so I have no idea how this sounds.

OpenZFS uses hashes almost everywhere. A hash is an algorithm that takes a chunk of data and computes a fixed-length string from it. The interesting thing about a hash is that minor changes in the original data dramatically change the data’s hash. Each block of storage includes the hash of its parent block, while each parent block includes the hash of all its immediate children. Whenever the system accesses data, it verifies the data’s correctness using these hashes. Upon discovering an error, OpenZFS corrects or self-heals them. If the underlying VDEVs have redundancy, ZFS either reconstructs the damaged block from RAID-Z or grabs the intact copy from the mirror. If both sides of a mirror have errors, ZFS can recover the files so long as the same blocks are not bad on both disks. If the VDEV has no redundancy, but a dataset has extra copies of the data (see Chapter 4), ZFS uses those extra copies instead.

OpenZFS Mastery is still open for sponsoring.

My new FreeBSD Journal column just went to the editor.

Oh AWOL, my succulent summer child,

You’re the latest model of Human(tm), a digital native, theoretically enlightened by access to the collected wisdom of our species. Are your sources of wholesome, cogent, and actionable advice so limited that reaching out to a grizzled sysadmin seemed wise? True, this is an advice column. I offer advice. I also offer trepanning and experimental epidemiology but you don’t see people queuing for either of those do you? The Internet has Captain Awkward and Doctor Nerdlove and Dear Prudence and Ask A Manager and Dear Abby and you decided that no, you want to live abhorrently so you’re beseeching a grant of “wisdom” from a system administrator known for enhancing rat operated vehicles with nitrous oxide boosters. If you’re from a civilized country you even have health care so you could access counseling, or perhaps I should say intensive counseling because a few minutes into your first session the therapist would hit the big red button and you’d have a comfortable new home with three hot meals a day, your own bed guaranteed 98% free from arthropod infestation, and a meticulously personalized medication regimen because, after all, you think that sysadmins dispense sensible advice for a better life.

For more of my ill-advised advice, grab a copy of Dear Abyss.

Talking pools in OpenZFS Mastery, which means yet again talking about blocks and sectors and alignment.

GPT partitions fill a number of sectors. If you partition a drive assuming 512-byte sectors, you can easily create a partition that would not cleanly start and end on 4K sector boundaries. Take a drive that has 4K sectors, but claims to have 512-byte sectors. Create a partition that fills, say, seven 512-byte sectors, then add another partition that fills the rest of the drive. Make a ZFS pool with 4K blocks on that large partition. Each filesystem block touches two physical sectors. The misalignment causes write amplification, destroying performance. For average spinning drives, assuming that all your disks use 4K physical sectors is safest. Certain SSDs also expect partitions to be aligned along 128 KB or 1 MB boundaries. Some enterprise drives use 8K sectors, however, and a handful of specialty devices even larger.

Avoid alignment problems. Make all GPT partitions begin and end on megabyte boundaries.

My whole stupid career is built on filesystems. You can still get your name in this book, though.

OpenZFS Mastery is staggering along. Here we talk about how ZFS maintains uber-integrity.

Not having dedicated special index blocks sounds great, but every data tree needs a root. ZFS stores a pointer to the filesystem root in a special uberblock. Every pool has a queue of 128 uberblocks stored at algorithmically-predictable locations. In keeping with the copy-on-write design (Chapter XXX), uberblocks are never edited. Every time a new transaction group gets written to disk, ZFS records the new root information in the next uberblock in line. When the 128th uberblock is used, ZFS loops back to the beginning. At boot, the system searches for the uberblock with the highest transaction group number and uses that to find the pool’s root. If the newest uberblock appears damaged or incomplete, ZFS falls back to the newest usable uberblock. A badly timed failure might cost you the latest transaction group’s worth of data, but the pool itself will be coherent and will not require an integrity check.

OpenZFS Mastery is still open for sponsorship.

Here’s OpenZFS Mastery on physical labeling. I have strong feelings on this.

Develop a consistent naming and numbering scheme for your storage arrays, and use it dogmatically. Many storage arrays have a standard naming scheme, often printed on the equipment. If your equipment already has numbered shelves, use that numbering. Otherwise, make simple rules like “shelf 0 is always at the top and disk 0 is always at the left.” You might use the prefix “f” for the front and “b” for the back, or whatever works for you.

Record the serial number of each drive as you install it in the array. Physically label each drive tray with the drive serial number and physical location. Use good labels that remain stuck over years of being ignored in a dry, dusty datacenter. Yes, this is tedious—but when a drive fails you must have this information. You can do this work in peace and quiet at your own pace, or you can desperately rush through it during an artificially prolonged, unnecessarily stressful outage.

OpenZFS Mastery is open for sponsorships.