Have some OpenZFS Mastery.
“ZFS reports that one of the seventy-nine drives in our array is failing? Great! We can replace it in a convenient maintenance window before it causes problems. Uh… which drive is it?”
If you have a large storage array, you need proper physical and logical labeling. Keep a spreadsheet of each drive’s physical location and the information presented to the operating system, and also document this within ZFS itself. It’s easier than it sounds. Proper preparations during installation let you zero right in on a failed disk—even a disk at a remote facility. Jude runs a lot of very dense storage arrays in locations all over the world, and uses this scheme to keep hard drive maintenance from overwhelming him. He even records warranty information, expiration dates, model numbers, and more within ZFS so his monitoring system can flag sixty days before the warranty runs out.
It’s a couple days before I wanted to announce it, but if the new OpenZFS book is of interest you might look at https://sponsor.mwl.io.