129: The New Linker Strangles

OpenZFS Mastery has reached chapter 9, on boot loaders.

The most tediously terrifying system administration task is an operating system upgrade. The new kernel might not boot the system, but that’s the least of your problems. What if a critical program requires an old version of a shared library? Maybe that new terminal mode is subtly incompatible with your application. Or perhaps the new linker strangles your mission-critical software. Sometimes problems aren’t immediately apparent but creeping damage becomes visible after a week or two, when reverting the upgrade becomes even more difficult. No matter what precautions you take or how much testing you perform, any upgrade can go bad.

Sysadmins have developed all sorts of tools and coping strategies to manage bad upgrades. Boot loaders help you quickly recover from bad kernels. Backups help you slowly recover from bad userlands. But none of these help you understand exactly what went wrong and duplicate the problem.

Unless you’re running ZFS.

OpenZFS Mastery sponsorships are still open.

August’s Aqueous Sausage

This post went to Patronizers at the beginning of August, and to the public at the beginning of September. Not a Patronizer? You could be!

The apartment complex opened the pool. It’s right beneath my balcony. Now that my smaller desktop works, I can roll out and work while sneering at the people in the pool doing weird things like “relaxing” and “having fun” and “not melting in the heat,” all activities I have never tried and can thus authoritatively declare lack merit.

I attended a writers’ gathering at the end of July. Why a gathering? Conference is too strong a word. There were no scheduled events. Party isn’t right, either. Two colleagues invited a few dozen writers, artists, and publishing professionals into the mountains of Washington State to one, discuss the state of the industry, and two, allow the locals to sneer at those of us who thought we were in the mountains. There was barbeque. I learned some things. I reinforced old contacts. I reminded myself of why I do things this way and that this silly career could be fun. Exhausting, but worth the trip.

More exhausting was getting up at 2AM to catch my flight home yesterday. I slept ten hours last night. I’m walking into doorframes today. My youthful resilience has gone on vacation. I hope it comes back when its two weeks are up.

OpenZFS Mastery is taking shape. This book is structured much like a castle. The first third of the book is a massive foundation of knowledge. We have finished that foundation, subject to dashing back in to fill in details we forgot. Now we’re proceeding to individual narrow towers built atop that knowledge, with occasional outbuildings for zvols and replication automation, not to mention the deduplication dungeon. In some ways, these smaller topics are easier. The foundational topics are deeply intertwined and require careful handling so the reader can deeply absorb them all. Topics like deduplication and DRAID are downright bizarre and require careful handling for entirely different reasons. The manuscript’s about a third of the way done, but the foundation is clearly the largest, most complicated part. Yes, there’s more complex topics like DRAID and deduplication, but they are smaller and thus easier to organize.

One strange thing about revisiting books from ten years ago is discovering where they were insufficiently accurate. We discussed native file compression in FreeBSD Mastery: ZFS, but on a fairly superficial basis. The discussion saved many people a lot of trouble. Great. But a decade later, Allan has gone from “interested in ZFS” to “running a consulting service that writes new features for OpenZFS.” I, of course, have gone from “eccentric who wrote a shelf of books” to “full-on nut job who wrote a case of books.” We both know too much to ever be happy again, and can’t stop ourselves from asking questions. Where, exactly, does OpenZFS perform file compression? Is it on a sector-by-sector basis, stripe-by-stripe, or somewhere else? For many readers it will be a detail of minor interest, but for a handful the answer will be vital.

The books need updating for hundreds of details like this.

That means OpenZFS Mastery will be an entirely new book. Starting over would have been easier than revision, but we’re committed now.

Speaking of easier and starting over, I’m trying to work on fiction again. Yes, the nonfiction makes more money, but if I only write nonfiction the words dry up. The words have dried up for many reasons, but the only way to get them flowing again is to prime the pump and get started. I’m trying to write at least 50 words of fiction every day that I’m not indisposed. What is “indisposed?” That depends. Getting up at 2AM to catch a eastbound transcontinental flight, that causes indisposition.

With any luck, 50+ words a day will get me an orc Christmas story by the end of the month, letting me prepare to launch Twisted Presents. I can’t launch it, but I can completely prepare it.

Twisted Presents is a collection of Christmas stories. Half of them are crime stories. The other half are fantasies. A Prohibition Orcs story is both. I enjoy writing orc stories, and there are folks who will back a collection solely to get a new orc tale. I need to include one.

But:

In Michigan, divorces end with an even division of assets. Crowdfunding proceeds are an asset. Fulfilling a Kickstarter takes about half the proceeds. The date my divorce is final is determined entirely by the court; I don’t get a say, and I don’t necessarily get much warning. There’s a race condition where I collect Kickstarter funds, have to give half of them away, then have to fulfill the Kickstarter. I’m more than willing to split net, but splitting the gross means I get zero. Yes, I could pay a lawyer to negotiate around this, but lawyer fees would cost more than Twisted Presents is going to bring in. The divorce should be settled well before OpenZFS Mastery goes to print, at least.

Twisted Presents is a Christmas book. If I have it ready to go I can launch a ten-day campaign as late as Halloween and still fulfill. That requires I have a date for the divorce, though. I hope to know something by then.

We’re also waiting for the house to sell. We set the initial price before the US attacked Iran, though. The attack knocked down the real estate market, so we’re trying to ride the changes and see what we can get. (The political and economic stupidity that attack displayed is flat-out stunning.) If you want to live in Grosse Pointe Woods, in a house that’s guaranteed probably free of systemd infestation, this is your chance.

I wish I could tell you I’m doing something dramatic, but truly, I’m just grinding through the book. Making words, day after day. Attending the dojo 4-5 times a week. Saving money by cooking meals and avoiding processed crap. Attempting to sleep. You know, the usual.

I deeply appreciate your support as I get back on my feet.

Penguicon 2027 Guest of Honor

Me, a few days ago: Hey, Penguicon has some great Guests of Honor lined up! Not a dud in the bunch.

Then they posted this.

Looks like I’m going to be a Guest of Honor at Penguicon 2027. I was previously a Penguicon GoH in 2013, so obviously they didn’t learn their lesson. The con runs April 8-11 2027 so if you’re in Detroit, do sign up. Other guests include Dr Rob Pyatt, Jennifer Brozek, John Chu, Phenderson Djéli Clark, and Seanan McGuire. With heavy hitters like this, I’m not sure why they need me?

Perhaps it’s just that if you give too many talks at Penguicon, they make you sit at the front of the class.

128: Filthy Land Animal Lungs

There’s one story in Twisted Presents I haven’t read here, so that’s what you get.

This isn’t a fairytale, but: Once upon a time, we were all friends.

And it’s not going to end happily ever after. On this Christmas Eve, three tedious years after the disastrous college summer when Kurt betrayed us, Phil and I are finally making an ending.
Kurt’s ill-gotten mansion looms on an isolated northern California cliff overlooking the Pacific Ocean. This deep in the night, the scents of ocean and eucalyptus rule the air. After crossing thousands of miles of open water without being soiled by filthy land animal lungs, the air is so rich that it makes me feel a little high. I deliberately focus on the basics of burglary, keeping my breath slow and placing each foot meticulously on the kitchen’s porcelain tile. Enough moonlight seeps through the fog to illuminate the mansion’s insane luxury. Had Jim died so that Kurt could have two ovens large enough to roast a calf in? Was Sarah rotting in a Turkish prison the price of a mahogany-topped island? Were the dozens of high-end cookpots dangling from the ceiling worth Danny’s awful death?

Kurt thought so.

When will Twisted Presents launch? As soon as my life is in order to allow it.

Crowdfunding Pricing and Costs

I’m preparing to launch the Twisted Presents Kickstarter at last. The US economy has gone wrong lately, for obvious reasons. Costs for printing and shipping have increased. We’re not quite in a capital strike, but it’s dang near. I needed to revisit costs and prices for this book.

I took notes as I did so. If you want to do crowdfunding, here’s my approach. This, other stuff from that site, tidbits from my blog, and assorted other detritus will probably form part of a long-requested “How I Make A Living Writing” book.

Those notes are on the Tilted Windmill Press Knowledge Base.

The hardest lesson I had to learn when I started writing professionally? “If someone puts a dollar in your hand, take it.”

Sponsorship Prices Increase

The US Post Office has yet again raised shipping prices. I’ve ignored the last couple hikes, but everyone else from the grocery store to the landlord is raising their prices as well.

Some point this week, I’ll be raising the price of the OpenZFS Mastery sponsorships.

I’ll probably have to raise the prices of published books soon. That’s a week’s worth of work. I don’t want to do it, but that’s life in capitalism.

127: A Fifth of Makers’ Mark

I’ve just finished prepping the print version of Twisted Presents, so here’s a tidbit from one of the stories therein.

The first Hogswatch Eve after the accident, Merle had stopped on the way back from the hospital for a fifth of Makers’ Mark. He knew if he started drinking he’d never stop, but the next day was Hogswatch and he couldn’t think of any other way to survive the night. He woke the next morning with an empty bottle, an open sewer for a mouth, and a witch’s brew in his stomach.
Through all that, he could still smell the sheets. He hadn’t changed them in weeks. There wasn’t a reason to. The bed made a warm greasy nest that suited the wreck of his life.

But it was Hogswatch. He had to get to the clinic. Soon as he could. They were reading Night Watch. For when his voice gave out, he’d loaded a couple of movies onto a tablet. White Christmas and Gremlins. He hadn’t done much in the way of shopping, but he’d picked up a fruit and nut bar at the hospital gift shop a couple days ago. It was the first treat he’d ever bought her back in college, and all the feast he could handle.

The alternative was to lie here and cry. Again. Crying always made him feel worse.

I plan to launch Twisted Presents in time for you to get your books before Christmas. Because nothing says “Happy Holidays” like curses, firearms, and orcs.

126: Low Distress Tolerance

I finished the OpenZFS Mastery chapter on volumes earlier this week.

Zvols support sparse volumes, also known as thin provisioning or overcommitting storage. A sparse volume is a volume that reserves less space than the volume actually needs. Sparse volumes permit allocating more space than the pool has available. You could, for example, create ten 1-exabyte sparse volumes on your 1 TiB dataset. So long as your volumes are never heavily used, nobody will notice that you’re overcommitted. Sparse volumes can be useful if you’re prepared to add more disks. Using a sparse volume means you can tell the Windows host that a disk is a specific large size, even if the underlying pool doesn’t have that much free space.

Sparse volumes are not generally recommended. If the pool fills up but the sparse volumes think they have space remaining, writes will fail with an “out of space” error. Guests will get errors they can’t hope to resolve. Windows in particular has low distress tolerance.

I’d appreciate your support keeping the rats fed as I finish writing this book.

125: A Useful Quirk

I’m trying to finish up the OpenZFS Mastery chapter on disk space use today. Well, my part of it. I’m planning to dump–delegate, delegate–the section on deduplication to Allan. When you have a co-author expert, use them.

Zstd offers compression levels from 1 to 19. The default is zstd-3, a reasonable compromise between performance and efficiency. Writing data at higher compression levels require more CPU but might save more space. Zstd has a useful quirk where decompressing higher levels uses no more processing power than decompressing lower levels. If you have data that must be written once but read many times, increasing the compression level makes sense.

ZFS still supports gzip compression, but Zstd is always faster and achieves better results while using less processor and memory.

ZFS also offers the ZLE algorithm, for data with long runs of zeroes. Some file formats, such as GIFs and certain scientific data, include long runs of zeroes. If you have a dataset for such data, you might find ZLE compression useful.

How does one realistically tell if your data has long runs of zeroes? The simplest way I know is to pass it across a T1 with 23 channels set to b8zs but one channel stuck on AMI. Such a circuit cannot pass large all-zero packets. Sadly, this recommendation does not suit a book on filesystems.

I would appreciate your help keeping the lights on as I finish this book.

Changing Language for Storage Capacity

Back in the 90s a notably pedantic coworker informed us ignorant peasants that, according to International Electrotechnical Commission standard IEC 80000-13:2008, disks and memory were now to be measured in “bi” units such as kibibyte and mebibyte. I agreed that he was technically correct, but nobody cared and saying “mebibytes” made you sound like either a pedant or an idiot.1 For all practical system administration purposes, the difference between a 1.544 MB floppy and a 1.544 MiB floppy was negligible. Yes, the operating system showed everything in mebibytes, but everything was labeled as being megabytes, and we all knew what it means.

This is no longer true.

My main test host has six hard drive bays. Each has a 3 TB drive, which is only 2.7 TiB. The operating system shows everything in binary-based tebibytes. I bought 18 TB of storage and got only 16.3 TiB.

People look to my books for information to keep their jobs. That missing two TiB is part of the information. If you’re doing capacity planning, it’s vital. From now on, my books need to use bi units for most storage.

In some cases, it looks stupid. Disks are also sold with 4K sectors. The package says that. “K” is an ambiguous label. It could mean kilobytes or kibibytes. Technically, disk sectors are kibibytes. I think I’ll just keep calling them 4K disks.

Memory? I want to say that nobody cares and that everybody knows what we mean. I don’t run any applications that use terabytes of memory, and for routine system administration tasks the difference between 32 GB and 32 GiB is negligible. But someone does care, and mixing GB of RAM with TiB of storage seems odd. Dang it. I think I have to be pedantic.

But in the 90s? Nobody cared, and we all knew what we meant. Unless you worked in embedded systems.