Blog

136: sshd and Emacs

Here’s a chunk from the caching chapter of OpenZFS Mastery.

The ARC surrenders memory readily upon request, but freeing memory doesn’t happen instantaneously. While the process of releasing memory from the ARC has been accelerated in recent OpenZFS, it will always take longer than allocating free memory. On a host with a very large ARC, dumping gigabytes of objects from the cache might take a measurable fraction of a second. Depending on your host’s purpose you might need to restrict the ARC’s maximum size, or require the kernel to preserve at least a minimum amount of RAM for the ARC.

Proxmox is intended for virtualization, and automatically caps the ARC at 10% of available memory. Linux OpenZFS defaults to capping the ARC at 50% of memory. FreeBSD goes the other direction and reserves 1 GiB of RAM for core system functions like sshd and Emacs, allowing ZFS to claim as much as it needs. No operating system vendor knows your specific workload, so all of these are equally incorrect.

I think I can get most of this book done before my surgery on 12 November. You sponsoring the book would help me pay my bills as I recover.

Posted on

October’s Orthopedic Sausage

This post went to Patronizers at the beginning of October, and to the public at the beginning of November. Not a Patronizer? You could be!

I’d like to say that the title of this month’s post is a clever pun in some way, meant to either expand your vocabulary or troll your deficient vocabulary.

It is not.

Terry Pratchett said, “They say that the prospect of being hanged in the morning concentrates a man’s mind wonderfully; unfortunately, what the mind inevitably concentrates on is that it is in a body that, in the morning, is going to be hanged.”

The mysterious pain in my arm that keeps me from sleeping well? I’ve severed my rotator cuff tendon and get to have surgery for it. Having surgery to fix a health problem is a privilege in the United States, but I’m not looking forward to the four to six weeks my right arm will be immobilized. It’ll happen some day soon. My hope is to be able to appear at my divorce hearings without being hopped up on painkillers.

What’s the date? No idea. I have an MRI tomorrow, then get to meet the surgeon

But I’ve stopped lifting my right arm as much as possible. I stop my martial arts practice. I’m still doing tai chi, but with only one arm. I can still dangle my arms at my sides, so my hip-mounted keyboards mean I can still write. For now.

Other projects that I talked about last month? On hold. I’m cramming to try to get as much of OpenZFS Mastery finished before I lose use of an arm. Ideally I’ll get a first draft completed and off to reviewers before No More Typing Day. I have no idea when that will be. I do know that if I wait too long to have it fixed, it will not be fixable and I will be in pain the rest of my life. So it’s getting fixed ASAP.

My Internet existence is typed. When No More Typing Day happens, I expect to be mostly missing for 4-6 weeks.

On the plus side, one of my future novels will feature a one-armed character and his struggles to adapt to a two-armed world. Writing a book about the experience will, of course, make the surgery tax-deductible. That’s how business works.

Fortunately, I can plan for temporary disability. The Twisted Presents Kickstarter is over, but I still must fulfill it. I’m ordering the signed books today, because they have to be shipped twice. My ever-trusty Director of Analog Distribution, Alex, is standing by to fulfill DAD duties such as the dropship spreadsheet and chasing survey respondents. Worst case, I must sign Patronizer and backer copies of Twisted Presents with my left hand. That’ll be a fun collectors’ item, for very low values of fun. (Yes, yes, before you comment I’ll add: “and collectors.” There. Happy?)

Fortunately, a friend has offered to stay with me for a few days after the surgery. I’ll have two spare hands to help me sort out what I missed preparing.

So am I going to spend six weeks watching Star Trek, in order?

Perhaps. Or perhaps I’ll… take the opportunity to try dictation again?

Dictation is weird. I cannot speak coherently. My vocabulary exceeds the abilities of any dictation program I’ve tried. Writing sysadmin books with dictation is not possible. With SSH Mastery I tried a hybrid approach of speaking parts of the prose and typing commands, but my copyeditor immediately identified the dictated parts by error density. Oops. My fiction uses the characters’ voice and vocabulary, which means training a single dictation program to manage all of my fiction is a non-starter. AI dictation? Bah! AI generates minimum viable text which, despite the label, is completely non-viable. I know professional novelists who use dictation. Only a handful use software. Many hire transcriptionists.

This might be the prompt I need to finally write that book on how to make a living writing.

That’s a book I don’t particularly need to write. Whatever I say, people will tell me I’m doing it wrong. People will tell me I’m lying, or greedy, or arrogant. Greed is relative, of course. I want to pay my rent and get through my surgery without needing a GoFundMe. Some people aren’t doing as well as me. If I needed big money I would have stayed in tech. For decades, my goal has been to make “enough.”

The book would make trouble, but it sat in my brain like a noisy cricket hiding in the bedroom. I outlined it a few months ago, mostly to inescapably bind it within my literary spell book.

BUT–I would write that book in my own voice, using a comparatively simple vocabulary. Which means dictation might work. It won’t. I know it won’t. But it might. The attempt might distract me as I recover. Or, I could just give up and watch the ~850 existing hours of Star Trek, except that would involve once again witnessing “Turnabout Intruder” and I’ve sworn an oath in that regard.

The good news is, the core structure of OpenZFS Mastery is complete. The difficult work was coherently explaining the core concepts. What remains are crenellations and outbuildings built on that knowledge. The remaining chapters?

  • “Encryption” is brand-new.
  • “Replication” has a working shell from the first edition, but needs updating and expansion for new features.
  • “Hot Spares” will be easier than folks think, because mostly you shouldn’t.
  • “DRAID and AnyRAID” won’t be too horrid.
  • “Caches” needs updates, but the first edition text isn’t actively terrible
  • “Sharing” is a bunch of straightforward small topics riveted together
  • “Performance,” “Tuning,” and “The ZFS Debugger” are my favorite type of chapters: Allan’s problem

Can I finish this before surgery? Who knows. I’m gonna try, though.

After all, my mind is concentrated upon being hanged.

135: What Flunkies Are For

Today I’m working on snapshot replication for OpenZFS Mastery.

While some environments require backups be dumped to offline media, others find online backups sufficient, or use online backups for warm spares. Lucas assumes that every system he controls will simultaneously betray him, so he uses both. Jude does something that he’s gonna fill in here.

Backups that must be run manually are not backups. You won’t do them. Sure, you’ll do them once or twice, but one day the coffee pot will break and you’ll barely remember where you left your spleen. If you delegate them, your flunky will get overloaded and forget. That’s what flunkies are for. Actual routine backups demand automation. The two popular tools are syncoid and zelta.

Syncoidis the integrated sanoid replication tool.

Zelta is a powerful policy-guided backup suite designed to satisfy compliance auditing. If you need to configure massive backups via metadata, satisfy external auditors, or if you daily use the word “pebibytes” in anger, investigate zelta. Most people don’t need that much power, however, so we’ll discuss syncoid.

For Halloween, why not treat me to an OpenZFS Mastery sponsorship? Really trying to get a first draft done before 12 November, so I can get it to review before my shoulder surgery disables me through the end of the year.

Posted on

New story: Tormented by Teledildonics

Let me get this out of the way: I am a bad person and I feel bad.

I thought I was done with this crap.

But last week was bad in computer security. PHP got rooted–no, no, you can’t root a computer language, except when every web site based on PHP is vulnerable to an attack, PHP got rooted. On the same day, digital security company eMudhra was found to have deployed a poor private key for signing X.509 certificates. Only a digital security company with very special skills would use a private key of “4.”

People without deep expertise might ask “how bad was this, really?” Several other security firms immediately disclosed failures that, any other time, would have triggered blaring headlines and deeply annoyed techies. They were mere blips. This was the kind of week where everyone affected needed a shot of whiskey and a prolonged hug.

I had neither—but I did have a full-day meltdown. I bounced it past fellow author Lucy Snyder. She raised it from a shriek of pain to a tale with depth. After bouncing it between us a few times, there’s no way to tell who wrote what without digital analysis.

Allow me to present: Tormented by Teledildonics: A BDSM Cybersecurity Incident by myself and L A Snyder. (Why is Lucy using a pen name? Because BookScan. If you don’t know what that is, congratulations, you have lived a good life.)

Print is on the way. Patronizers, your ebook is coming. Other stores are coming, but they’re slower than my store.

If you search book catalogs you’ll find this in the categories Fiction/Erotica/BDSM, Fiction/Crime, and Computers/Embedded Computer Systems. After perusing Amazon’s offerings, this work is positioned to become the industry’s standard reference cautionary tale on teledildonics.

134: The Clunky NVMe Interface

One PHP Apocalypse later, here’s OpenZFS Mastery on NVMe.

The slowest, most complicated, and most error-prone part of an SSD is the Flash Translation Layer. Pretending to be as stupid as a 1980s spinning disk is hard labor. Non-Volatile Memory Express (NVMe) uses very similar hardware but drops this clumsy pretense, instead offering an interface designed for fast flash memory. NVMe itself is a physical interface specification like SATA or SCSI/SAS, but designed for extremely fast access.

Where spindles and SSDs have one queue per disk, NVMe allows 65,536 queues per device, of 65,536 commands each. Most NVMe drives use one read and one write queue per CPU. By spreading the load across CPUs, NVMe queues can be kept relatively shallow, allowing high-priority tasks fast responses and boosting general performance. The NVMe interface requires less locking while providing far greater parallelism and performance.

Still grinding to get a first draft done before my shoulder surgery next month. I’d appreciate your support.

133: Cleverness Is Your Only Option

Here’s a tidbit of OpenZFS Mastery about replication.

Replication also greatly simplifies migrating data, even huge amounts of data over ridiculous distances. With enough data, even 40-gigabit Ethernet in the same datacenter seems slow! Maybe you have to migrate a massive dataset to the other side of the country, or across the planet. Suppose you have many tebibytes of data undergoing constant minor churn, such as an orders database for a busy ecommerce site. Copying that data across the country or around the world will take days—but by then, the data will have changed. An rsync-based replication process takes so long that it will probably never catch up. Your database administrators are smart folks, and probably can brew up with a clever migration plan with complicated data segmentation. “Clever” is a bad word in system administration but sometimes, cleverness is your only option. These plans increase risk and even when successful impose frustration and stress. When they go wrong, they go wrong big.

I’m pushing to get a first draft done by the middle of the next month, before my shoulder surgery leaves me unable to work until the end of the year. Your support would be appreciated.

132: A Monstrous Heap of Redirections

I’m pushing to complete a first draft of OpenZFS Mastery in the next few weeks. Here’s a bit on ZFS and lightweight virtual machines.

Lightweight virtualization like Linux containers, Illumos zones, and FreeBSD jails are the most efficient way to isolate virtual systems from the host. Rather than making the virtual machine deploy its own networking stack, memory cache, and filesystem, the main host’s kernel handles those tasks and reduces the virtual machine to an isolated process space. Some applications either need to control the filesystem or can benefit from controlling the filesystem, however. For most filesystems, you’d need to deploy a full virtual machine and stack a filesystem on a disk image on a file system in a monstrous heap of redirections. A host running ZFS can delegate a dataset to a lightweight virtual machine, granting it complete control.

FreeBSD and Linux have different approaches to lightweight virtualization. FreeBSD creates single namespaces that can be further subdivided, while Linux offers smaller namespaces that can be aggregated into larger entities. Neither is wrong.

You sponsoring OpenZFS Mastery would help me meet my bills.

Posted on

September’s Syzygant Sausage

This post went to Patronizers at the beginning of September, and to the public at the beginning of October. Not a Patronizer? You could be!

Despite my nature, I am committing socialization once a week. It’s either that, or pillage my next book title from the Simpsons.

So I’m visiting friends a few weekends ago. We’re looking through their movie library for the evening’s entertainment, because nothing says “hang out with introverts” like a social activity that requires no interaction with live humans. They go past Lord of the Rings and say, “yeah, that’s great, but it’s twelve hours.” I said, “Yeah, one day I have to make time to see that.”

Everyone in the room froze. I’m pretty sure I heard a record scratch.

One says, “Are you saying you never saw Lord of the Rings?”

“I wasn’t a full time writer yet. I spent that time writing.”

Silence.

“That’s how it’s done. You want to be a writer, you give up fun things in favor of writing.”

The group silently achieved immediate consensus. I found myself inexorably propelled to a prime seat directly in front of the big screen while someone queued up The Fellowship Of The Ring–the extended version, obviously. I didn’t get a vote, because my decision-making skills are clearly so deficient that I shouldn’t be allowed one. Once that movie finished, an appointment was declared to watch The Two Towers.

Anyway, I have now seen the entirety of The Lord of the Rings.

I have read LotR several times, of course. The world is magnificent. Every reread renews my deep objections to Aragorn’s whole Chosen One narrative and the authoritarian premises underlying it. A rightful king won’t save us. On the other hand, one man does make a difference–but only with the support of a broad community. Except he fails at the end and a friend must rescue him. That part’s pure truth. Also, women have always fought. That’s a long way to go to say that every time I reread LotR, I fight an impulse to do my own take on the saga. The movies also triggered that urge.

But first: finish works in progress.

OpenZFS Mastery is progressing. The book will have 21 chapters (numbered 0-20 of course), making it the largest Mastery book by far. It will be, as the kids say, a “chonk.” It’s an aggregated, rewritten edition of two books so that makes sense. When FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS came out, print-on-demand technology couldn’t cope with thick books. It can now. The downside is, it will also be the most expensive Mastery title. I’m thinking that the ebook will be $19.99 USD. Prices here in the US are going up, though, which makes me think that perhaps I should raise the price of all my ebooks. That’s a huge amount of work, though, so I’m going to continue letting the thought simmer on my brain’s back burner while I cook new books up front.

I hope to launch the Twisted Presents Kickstarter on 22 September, depending on various legal issues to be discussed the day before. The campaign is ready to go and a print proof is on its way to me.

Given the recent price increases for printing and shipping, I revisited my cost spreadsheet to recalculate how I needed to price the paperback and hardcover. To simplify the next recalculation, I documented the process of doing so in the TWP knowledge base and, in an effort to make other people’s lives easier, shared that doc on social media. Reaction was very positive. Apparently many people calculate their Kickstarter prices by looking at what successful campaigns charge and doing that, to which all I can say is “golly jeepers, you’re gonna lose your home that way.” I did my first indie cost calculations in 1991, so I had forgotten just how overwhelming this can seem to the uninitiated. I also did that calculation without a spreadsheet, uphill in the snow both ways, barefoot, so feel free to insert an old man rant here. Anyway, that made me realize that I have enough material to write the oft-requested “How I Make a Living Writing” book. I’m not sure I want to write that book, though. The title alone feels even more narcissistic than the rest of my books. I did sketch out an outline for it, but mostly to confine the idea to paper in the hope that it leaves my brain.

On the fiction side, I’m loading the Giant Fiction Project back into my brain and contemplating smaller things. With another 10,000 words or so I could launch a Rats’ Man’s Lackey collection. I’m about 35,000 words short of the next Prohibition Orcs collection, which is a bigger reach. At least that one has a title: “Found Meat.” It’s a good day when you find meat. I have most of the parts to do the final $ git commit murder novel, but if I keep getting diverted I’ll never finish Big Fiction Project. One could argue that BFP is a diversion from the mystery trilogy, though. I want something on Kickstarter during my Penguicon GoH appearance, though, and OpenZFS Mastery had better be launched several months before then.

Really, the key is to get back to routinely making words. Mind you, my Amazon wish list now includes the Blu-Ray of LotR, so there might be… issues.

From the pit of my tarry heart, I thank you for your support.

131: Lucas is Incompetent or Evil

Folks sometimes forget that OpenZFS Mastery has a co-author. Here’s a bit Allan Jude originally wrote.

But what if Lucas is incompetent or evil? You give him access to take his own snapshots and he will try to create a million “just to see what would happen.” You know people like that.

ZFS has your back when dealing with these difficult users. The snapshot_limit and filesystem_limit properties allow you to restrict the number of snapshots or child filesystems that can be created under a dataset. Set these properties to one greater than the number of snapshots or datasets you want the user to create. If you set snapshot_limit to 10, the user can create nine snapshots. The tenth generates an error. The read-only filesystem_count and snapshot_count properties allow you to quickly see how many filesystems or snapshots exist, and compare that number to the limit. To contain the pure evil that is Lucas, here we restrain him to only two snapshots.

OpenZFS Mastery is looking for sponsors, and the Twisted Presents Kickstarter is open for a couple more days.