62: Isolated But Not Alone

Got some words on #projectIDGAF this week, hurrah!

Music might have saved Will, but the valley’s only radio station spent ten minutes out of each hour praising Jesus, ten minutes advertising albums of popular music rewritten to praise Jesus, and forty minutes begging for donations to help spread the Word. The county health department knows that the station mostly spreads clamydia, but it also spreads bribes so everything works out. There wasn’t any chance that he’d encounter any of the music that would have helped. The Cure would have taught him he might be horribly isolated, but he wasn’t alone. A hit of Ministry or Front Line Assembly would have given a voice to his malformed masculinity, while Depeche Mode would have shown him that men are allowed to have the Forbidden Feelings. A hit of Prince would have taught him to move his feet. That alone might have saved him. Staying up late one night last summer he’d caught skip from an alternative station way out in Los Angeles and captured eighty-two seconds of enlightenment, but he had no idea the song was called “Assimilate,” or the band Skinny Puppy, so it didn’t help.

Today I’m listening to Bill Leeb’s lush solo album Model Kollapse and Allie Goerz’s delightful album of acoustic Nine Inch Nails covers. If you’re catching this episode on release day, October 4 is Bandcamp Friday. Buy music and the band gets all the money.

Mail Talk 8 October 2024, with bonus Craig Maloney Memorial Charity Auction starting–NOW

Next Tuesday, 8 October 2024, I’ll be talking about Running Your Own Mail Server at mug.org, 6:30PM EDT. MUG is my local “hard-core Unixy People” group. Giving a talk during a book release is bad planning, but I am crap at scheduling.

One of its members was Craig Maloney. Many years ago Craig asked me if I was the same Michael Lucas who had written a couple RPG books in the 1990s. I admitted my guilt. He pulled an obviously-read plastic-bagged copy of Gatecrasher out of his backpack and asked me to sign it. The dude had friends across the world and did his best to boost us all. An all-around great guy, who sadly lost his life to cancer earlier this year.

Craig had sponsored Run Your Own Mail Server. I am now left with his sponsor gifts. I’ve checked with Craig’s family, and they’re okay with me auctioning them off for charity. The Craig Maloney Memorial Auction runs on this page from now until my MUG talk ends1.

The sponsor gifts will never be available in bookstores, at least not new. (I do have a few extras that I will auction off for charity over the rest of my life, but I’ll stretch those out.) I don’t want to describe them here because not all the sponsors have their gifts yet and I’d rather not spoil the surprise, but you can see photos at link 1 and link 2.

I’m going to end this auction a little differently, though. The auction will close at the end of Tuesday’s mug.org talk. I’ll ask live, online for any last bids. You can bid by posting on the page or in the video session. The auction will close when bidding stops. Comment on this post to bid. Once the auction ends, I’ll notify the winner. The winner sends me the donation receipt and I ship the gifts. I pay for shipping.

The beneficiary is Mutual Aid Disaster Relief. They’re as close to the ground as you can get these days, and donations are tax-deductible in the US. You can choose from several donation targets. I don’t care if you donate cash, fill an Amazon shipment with the North Carolina wishlist, target Puerto Rico, or whatever. Just get them the money and get a receipt.

Anyway, leave a comment to bid. Good cause. Ridiculous prize.

September’s Sibilant Sausage

[this post went to Patronizers at the beginning of September, and to the public at the beginning of October.]

Pretty sure August was eating locoweed.

The “Run Your Own Mail Server” Kickstarter owned most of my hide this last month. Not all of it. A patch on the back of my neck remains freehold. I managed to make a few words on what I’m calling #projectIDGAF, but mostly it’s been investing in production stuff. Which means spreadsheets.

My main printer, IngramSpark has facilities in the US, UK, Australia, and Italy. When I launched the RYOMS Kickstarter, I intended to dropship copies through them. Turns out, it’s not quite that easy. Part of the problem was scale. Based on previous Kickstarters, I thought I might need to dropship to thirty, perhaps fifty people. I got over seven hundred. The IngramSpark ordering interface is tortuous. I am not capable of correctly entering seven hundred orders in that interface. I began looking for a virtual assistant. Found one.

Then I discovered a way out.

Turns out that IngramSpark has a secret industrial-scale ordering system that accepts orders via spreadsheet. Gaining access to it requires you have a friend who already has access, who is willing to vouch for you. Fortunately, I have such a friend. You also need to be submitting several hundred orders. I barely qualified. (Random people on the Internet, please don’t contact me asking me to vouch for you. I don’t endorse random Internet people.) It’s an Excel spreadsheet, complete with macros, that must be filled out in a very specific manner. You know, like every application written in-house by non-programmers. Once you grow accustomed to its quirks, though, it’s infinitely better than entering orders by hand.

I’m keeping the virtual assistant info, though. With luck, I’ll need them later.

When it came to ordering books for backers in the EU, the plan fell apart. The EU has VAT. I have never worried about VAT. I don’t have to worry about VAT until I hit ten thousand euros of EU business per year. Even with RYOMS, I didn’t hit that. When I ship from the US, recipients pay VAT as part of the delivery. It varies by country, but the general pattern seems to be “recipient is contacted, recipient goes to a web site to pay, carrier delivers package.” My sponsors and Patronizers are pretty familiar with how that works.

When I cross ten thousand euros a year, I have to register for the Internet One Stop Shop VAT. This is expensive, but if I’m doing over ten thousand euros a year it would start to be worth it. That’s very much a First World Author problem, though.

If I print books inside the EU, the books would be mailed to recipients without those fees. The problem is getting people to print books in the EU. IngramSpark’s interface to their Italian plant is in the UK, and is legally treated as a UK entity. (I don’t pretend to understand the details, but presumably they have the contracts and lawyers to make it legit.) Brexit fubar’d everything for me there. There are other printers in the EU, however. Some of them would print a few hundred books for me! Except every one of them wants my IOSS paperwork beforehand. It doesn’t matter that I don’t need IOSS. Printers run quite conservative businesses, and take zero risks. It doesn’t matter that even with the lightning strike of RYOMS I don’t meet IOSS limits.

So I’m shipping most backers globally from IngramSpark. Based on the advice of assorted experienced folks, I’m using BookVault to fulfill EU orders. The books will be shipped from the UK, which is greener than shipping from the US.

I started fulfilling dropship orders in Australia, mostly because I needed a smaller group to test Ingram’s spreadsheet ordering but also because Australia is traditionally last in everything. The Australian copies have started to arrive. The rest of the world should follow shortly.

Then there’s books for me to sign. I have four crates of paperbacks in my living room to sign, pack, and ship. Hoping the hardcovers arrive soon, as well as the backer-exclusive special editions. I have something special for print level sponsors and Patronizers this time. Silly, but special. I’m hoping I can tell you about that next month, but the recipients need to receive them first.

Once those go out, I can launch the Dear Abyss Kickstarter. Quite a few people are telling me that the RYOMS Kickstarter is my new normal. As much as I’d love to trade up to that problem, I have no reason to believe that’s so. And seriously, Dear Abyss is not going to push me over the IOSS limit. If the new edition of Networking for Systems Administrators was to also experience explosive crowdfunding I’d look more seriously at IOSS, but not before. I don’t act based on lightning strikes until I start consistently attracting lightning.

Am I ignoring the success of RYOMS in my planning? Nope. There’s clearly a market for crowdfunding tech books. I’m hoping it will raise $20k, but will leave headroom for more. Hope for the best and plan for WTF, that’s the business.

After all these big projects, I need to write a palate cleanser. Something daft, and quick, and fun. I’m starting something I won’t talk about in public, yet, but if you’re curious you could follow #projectIDGAF on the fediverse. Why that hashtag? I have no idea if this thing will work, or even if it can work, but I’m going to have fun with it and that’s all that matters. I’d like to knock a full draft off by the end of September, but we all know that’s not going to happen. After a few years of these heavy projects like RYOMS, SNMP, TLS, and so on, my spirit needs a quick hit of weirdness.

In unrelated news, I sold five short stories to various anthologies at the beginning of the month. They include a new Aidan Redding tale, a Rats’ Man’s Lackey tale, and some one-offs. Look for those to escape in 2025. I’ll also have a new Rats’ Man’s Lackey tale in the next issue of Pulphouse.

Anyway. Off to sign a bunch of paperbacks, and maybe even get them mailed!

“Run Your Own Mail Server” is leaking out

My latest tech book, Run Your Own Mail Server, is starting to creep into bookstores. The book entry on my web site links to various stores that carry it, and will be updated as more stores appear.

Paperbacks are available on Amazon and will reach other stores shortly. They’ll be in the Ingram catalog, so you can have your local bookstore order them via ISBN 9781642350784.

Hardcovers are pending. Once the Ingram databases finish churning, they’ll also be available everywhere. Ask your bookstore to order ISBN 9781642350791.

I’m working with BookVault to manage direct print sales from my web site, but their Woocommerce plugin hit my store and promptly soiled itself. In their defense very few authors have been running direct sales for over ten years, and most of those don’t have as many features as tiltedwindmillpress.com.

Speaking of my bookstore, you can get the ebook there.

Moving Virtual Machines to Jails

I recently learned that I could rent a dedicated machine from bloom.host for less than I’ve been paying for my virtual machines. Time to move some VMs to jails! Here’s the notes I’ve left for myself. All of my VMs run ZFS.

First, clean up unneeded boot environments, remove any unnecessary crap that lingered on the VM, apply all security updates, and in general tidy up the source VM.

Then decide how you want to flip services over. The cleanest way is to shut down all services and start the migration, but you might need to guarantee uptime. It’s up to you. I chose to leave services running during an initial replication, shut down services, do an final snapshot with an incremental replication, start the new jail, and change DNS to the new addresses. Figure out your own uptime requirements.

Start by creating a recursive snapshot of the system.

# zfs snapshot -r zroot@bloom

At a convenient time, I’d go to destination host and pull the snapshots over. The snapshots need to go into a directory on the zroot/jails dataset, named after the VM the jail will replace.

$ ssh mwlucas@www.mwl.io zfs send -Rc zroot@bloom | zfs recv -v -o mountpoint=/www zroot/jails/www

This might take a while, so follow up with an incremental right before you want the actual the migration.

$ ssh mwlucas@www.mwl.io zfs send -Rci zroot@bloom2 zroot@bloom3 | zfs recv -v -o mountpoint=/jails/mail zroot/jails/www

if you’ve tampered with new datasets between copies, you’ll get an error.

receiving incremental stream of www/ROOT@bloom3 into zroot/jails/www/ROOT@bloom3
cannot receive incremental stream: destination zroot/jails/www/ROOT has been modified
since most recent snapshot
warning: cannot send 'www/ROOT/default@bloom3': signal received
Broken pipe

Roll back the problem dataset.

# zfs rollback zroot/jails/mail/ROOT@bloom2

Data’s moved over, but there’s trouble.

$ zfs list
...
zroot/www 39.6G 776G 132K /www
zroot/www/ROOT 22.5G 776G 132K /www/ROOT
zroot/www/ROOT/default 22.5G 776G 21.8G /www/ROOT/default
zroot/www/usr 10.9G 776G 132K /www/usr
zroot/www/usr/home 9.37G 776G 384K /www/usr/home
zroot/www/usr/home/acme 7.10M 776G 7.10M /www/usr/home/acme ...

The jail boots from the boot environment /www/ROOT/default, but the jail’s root dataset is /zroot/www. It’s empty. Shuffling datasets and rearranging inheritance is a pain. I just duplicated the contents

# zfs mount zroot/jails/mail/ROOT/default

$ tar cfC - /jails/www/ROOT/default/ . | tar xvpfC - /jails/www/

# zfs list zroot/www
NAME USED AVAIL REFER MOUNTPOINT
zroot/www 41.4G 774G 132K /www
zroot/www/ROOT 22.5G 774G 132K /www/ROOT
zroot/www/usr 10.9G 774G 132K /www/usr
zroot/www/var 7.96G 774G 132K /www/var

Go into the jail’s root directory. Edit /etc/sysctl.conf to remove non-jail settings. You can also edit rc.conf for the new network interface and the new IP.

I’m using VNET, because otherwise I must configure on-system daemons to avoid binding to localhost. (Remember, in a non-VNET jail localhost is aliased to the public IP!) That means I need a bridge interface. This host has one live Ethernet, igb0 so I make it a bridge.

autobridge_interfaces="bridge0"
autobridge_bridge0="igb*"
cloned_interfaces="bridge0"
ifconfig_igb0="UP"

I then add a public IP to the bridge, for the host’s use.

Now for jail.conf for a VNET install. I need to allow devfs for running named(8) on some of the VMs, and I want raw sockets.

path = "/jails/$name";
mount.devfs;
devfs_ruleset=5;
exec.clean;
allow.mount.devfs=1;
allow.raw_sockets=1;

exec.consolelog="/jails/$name/var/log/console.log";

vnet;
exec.prestart += "/sbin/ifconfig epair${jid} create up";
exec.prestart += "/sbin/ifconfig epair${jid}a descr 'vnet-${name}'";
exec.prestart += "/sbin/ifconfig bridge0 addm epair${jid}a up";
vnet.interface="epair${jid}b";

exec.start = "sh /etc/rc";

exec.created="logger jail $name has started";

exec.stop = "sh /etc/rc.shutdown";
exec.poststop += "ifconfig epair${jid}a destroy";
exec.poststop +="logger jail $name has stopped";

.include "/etc/jail.conf.d/*.conf";

This reduces individual jail.conf entries to this.


www {
jid = 80 ;
}

At this point, I could start the jail and see what broke. Some common errors included /tmp losing the sticky bit and MariaDB directories being owned by root rather than mysql.

Change the DNS, and watch traffic shift to the new host.

Am I confident in this process? No. That’s why I make sure I have a last backup in Tarsnap, and wait 30 days to delete the source VM.

Why “Run Your Own Mail Server” is not in Amazon’s Kindle store

I expect folks to ask this, so here’s a pre-emptive blog post. TLDR: for the same reasons OpenBSD Mastery: Filesystems is not. Amazon’s deal is unacceptable.

You can get Run Your Own Mail Server for Kindle direct from me at Tilted Windmill Press or at Gumroad. You can get a Kindle-friendly ebook from any number of other retailers, but while they’re all supposed to be DRM-free I can’t advise on prying the file out of another vendor’s ecosystem. The one place you cannot buy RYOMS for Kindle is Amazon’s Kindle bookstore.

TLDR: Amazon pays roughly 70% of retail price for books priced up to $9.99, and 35% for books $10 and over. Amazon is the only retailer that does this. Other retailers, I make somewhere around 65%-70% no matter the retail price. Everything follows from that math, but if you want the details read on.

According to economists, prices have gone up about 30% since I started releasing the Mastery books. According to my wallet, not so much. In 2012 I could get a cheap lunch for my wife and I for $10. I paid $18 last weekend. But let’s go with the official numbers. Just as “dime novels” now cost $10, I must raise prices. While book pricing is hotly debated, $14.99 is a reasonable price for a 350-page tech book like Run Your Own Mail Server.

If I charge $9.99 for this ebook, I make about $7.

If I charge $14.99 for the ebook, I make about $10.50 everywhere but Amazon. At Amazon, I make $5.25. For me to make that $10.50 at Amazon, I must price the book at $29.99. I’m fond of the book, but it ain’t worth that! And if I did, giving Amazon a $20 slice of every sale for no reason sticks in my craw.

Charge $29.99 at Amazon and $14.99 elsewhere? Amazon’s program has a Most Favored Nation clause. They can price match any other major vendor.

Will Amazon change their business because of this? No. Authors are plentiful and of low value. I am not worth Amazon’s time.

Amazon’s business model is based on squeezing prices down, and they play a long game. I don’t expect them to ever raise that $9.99 limit. A novel might sell tens or hundreds of thousands of copies. If I’m lucky, a book like RYOMS might sell five thousand copies at retail. (Why that many? The Kickstarter went viral, and I suspect it ate through the market.) The few extra bucks I’ll make by raising prices are important. That’s also why I’ve focused so hard on disintermediation through my Patronizers, sponsorships, and lately Kickstarter.

I have been expecting this for years now. I do not expect to publish future Mastery books on Amazon’s Kindle store, unless by some chance I write another very short one.

61: An Abiding and Passionate Desire

Shipping backer copies of Run Your Own Mail Server has owned me for days. I haven’t been able to write a word for two weeks now. So here’s a chunk from Network Flow Analysis.

Network administrators of all backgrounds share one underlying, overwhelming desire.
It doesn’t matter if you manage a network with 400 separate manufacturing plants connected by a global MPLS mesh or if you’re responsible for three computers and an elderly printer. Network administrators all share an abiding and passionate desire for just one thing: We want our users to shut up.

Blaming the network is easy. The network touches everything. Businesses assume that the network will work perfectly and make decisions accordingly. A user can’t open that 900MB Excel spreadsheet on the file server on another continent from his 20th-century PC? Network problem. A website in Farawayistan is slow? Network problem. A user can’t get a faster response over a 33.6Kbps modem? Network problem.

Yes, a real publisher paid me for this tripe.

“Run Your Own Mail Server” sponsor and Patronizer gifts

I spent the weekend transforming crates of stuff into a heap of packages.

Dear sponsors, you have a gift coming. It is not a copy of Run Your Own Mail Server. Go read the fine print on the description of what you backed: I said I will send you a gift, not a copy of the book. Your package contains not one but two items. They are irreplaceable, so when you trick them into falling into the smelter they will be gone forever.

Tricking them won’t be hard. They’re not that smart. Merely irreplaceable.

This heap also contains books for folks who chose the special edition during the RYOMS Kickstarter campaign. I have to make one more pass through the backer list to catch people who gave me their shipping information after I did the initial shipment. After that, I’ll check back in a month or so. I can’t force folks to give me their addresses.

I learned some important things in creating this heap.

Next time I ship gifts for a sponsorship that’s open longer than a year, I need to contact every backer and get their current shipping addresses. Probably some sort of web form for address collection.

Signing and shipping hundreds of books is a right pain. Next time, I must hire help. A teenager willing to help stuff books and carry boxes would have made this so much easier.

You must have IOSS paperwork to work with printers inside the EU. It’s not a legal requirement, but printers don’t want to work with any outsider who doesn’t have proper tax paperwork. You only need an IOSS number if you’re doing 10,000EUR or more of direct sales with the EU, however. That excludes me. That could change, but I don’t anticipate that happening. When I do a dropship-based sales, I’ll plan on shipping from the UK.

I still believe that this book was what authors call “a lightning strike.” These sales are not my new normal. The next book will have fewer backers, and that’s fine. I’ll enjoy the brief triumph and get on writing the next book.

Once the acetaminophen kicks in, that is.

60: Free Probing from the Greys

My brain is tired after the email book, and needs the concrete beaten out of it.

The more experienced GalactiCop is on his fifth life, old enough to actually be grey. His people came from the sunny side of a tidelocked inner planet, like Mercury with ice machines. Most bright siders never leave their tunnels, so his colleagues named him Bright Land. That happens to be the meaning of our name Lambert, so we’ll go with that.

The newer cop still has the bronze hide of his first life and has the impish humor of the young, but he’s serious about being the best police he can. He actually reads Blackstar’s Simplified Law for The Fuzz and makes notes in the margins. They’re not the kind of notes about how a GalactiCop could leverage the law and his position to get free probing from the Greys Of The Night, either. His colleagues call him Serious but that’s not a name we know, so we’ll call him Earnest.

By the time the Professor learned the importance of bringing water when hiking through the inland California desert in high summer, Bert and Ernie were landing.

If this amuses you, you might check out my collections Apocalypse Moi and Devotion and Corrosion.

“Run Your Own Mail Server” Auction for BSD Conference AV Team

A team of volunteers led by the stalwart Patrick McEvoy records the talks for EuroBSDCon, BSDCan, and AsiaBSDCon and makes them available. They rent equipment from local suppliers every year. The rental fees approach the cost of purchasing the equipment, and the team has to configure the gear from scratch and desperately hope that the previous renter didn’t break any connectors or fry any capacitors, but at least they don’t have to lug heavy gear around the world.

Video equipment now small enough that they can lug it around the world.

The BSD A/V team is raising money to purchase their own equipment. They can configure it properly at home so (in theory) they arrive, plug in, and are ready to go. They’re taking donations directly via bsdfund.org, or if you need a charitable donation receipt you could funnel it through the FreeBSD Foundation and say “community AV team” in the comments.

If they don’t raise enough money to buy the gear they need? The conference will pay to rent it. You’ll still get videos. But it’ll stress out Pat and the gang. Let’s not stress out Pat.

In wholly unrelated news, I have a spare hardcover of Run Your Own Mail Server. Not yet available in stores. Not available anywhere.

I’m auctioning it off to benefit the AV crew.

Comment on this post to bid. All bids in US dollars.

The auction runs from now until 5PM EDT 15 September. If the bidding goes nuts in the last few minutes, I’ll leave it open until it settles down. There’s no sniping this auction at the last moment, as I want the bids to escalate beyond all sensible limits.

They need the money soon, so once I acknowledge your victory I’ll ask you to donate the money within the next day and send me the receipt. I will sign this book and send it to you on the 16th, along with the ebook.

Bid early, bid often.

Thank you.