91: Vice Without a Biological Limit

Networking for Systems Administrators is out for tech review, so I’m working on my forthcoming Christmas collection. Here’s a snippet from Twisted Presents.

Money is the one vice without a biological limit. We can gorge on food and wine until we puke, and the feast ends. Do enough drugs and you die. Despite what teenagers think, you can’t have sex twenty-four seven. Vices are self-correcting.

But if you’re greedy enough, you can pile up money forever.

And you can teach your children that your vice is their birthright.

The only reason I don’t say “eat the rich” is because toxic metals accumulate at the top of the food chain.

But even among the rich, there’s some I hate more than others. Until today, “people who scam charities” weren’t even on the list. I mean, the biggest charities in the world spend eighty percent of their donations on administration. They’re just another face for the oligarchy.
But a meager fraction of charities do charitable work.

There are wholesome people who honestly want to help others, and invest their time and effort and meager income towards that.

Some of those handle large amounts of money, and spend that money on their cause.

But every dollar spent helping the helpless isn’t invested in cybersecurity.

If you have the skills, and totally lack decency, these charities are easy targets.

Twisted Presents will be coming to Kickstarter in July, so I can get it to you well before Christmas. For twelve days, of course.

Buy Your Paperbacks Directly From Me

All Tilted Windmill Press titles are now available directly from me in paperback and ebook at https://tiltedwindmillpress.com. All paperback purchases include the ebook. You’ll get the ebook immediately1, and the print will arrive in a week or so.

Books will be printed in the US, Canada, UK, and Australia. This reduces both shipping costs and environmental impact. Books aren’t exactly green, but local printing makes them less brown. (Are ebooks greener? That’s a great argument over a drink.)

I am excited beyond words. I have been working towards this ever since my first book came out in 1992.

Benefits to you? Those bundles I offer, like the FreeBSD Storage Mastery bundle? There’s now a discount print version. That ridiculous The Full Michael bundle that includes everything I’ve indie published? You can now buy the whole thing in paperback.

Do I expect anyone to drop $624 on a stack of books? No. But I am delighted to have that degree of control.

Books from No Starch Press (Absolute FreeBSD, Absolute OpenBSD, and Network Flow Analysis) are not included. Sorry. I don’t have the access to ship those touch-free on demand. The ILUVMICHAEL coupon code still gets you 30% off at their site and gives me a couple bucks extra, though!

Completing this was a huge amount of work, but the publishing industry is doing its best to eat writers alive. The only way to survive is disintermediation.

I haven’t made hardcovers available yet. Hardcover sales are minuscule next to paperbacks. Some books present challenges, and I’m not sure selling them direct is worth it. I’m doing the easy hardcovers first in the hope that inspiration strikes.

Future books will be released in on my site a month before they’re available at retailers. If they’re trying to eat my career, I see no reason to prioritize them.

90: Demands a Little Madness

Now that Laserblasted is available in my store, here’s one final snippet from that book.

For all you might hear about stories of weapons haunted by previous owners, there aren’t that many actual legends. The modern archetype would be the Elric saga, overflowing with “I’m suddenly strong and my body’s acting weird and I keep breaking things please help me oops I’m doomed so I’m taking all of you with me.” There’s no better metaphor for a young boy transitioning to adulthood. Stormbringer isn’t exactly haunted, only ravenous for everything it can devour—again, like teenage boys. The Norse legend of Tyrfing teaches the essential lesson of “if you’re going to kidnap me and make me forge a mighty magic blade for you, you better expect that magic to fuck you up something fierce.” Master Japanese swordsmith Muramasa was rumored to pass his blood thirst and madness into his blades. Even unenchanted swords made by mentally healthy swordsmiths are viciously dangerous. Merely training with live steel, especially before antibiotics, demands a little madness.

If you buy the print from me you get the ebook free. Or you can wait a month to get either version from other, lesser bookstores.

“Networking for System Administrators” sponsorships closing and schedule.

Yesterday I finished a raw draft of the new Networking for System Administrators. It’s not ready for technical review yet; the engine has all the pieces, but there are loose bolts everywhere and a couple of the belts are repurposed nylons. I’ll get it out for tech review this weekend.

On 1 June 2025, I close sponsorships. If you want to sponsor it, this week is your last chance. I promised to do a challenge coin for print sponsors and Patronizers so I will, but the next one probably won’t. I’ll happily absorb $10 per sponsor to do something daft, but not the $25 the US’ Wheel of Tariffs threatens. (Regardless of your politics, unpredictability is death to business.)

The tentative schedule for N4SA2e is:

  • June: Technical Review
  • July-August: Copyedit
  • September: Kickstarter
  • October: ship sponsor and Patronizer copies, both print and ebook exclusive to tiltedwindmillpress.com
  • November: standard retail release

The print version will come in a special backer-exclusive edition available only to print sponsors, Patronizers, and Kickstarter backers. (Kickstarter backers can’t get the challenge coin; that’s exclusive to early backers.) I can’t say it’ll be as daft as Ruin Your Mail By Running It Yourself or the Networknomicon, but it will exist.

Then again, I always think my special editions are lame. You can make your own opinion.

89: Cheap Cat5 Cable

I’m grinding on the new Networking for System Administrators so here’s a chunk.

Ultimately, the Internet is a bunch of routers, switches, firewalls (however you define them), and other devices that connect a tangle of cables. Once a client request traverses the local WiFi connection, it travels through a bunch of wires and devices until they reach your server. Ultimately, every Internet node is connected by wire that can be traced from the local café to downtown, where it joins a bigger cable that goes across the country, perhaps joining a huge cable that runs under an ocean or three to reach another continent. That huge cable gets broken up into finer and finer wires until it finally reaches the cheap CAT5 that connects the server to the patch panel. Some parts of this link might run over satellite connections or carrier pigeons or who-knows-what. Every one of these components is fragile.

It’s a miracle the Internet works. At all.

When I finish this draft and get the book to tech review, sponsorships will close. If you want your name in the book or the challenge coin, grab it now. And when the Laserblasted print arrives in my store, I’ll be reading one last tibdit from it. If you prefer ebook you can get it now.

“Networking for System Administrators” restructuring

No, not the book this time. The product. Previously you picked a format, print or ebook. If you sponsored for print, Woocommerce used your address to calculate shipping. Cool. It took me a couple iterations to get that working, but it’s the way the rest of the world works.

Then I added print books via BookVault.

Turns out that Woocommerce does not like multiple shipping systems. It says it’s fine. It is not. After months of fighting with this, I realized that my attachment to sponsor shipping autocalculation was causing pain. I have restructured the product so that you choose a destination and pay accordingly.

The total price has not changed. The list price is now shipping-inclusive to avoid Woo’s clunky shipping system, that’s all. While sponsorship is an especially terrible deal for my Australian backers, it is no more terrible than before.

I’m still pushing to get the first draft of this book finished by the end of the month.

Also: attachment is the source of all pain. Well, that and blunt instruments. Those hurt, too.

Laserblasted Update

My copyeditor got the manuscript back to me last weekend. I’ll be getting it into production this week and next, amidst finishing the new Networking for Systems Administrators. Once the book can be purchased both print and ebook will be exclusive to my store for a month or so, then I’ll release it to the wider public.

I was hanging out with ZZ Claybourne and a couple friends, so we picked the movie we’re going to watch and review for the Kickstarter stretch goal. It is… drum roll, please…

Evil Brain from Outer Space.

I see no way this will end well.

88: The Same Bucket

The garbage truck that exploded outside my house means I am karmically bound to share a piece of the networking book.

Authoritative nameservers contain the information for one or more specific domains. I run authoritative DNS servers for my domains, such as mwl.io and prohibitionorcs.com. Anyone in the world who performs DNS queries on my domains gets their authoritative answers only from my servers.

Recursive nameservers provide DNS lookups for clients. When you browse to my web site your computer asks a recursive nameserver for the IP address to connect to. The recursive nameserver finds the authoritative nameserver for the destination site, queries it, and returns the answer to your computer.

Put your authoritative and recursive nameservers on different hosts. The twentieth-century practice of combining authoritative and recursive DNS on one server led to many security problems. In hindsight, the “store the sacrosanct Single Source of Truth for our company’s public face” function and the “collect and cache random data from any system anywhere on the Internet” function should not share the same bucket.

Remember folks, don’t throw lithium-ion batteries in the trash! But do sponsor Networking for System Administrators.

Migrating from Apache 2.4 to Caddy

I’ve been using Apache since the 1990s. The networking book requires information about QUIC, so I need experience with QUIC, so I need HTTP/3, so I can’t use Apache.

I experimented with Caddy on my test host. It worked well as a reverse proxy, so I began putting it in place in production this weekend. (If you deploy Caddy, definitely have it run as a user other than root!)

As I went through the docs to prepare, though, I realized that not only would it would be less complex and more robust to drop Apache and use Caddy, it would also be easier.

My Apache configuration files are large and complex because Apache can do anything. I don’t need a web server that can do anything. I need a web server that serves static files, talks to php-fpm, and supports TLS. The Caddy docs are complete, but I didn’t find a simple guide for what I wanted to do, so I’m posting this. I suspect that guide exists but is buried beneath pages of search engine poison.

This uses Caddy 2.9.1 on FreeBSD 14. My config files are in /usr/local/etc/caddy, symlinked to /etc/caddy.

The main config file, /etc/caddy/Caddyfile contains only:

import sites/*conf

The /etc/caddy/sites directory contains each of my sites in its own file. Mostly.

Here’s one of my old sites in blackhelicopters.org.conf:

blackhelicopters.org www.blackhelicopters.org {
        root * /var/www/bh
        file_server

        log {
                output file /var/log/bh/bh-caddy.log
                format json
        }
}

The first entry on the first line is the server’s main name, blackhelicopters.org. (I could probably let that domain go, but my oldest friends have that email in their address books and it’s worth a couple bucks a year to not inconvenience them.) The following hostnames are what Apache would call ServerAlias entries: other names this host responds to. Every name here goes into the X.509 certificate.

The root statement tells Caddy where to find the files for this site. Every URL goes under here. If I had Apache Directory statements, I could put them here.

The file_server statement means “hand out files.”

Last, there’s a logging statement. Caddy logs are written in JSON, making them harder to eyeball but easier to mechanically parse. Pipe the logs through jq(1) to read the parts you want.

Several of my domains exist only as legacy redirects. While https://michaelwlucas.com and https://michaelwarrenlucas.com made sense in a keyboard-centric era, they’re a pain to type on a phone.

blather.michaelwlucas.com www.michaelwarrenlucas.com michaelwarrenlucas.com www.michaelwlucas.com michaelwlucas.com mwlucas.org www.mwlucas.org {
        redir https://mwl.io
        }

This config doesn’t even serve files. It’s like setting DocumentRoot to /var/empty. Any traffic to these hostnames should be redirected to my current web site.

So what about that all-important main web site?

mwl.io  www.mwl.io {
        tls mwl@mwl.io

        log {
                output file /var/log/mwl/io-caddy.log
                format json
        }

        root * /var/www/io
        file_server
        php_fastcgi localhost:9000

        @disallowed {
                path /xmlrpc.php
                path *.sql
                path /wp-content/uploads/*.php
                path *~
        }

        rewrite @disallowed 'index.php'

        redir "/ks" https://www.kickstarter.com/projects/mwlucas/mwls-next-1-april-book"
...
}

the tls statement puts my email address in the Let’s Encrypt certificate request. I should probably go back and add that to the sites I did earlier.

The php_fastcgi option tells Caddy where to find the php-fpm engine.

The @disallowed statement defines a list named “disallowed.” The following rewrite statement transforms requests to files with those names, redirecting them to the index.

Finally, I have several redirect statements for my convenience.

Test a configuration by going to /etc/caddy and running caddy validate, much like apachectl configtest. The configuration files are JSON, so the parser isn’t quite as straightforward as you might expect.

# caddy validate
2025/05/05 15:02:38.489 INFO using adjacent Caddyfile
2025/05/05 15:02:38.489 INFO using config from file {"file": "Caddyfile"}
Error: adapting config using caddyfile: /usr/local/etc/caddy/sites/test-twp.conf:1: unrecognized directive: test.tiltedwindmillpress.com
Did you mean to define a second site? If so, you must use curly braces around each site to separate their configurations.

Here’s the problem: the error is not where it says the error is. The error is before the cited point. The sensible thing to do is to test after creating each site’s configuration file. If you get bored and do all your sites while watching reruns of Adam and Jamie welding JATO units to a hamster ball so they can replicate that urban legend about the Syria-Guam War, you’ll have to do a binary search of your files to see where the problem is. Test each one as you finish it.

Once you have a parseable configuration, shut off Apache and start Caddy. Watch /var/log/caddy/caddy.log for errors. Test all of your sites.

Am I happy with Caddy? Yes, so far. Am I keeping my known-working Apache configuration around? Also yes, so far. If I suffer an attack of the AI scrapers, I might need to fall back to a Caddy reverse proxy so that I can implement Anubis. Yes, there’s an Anubis Caddy module but it’s a proof-of-concept.

What kind of impact has Caddy had on my site? It seems faster, but that might be QUIC aka HTTP/3 rather than any difference between Caddy and Apache. Of course, QUIC is a difference between the two. How much of my traffic is QUIC now? QUIC runs on UDP port 443. First, let’s check how much traffic went to and from port 443 yesterday, on all protocols.

# nfdump -R . -B ip 23.139.82.3 and port 443
...
Summary: total flows: 58605, total bytes: 6.9 G, total packets: 7.0 M, avg bps: 1.0 M, avg pps: 127, avg bpp: 990
Time window: 2025-05-04 00:00:00 - 2025-05-04 23:59:59

6.9 GB. How much of that is UDP?

# nfdump -R . -B ip 23.139.82.3 and port 443 and proto udp | tail -4
Summary: total flows: 1620, total bytes: 428.4 M, total packets: 412444, avg bps: 62756, avg pps: 7, avg bpp: 1038
Time window: 2025-05-04 00:00:00 - 2025-05-04 23:59:59
Total flows processed: 750342, passed: 1620, Blocks skipped: 0, Bytes read: 66537172
Sys: 0.0209s User: 0.0209s Wall: 0.0399s flows/second: 18806544.9 Runtime: 0.0423s

428.4 Mb of my traffic is QUIC? Firefox and Chrome derivatives both use QUIC if available. The only clients that should be using TCP are stupid bots and crawlers–

Oh. Maybe I do need to implement Anubis. Dammit.

BSDCan Travel Fund Auction in honor of Mike Karels

Mike Karels has been around the BSD community since the last century, and was integral to our projects. How integral? If your name is on the definitive book on the topic, you’re integral.

On his way home from BSDCan 2024, Mike passed away.

I could go on and on about what a humble guy he was, and how he helped many folks. Or I can tell you that he backed Run Your Own Mail Server. He had no need for my book, but thought it was worthwhile? I was stunned. And appreciative.

With his family’s permission, I am auctioning off his reward in his honor. And something extra.

Here’s a copy of the backers-only edition of RYOMS, Ruin Your Mail By Running It Yourself, with a sponsors-only challenge coin. After fulfilling sponsor gifts, I have a scant handful of coins left. I don’t sell them, despite repeated requests, the occasional threat, and one ham-fisted blackmail attempt. The only way to get one today is by winning this auction.

Bid on the set by leaving a comment on this page.

The auction runs from now until 5PM EDT 12 May. If the bidding goes nuts in the last few minutes, I’ll leave it open until it settles down. There’s no sniping this auction at the last moment, as I want bids to escalate beyond all sensible limits.

Mike was a cool dude. Honor him by giving the next generation a chance to join us.