90: Demands a Little Madness

Now that Laserblasted is available in my store, here’s one final snippet from that book.

For all you might hear about stories of weapons haunted by previous owners, there aren’t that many actual legends. The modern archetype would be the Elric saga, overflowing with “I’m suddenly strong and my body’s acting weird and I keep breaking things please help me oops I’m doomed so I’m taking all of you with me.” There’s no better metaphor for a young boy transitioning to adulthood. Stormbringer isn’t exactly haunted, only ravenous for everything it can devour—again, like teenage boys. The Norse legend of Tyrfing teaches the essential lesson of “if you’re going to kidnap me and make me forge a mighty magic blade for you, you better expect that magic to fuck you up something fierce.” Master Japanese swordsmith Muramasa was rumored to pass his blood thirst and madness into his blades. Even unenchanted swords made by mentally healthy swordsmiths are viciously dangerous. Merely training with live steel, especially before antibiotics, demands a little madness.

If you buy the print from me you get the ebook free. Or you can wait a month to get either version from other, lesser bookstores.

“Networking for System Administrators” sponsorships closing and schedule.

Yesterday I finished a raw draft of the new Networking for System Administrators. It’s not ready for technical review yet; the engine has all the pieces, but there are loose bolts everywhere and a couple of the belts are repurposed nylons. I’ll get it out for tech review this weekend.

On 1 June 2025, I close sponsorships. If you want to sponsor it, this week is your last chance. I promised to do a challenge coin for print sponsors and Patronizers so I will, but the next one probably won’t. I’ll happily absorb $10 per sponsor to do something daft, but not the $25 the US’ Wheel of Tariffs threatens. (Regardless of your politics, unpredictability is death to business.)

The tentative schedule for N4SA2e is:

  • June: Technical Review
  • July-August: Copyedit
  • September: Kickstarter
  • October: ship sponsor and Patronizer copies, both print and ebook exclusive to tiltedwindmillpress.com
  • November: standard retail release

The print version will come in a special backer-exclusive edition available only to print sponsors, Patronizers, and Kickstarter backers. (Kickstarter backers can’t get the challenge coin; that’s exclusive to early backers.) I can’t say it’ll be as daft as Ruin Your Mail By Running It Yourself or the Networknomicon, but it will exist.

Then again, I always think my special editions are lame. You can make your own opinion.

89: Cheap Cat5 Cable

I’m grinding on the new Networking for System Administrators so here’s a chunk.

Ultimately, the Internet is a bunch of routers, switches, firewalls (however you define them), and other devices that connect a tangle of cables. Once a client request traverses the local WiFi connection, it travels through a bunch of wires and devices until they reach your server. Ultimately, every Internet node is connected by wire that can be traced from the local café to downtown, where it joins a bigger cable that goes across the country, perhaps joining a huge cable that runs under an ocean or three to reach another continent. That huge cable gets broken up into finer and finer wires until it finally reaches the cheap CAT5 that connects the server to the patch panel. Some parts of this link might run over satellite connections or carrier pigeons or who-knows-what. Every one of these components is fragile.

It’s a miracle the Internet works. At all.

When I finish this draft and get the book to tech review, sponsorships will close. If you want your name in the book or the challenge coin, grab it now. And when the Laserblasted print arrives in my store, I’ll be reading one last tibdit from it. If you prefer ebook you can get it now.

“Networking for System Administrators” restructuring

No, not the book this time. The product. Previously you picked a format, print or ebook. If you sponsored for print, Woocommerce used your address to calculate shipping. Cool. It took me a couple iterations to get that working, but it’s the way the rest of the world works.

Then I added print books via BookVault.

Turns out that Woocommerce does not like multiple shipping systems. It says it’s fine. It is not. After months of fighting with this, I realized that my attachment to sponsor shipping autocalculation was causing pain. I have restructured the product so that you choose a destination and pay accordingly.

The total price has not changed. The list price is now shipping-inclusive to avoid Woo’s clunky shipping system, that’s all. While sponsorship is an especially terrible deal for my Australian backers, it is no more terrible than before.

I’m still pushing to get the first draft of this book finished by the end of the month.

Also: attachment is the source of all pain. Well, that and blunt instruments. Those hurt, too.

Laserblasted Update

My copyeditor got the manuscript back to me last weekend. I’ll be getting it into production this week and next, amidst finishing the new Networking for Systems Administrators. Once the book can be purchased both print and ebook will be exclusive to my store for a month or so, then I’ll release it to the wider public.

I was hanging out with ZZ Claybourne and a couple friends, so we picked the movie we’re going to watch and review for the Kickstarter stretch goal. It is… drum roll, please…

Evil Brain from Outer Space.

I see no way this will end well.

88: The Same Bucket

The garbage truck that exploded outside my house means I am karmically bound to share a piece of the networking book.

Authoritative nameservers contain the information for one or more specific domains. I run authoritative DNS servers for my domains, such as mwl.io and prohibitionorcs.com. Anyone in the world who performs DNS queries on my domains gets their authoritative answers only from my servers.

Recursive nameservers provide DNS lookups for clients. When you browse to my web site your computer asks a recursive nameserver for the IP address to connect to. The recursive nameserver finds the authoritative nameserver for the destination site, queries it, and returns the answer to your computer.

Put your authoritative and recursive nameservers on different hosts. The twentieth-century practice of combining authoritative and recursive DNS on one server led to many security problems. In hindsight, the “store the sacrosanct Single Source of Truth for our company’s public face” function and the “collect and cache random data from any system anywhere on the Internet” function should not share the same bucket.

Remember folks, don’t throw lithium-ion batteries in the trash! But do sponsor Networking for System Administrators.

Migrating from Apache 2.4 to Caddy

I’ve been using Apache since the 1990s. The networking book requires information about QUIC, so I need experience with QUIC, so I need HTTP/3, so I can’t use Apache.

I experimented with Caddy on my test host. It worked well as a reverse proxy, so I began putting it in place in production this weekend. (If you deploy Caddy, definitely have it run as a user other than root!)

As I went through the docs to prepare, though, I realized that not only would it would be less complex and more robust to drop Apache and use Caddy, it would also be easier.

My Apache configuration files are large and complex because Apache can do anything. I don’t need a web server that can do anything. I need a web server that serves static files, talks to php-fpm, and supports TLS. The Caddy docs are complete, but I didn’t find a simple guide for what I wanted to do, so I’m posting this. I suspect that guide exists but is buried beneath pages of search engine poison.

This uses Caddy 2.9.1 on FreeBSD 14. My config files are in /usr/local/etc/caddy, symlinked to /etc/caddy.

The main config file, /etc/caddy/Caddyfile contains only:

import sites/*conf

The /etc/caddy/sites directory contains each of my sites in its own file. Mostly.

Here’s one of my old sites in blackhelicopters.org.conf:

blackhelicopters.org www.blackhelicopters.org {
        root * /var/www/bh
        file_server

        log {
                output file /var/log/bh/bh-caddy.log
                format json
        }
}

The first entry on the first line is the server’s main name, blackhelicopters.org. (I could probably let that domain go, but my oldest friends have that email in their address books and it’s worth a couple bucks a year to not inconvenience them.) The following hostnames are what Apache would call ServerAlias entries: other names this host responds to. Every name here goes into the X.509 certificate.

The root statement tells Caddy where to find the files for this site. Every URL goes under here. If I had Apache Directory statements, I could put them here.

The file_server statement means “hand out files.”

Last, there’s a logging statement. Caddy logs are written in JSON, making them harder to eyeball but easier to mechanically parse. Pipe the logs through jq(1) to read the parts you want.

Several of my domains exist only as legacy redirects. While https://michaelwlucas.com and https://michaelwarrenlucas.com made sense in a keyboard-centric era, they’re a pain to type on a phone.

blather.michaelwlucas.com www.michaelwarrenlucas.com michaelwarrenlucas.com www.michaelwlucas.com michaelwlucas.com mwlucas.org www.mwlucas.org {
        redir https://mwl.io
        }

This config doesn’t even serve files. It’s like setting DocumentRoot to /var/empty. Any traffic to these hostnames should be redirected to my current web site.

So what about that all-important main web site?

mwl.io  www.mwl.io {
        tls mwl@mwl.io

        log {
                output file /var/log/mwl/io-caddy.log
                format json
        }

        root * /var/www/io
        file_server
        php_fastcgi localhost:9000

        @disallowed {
                path /xmlrpc.php
                path *.sql
                path /wp-content/uploads/*.php
                path *~
        }

        rewrite @disallowed 'index.php'

        redir "/ks" https://www.kickstarter.com/projects/mwlucas/mwls-next-1-april-book"
...
}

the tls statement puts my email address in the Let’s Encrypt certificate request. I should probably go back and add that to the sites I did earlier.

The php_fastcgi option tells Caddy where to find the php-fpm engine.

The @disallowed statement defines a list named “disallowed.” The following rewrite statement transforms requests to files with those names, redirecting them to the index.

Finally, I have several redirect statements for my convenience.

Test a configuration by going to /etc/caddy and running caddy validate, much like apachectl configtest. The configuration files are JSON, so the parser isn’t quite as straightforward as you might expect.

# caddy validate
2025/05/05 15:02:38.489 INFO using adjacent Caddyfile
2025/05/05 15:02:38.489 INFO using config from file {"file": "Caddyfile"}
Error: adapting config using caddyfile: /usr/local/etc/caddy/sites/test-twp.conf:1: unrecognized directive: test.tiltedwindmillpress.com
Did you mean to define a second site? If so, you must use curly braces around each site to separate their configurations.

Here’s the problem: the error is not where it says the error is. The error is before the cited point. The sensible thing to do is to test after creating each site’s configuration file. If you get bored and do all your sites while watching reruns of Adam and Jamie welding JATO units to a hamster ball so they can replicate that urban legend about the Syria-Guam War, you’ll have to do a binary search of your files to see where the problem is. Test each one as you finish it.

Once you have a parseable configuration, shut off Apache and start Caddy. Watch /var/log/caddy/caddy.log for errors. Test all of your sites.

Am I happy with Caddy? Yes, so far. Am I keeping my known-working Apache configuration around? Also yes, so far. If I suffer an attack of the AI scrapers, I might need to fall back to a Caddy reverse proxy so that I can implement Anubis. Yes, there’s an Anubis Caddy module but it’s a proof-of-concept.

What kind of impact has Caddy had on my site? It seems faster, but that might be QUIC aka HTTP/3 rather than any difference between Caddy and Apache. Of course, QUIC is a difference between the two. How much of my traffic is QUIC now? QUIC runs on UDP port 443. First, let’s check how much traffic went to and from port 443 yesterday, on all protocols.

# nfdump -R . -B ip 23.139.82.3 and port 443
...
Summary: total flows: 58605, total bytes: 6.9 G, total packets: 7.0 M, avg bps: 1.0 M, avg pps: 127, avg bpp: 990
Time window: 2025-05-04 00:00:00 - 2025-05-04 23:59:59

6.9 GB. How much of that is UDP?

# nfdump -R . -B ip 23.139.82.3 and port 443 and proto udp | tail -4
Summary: total flows: 1620, total bytes: 428.4 M, total packets: 412444, avg bps: 62756, avg pps: 7, avg bpp: 1038
Time window: 2025-05-04 00:00:00 - 2025-05-04 23:59:59
Total flows processed: 750342, passed: 1620, Blocks skipped: 0, Bytes read: 66537172
Sys: 0.0209s User: 0.0209s Wall: 0.0399s flows/second: 18806544.9 Runtime: 0.0423s

428.4 Mb of my traffic is QUIC? Firefox and Chrome derivatives both use QUIC if available. The only clients that should be using TCP are stupid bots and crawlers–

Oh. Maybe I do need to implement Anubis. Dammit.

BSDCan Travel Fund Auction in honor of Mike Karels

Mike Karels has been around the BSD community since the last century, and was integral to our projects. How integral? If your name is on the definitive book on the topic, you’re integral.

On his way home from BSDCan 2024, Mike passed away.

I could go on and on about what a humble guy he was, and how he helped many folks. Or I can tell you that he backed Run Your Own Mail Server. He had no need for my book, but thought it was worthwhile? I was stunned. And appreciative.

With his family’s permission, I am auctioning off his reward in his honor. And something extra.

Here’s a copy of the backers-only edition of RYOMS, Ruin Your Mail By Running It Yourself, with a sponsors-only challenge coin. After fulfilling sponsor gifts, I have a scant handful of coins left. I don’t sell them, despite repeated requests, the occasional threat, and one ham-fisted blackmail attempt. The only way to get one today is by winning this auction.

Bid on the set by leaving a comment on this page.

The auction runs from now until 5PM EDT 12 May. If the bidding goes nuts in the last few minutes, I’ll leave it open until it settles down. There’s no sniping this auction at the last moment, as I want bids to escalate beyond all sensible limits.

Mike was a cool dude. Honor him by giving the next generation a chance to join us.

April’s Abjurational Sausage

This post went to Patronizers at the beginning of April, and to the public at the beginning of May. Not a Patronizer? You could be. It’s a terrible deal, but you could be.

I’ve considered myself well-prepared for personal financial disaster, but we’re not headed into personal financial disaster. We’re headed into a global one. I would say that I’m conservative–I save money, look after my family, mind my own business–but the reactionaries have stolen that term from me. Plus, I’m not conservative enough to cope with this economy. Nobody could be.

Last year was an all-time writing income peak, thanks to the Run Your Own Mail Server Kickstarter.

This year? Well, the Laserblasted Kickstarter is concerning. Yes, this is something of a gag book–but it’s a real novel. Folks who would normally take a $200 leather-cased book are backing for $6 ebooks. Several folks wrote me to say that they wanted to support the book, but simply couldn’t right now. I’ve also lost Patronizers.

I don’t blame them. But the upshot is, I’m expecting this year to be ugly.

Plus, this month I’m writing checks for taxes and retirement fund. Because of RYOMS, those checks are large. While I understand the logic of “feed the IRA while the stock market is low,” it definitely feels like I should just set the money on fire instead. The IRS would let me record that as a business loss.

From the latest happenings, I think it’s clear that I need to start increasing my overseas business. I need my books to become more accessible to European and Asian markets. That means dealing with VAT. Even with the RYOMS Kickstarter, I didn’t quite make enough in Europe to qualify for IOSS. I suspect that if I can include VAT in my book prices more Europeans will buy them. If I get an IOSS number I can have my books printed inside the EU, which would make them still more accessible. To get there, I need to sell more.

Chicken, meet egg.

I need to ask folks who run businesses in the EU for their thoughts. Fortunately, there’ll be a bunch at BSDCan.

Anyway. Money sucks.

The good news is, my readers seem to like ordering print books through my store! Each week there’s a handful of print sales from my bookstore. While I still have no idea what my best-selling titles are, I used a couple months of Amazon print sales as a proxy and have started working my way down that list. I’ve set up PAM Mastery, the two versions of TLS Mastery, and FreeBSD Mastery: Jails with the new printer and ordered the proofs. If they come out okay, I’ll have them on the store as print/ebook bundles straight away.

When converting books to the new printer, my titles are either simple or disastrous. I plan to focus on the simple ones first, but ideally I want to offer “The Full Michael” in print. I don’t know that anyone will actually buy that stack, but being able to offer that makes me as independent as possible.

The other good news about direct print sales? Bookvault recently added printers in Australia and Canada. Cracking Canada has long been a goal. Yes, I can ship from the States, but for two countries right next to each other, shipping to Canada sure is expensive and annoying. Printing and shipping from Canada also lets me skip the annoyance of customs.

The funny thing about this exercise is: chapbooks. When Amazon shifted their prices, I stopped putting short stories in print. I might bring short stories into print again, but make them exclusive to my store. If I was smart, I’d put them in a series like William Meikle did. Number them. Make them seem collectible. Brand them like a unit. See if I can trigger that completist urge in collectors.

Oh, wait. My hardcore collectors are mostly Patronizers. I’d just have to ship them copies. Huh. Well, I might do it anyway just for the laughs.

On the writing front: Project IDGAF, aka Laserblasted, survived first readers. People like the stupid thing. Many Patronizers will get a copy as part of their benefits, but if you’re not one of them you might take a look at the first chapter as see if it interests you. First reader reaction was universally positive, which was quite a shock. I wanted to write something that I had no stakes in, that people were not expecting or demanding, so I’d be free to stretch myself and be a little daft. (More daft. Whatever.) It seems to have worked, which is nice.

Which leaves me grinding on the new Networking for Systems Administrators. Writing a cross-platform book is weird. We pretend that TCP/IP is a universal standard, but everybody made different decisions regarding that standard. Sigh. Still, it’s starting to coalesce into something resembling a book.

Hopefully I’ll get N4SA2e to Kickstarter this summer, along with the Twisted Presents Christmas story collection. Most of the stories exist for that one. I still need to write an exclusive Prohibition Orcs tale for it, but I’m hoping that doesn’t go horribly wrong. Last thing I need to do is accidentally write a Christmas novel about a cranky little old lady orc (no matter how much JG would like that).

Anyway, I better get back to these edits. Thank you for your support. I appreciate every one of you, especially in these troubled times.

Notes on caddy as QUIC reverse proxy with mac_portacl

As I wrote yesterday, I need QUIC for my web sites. The servers I have data on run FreeBSD, because ZFS. I use Apache everywhere, because it’s what I learned back in the 486 Age. My web site is critical to my business, so I must minimize downtime. I chose to implement a Caddy reverse proxy, because it looked easier than Envoy or migrating to nginx. (Nothing against either tool, of course.)

These are my notes, not a tutorial. If they help you, that’s grand. I pillaged Thomas Hurt’s post for this.

QUIC for HTTPS runs on UDP port 443. I suggest you start by opening UDP port 443 on your packet filter. Or you can follow my example, not open it, and spend half an hour staring at the screen shrieking why doesn’t this work? Up to you.

Caddy defaults to running as root, so it can bind to privileged ports. I played “run servers as root” in 1995 and have no desire to get rooted again, so I need to allow an unprivileged user to bind to privileged ports. That’s where FreeBSD’s mac_portacl comes in. It allows unprivileged users to bind to privileged ports according to a policy you set.

I’ve written about mac_portacl before, but my hosting architecture has changed. Instead of VMs scattered around the world, I now rent a single dedicated machine and use VNET jails. It saved me some money and gave me flexibility.

But mac_portacl is not jail-aware. You set rules per UID, but those rules apply across all jails. Individual jails can declare if they use mac_portacl or if they use the traditional scheme. You need to use consistent UIDs across all your jails, meaning that the user www must run whatever’s on port 80 and 443. All services need to run as different unprivileged users, so I’ll need to create a separate user for Apache.

Start on the host.

# pkg install portacl-rc

This gives /etc/rc.conf integration into mac_portacl. So go into /etc/rc.conf.

portacl_users="www"
portacl_user_www_tcp="http https"
portacl_user_www_udp="https"
portacl_enable=yes

Reboot the system and verify that portacl is working.

# sysctl security.mac.portacl
security.mac.portacl.rules: uid:80:tcp:443,uid:80:tcp:80,uid:80:udp:443,uid:80:udp:80
security.mac.portacl.port_high: 1023
security.mac.portacl.autoport_exempt: 1
security.mac.portacl.suser_exempt: 1
security.mac.portacl.enabled: 1

Looks good. Now go to the jail.

Control traditional privileged ports with the net.inet.ip.portrange.reservedhigh sysctl. By setting it to 0, you disable privileged ports.

# sysctl net.inet.ip.portrange.reservedhigh=0
net.inet.ip.portrange.reservedhigh: 1023 -> 0

Make the change permanent in sysctl.conf
net.inet.ip.portrange.reservedhigh=0

mac_portacl now controls access to ports 1 through 1023.

I need a separate user for Apache. Yes, I could run both as www but I survived the “run everything as nobody” era and learned my lesson.

# pw groupadd -n apache -g 81
# pw useradd -n apache -u 81 -g 81 -d /nonexistent -w no -s /usr/sbin/nologin

In theory, I can switch apache to run as this user and it’ll be fine. Reality will have a short sharp shock for me, I’m sure.

Now go to httpd.conf. Bind it to 127.0.0.1 port 8080

Listen 127.0.0.1:8080
User apache
Group apache

I also commented out mod_ssl. Apache will provide everything unencrypted, but only on localhost.

Go into the virtual host config. All those VirtualHost *:443 entries? They need changing to VirtualHost 127.0.0.1:8080

I also comment out all of the TLS entries. We’ll have Caddy manage TLS for us.

Restart Apache. Watch the error logs. It’ll gripe about a few files being owned by www. Change their owner.

Now configure caddy in /usr/local/etc/Caddyfile.

Permit me to put on my old sysadmin hat and shriek: “Don’t start config files with capital letters, people! You know better! WHYYY.” Yes, I know the world has moved on. Come closer so I can smack you with my cane.

Caddy does nothing except get X.509 certificates and forward connections to Apache.

test.mwl.io {
  reverse_proxy localhost:8080

  # Enable logging:
  log {
    output file /var/log/caddy/access.log
    format json
  }
}

That’s it.

Start Caddy. Point the browser at the HTTP site and it gets redirected to the HTTPS site. All this work, and I have replicated what I started with!

So let’s turn on QUIC and HTTP/3.

Sites must inform clients that QUIC is available through an HTTP header. The client makes an initial connection of HTTP/2, sees the header, and switches to HTTP/3 and QUIC. Add the header in the virtual host configuration.

Header set alt-svc "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600

Reload Apache. Set up your packet sniffer to watch UDP port 443. Point your browser at the web page.

So far, I like Caddy. It seems simpler than Apache. It is owned and backed by a commercial firm (ZeroSSL). I am careful going all-in on commercially-backed tools because the Internet’s business model is betrayal. I have other options if that happens.

I’ll deploy this on my main site to get some QUIC experience for the new Networking for System Administrators. QUIC isn’t essential today, but I want to future-proof it. I would be remiss if I didn’t mention that the book is open for sponsorships, for a little while longer at least.

Future path: do I need Apache? For some stuff, probably. But can I serve simple sites straight out of Caddy? Yes.