Category: bsd

As I wrote yesterday, I need QUIC for my web sites. The servers I have data on run FreeBSD, because ZFS. I use Apache everywhere, because it’s what I learned back in the 486 Age. My web site is critical to my business, so I must minimize downtime. I chose to implement a Caddy reverse proxy, because it looked easier than Envoy or migrating to nginx. (Nothing against either tool, of course.)

These are my notes, not a tutorial. If they help you, that’s grand. I pillaged Thomas Hurt’s post for this.

QUIC for HTTPS runs on UDP port 443. I suggest you start by opening UDP port 443 on your packet filter. Or you can follow my example, not open it, and spend half an hour staring at the screen shrieking why doesn’t this work? Up to you.

Caddy defaults to running as root, so it can bind to privileged ports. I played “run servers as root” in 1995 and have no desire to get rooted again, so I need to allow an unprivileged user to bind to privileged ports. That’s where FreeBSD’s mac_portacl comes in. It allows unprivileged users to bind to privileged ports according to a policy you set.

I’ve written about mac_portacl before, but my hosting architecture has changed. Instead of VMs scattered around the world, I now rent a single dedicated machine and use VNET jails. It saved me some money and gave me flexibility.

But mac_portacl is not jail-aware. You set rules per UID, but those rules apply across all jails. Individual jails can declare if they use mac_portacl or if they use the traditional scheme. You need to use consistent UIDs across all your jails, meaning that the user www must run whatever’s on port 80 and 443. All services need to run as different unprivileged users, so I’ll need to create a separate user for Apache.

Start on the host.

# pkg install portacl-rc

This gives /etc/rc.conf integration into mac_portacl. So go into /etc/rc.conf.

portacl_users="www"
portacl_user_www_tcp="http https"
portacl_user_www_udp="https"
portacl_enable=yes

Reboot the system and verify that portacl is working.

# sysctl security.mac.portacl
security.mac.portacl.rules: uid:80:tcp:443,uid:80:tcp:80,uid:80:udp:443,uid:80:udp:80
security.mac.portacl.port_high: 1023
security.mac.portacl.autoport_exempt: 1
security.mac.portacl.suser_exempt: 1
security.mac.portacl.enabled: 1

Looks good. Now go to the jail.

Control traditional privileged ports with the net.inet.ip.portrange.reservedhigh sysctl. By setting it to 0, you disable privileged ports.

# sysctl net.inet.ip.portrange.reservedhigh=0
net.inet.ip.portrange.reservedhigh: 1023 -> 0

Make the change permanent in sysctl.conf
net.inet.ip.portrange.reservedhigh=0

mac_portacl now controls access to ports 1 through 1023.

I need a separate user for Apache. Yes, I could run both as www but I survived the “run everything as nobody” era and learned my lesson.

# pw groupadd -n apache -g 81
# pw useradd -n apache -u 81 -g 81 -d /nonexistent -w no -s /usr/sbin/nologin

In theory, I can switch apache to run as this user and it’ll be fine. Reality will have a short sharp shock for me, I’m sure.

Now go to httpd.conf. Bind it to 127.0.0.1 port 8080

Listen 127.0.0.1:8080
User apache
Group apache

I also commented out mod_ssl. Apache will provide everything unencrypted, but only on localhost.

Go into the virtual host config. All those VirtualHost *:443 entries? They need changing to VirtualHost 127.0.0.1:8080

I also comment out all of the TLS entries. We’ll have Caddy manage TLS for us.

Restart Apache. Watch the error logs. It’ll gripe about a few files being owned by www. Change their owner.

Now configure caddy in /usr/local/etc/Caddyfile.

Permit me to put on my old sysadmin hat and shriek: “Don’t start config files with capital letters, people! You know better! WHYYY.” Yes, I know the world has moved on. Come closer so I can smack you with my cane.

Caddy does nothing except get X.509 certificates and forward connections to Apache.

test.mwl.io {
  reverse_proxy localhost:8080

  # Enable logging:
  log {
    output file /var/log/caddy/access.log
    format json
  }
}

That’s it.

Start Caddy. Point the browser at the HTTP site and it gets redirected to the HTTPS site. All this work, and I have replicated what I started with!

So let’s turn on QUIC and HTTP/3.

Sites must inform clients that QUIC is available through an HTTP header. The client makes an initial connection of HTTP/2, sees the header, and switches to HTTP/3 and QUIC. Add the header in the virtual host configuration.

Header set alt-svc "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600

Reload Apache. Set up your packet sniffer to watch UDP port 443. Point your browser at the web page.

So far, I like Caddy. It seems simpler than Apache. It is owned and backed by a commercial firm (ZeroSSL). I am careful going all-in on commercially-backed tools because the Internet’s business model is betrayal. I have other options if that happens.

I’ll deploy this on my main site to get some QUIC experience for the new Networking for System Administrators. QUIC isn’t essential today, but I want to future-proof it. I would be remiss if I didn’t mention that the book is open for sponsorships, for a little while longer at least.

Future path: do I need Apache? For some stuff, probably. But can I serve simple sites straight out of Caddy? Yes.

I encountered yet another discussion about OpenBSD PF versus FreeBSD PF. For those who are new to the discussion: OpenBSD developers created PF in 2001, and it rapidly improved to become the most approachable open source packet filter. FreeBSD ported PF over to its kernel in 2004, with occasional updates since. Today a whole bunch of folks who don’t program echo cultish wisdom that one or the other version of PF has fallen behind, not kept up on improvements, or otherwise betrayed their community. My subtler comments have been misinterpreted, so let’s try this.

These claims are garbage.

First, and most importantly: FreeBSD PF developers work with OpenBSD devs all the time, and OpenBSD PF developers pull stuff from FreeBSD¹. You get a lot of noise about certain people being jerks about the other project–and both projects absolutely have jerks. (And yes, anyone who has read my books knows that I am a cross-platform jerk.) But for the most part, folks want to work together.

PF is absolutely an OpenBSD creation, though, so why isn’t the OpenBSD version the Single Source of Truth? Why doesn’t FreeBSD just consider OpenBSD a vendor and pull that code in? Because the OpenBSD and FreeBSD kernels are wholly different.

Back when I wrote Absolute BSD, I could realistically write a single book that would basically apply to the three major open source BSDs. Yes, the various projects objected to being lumped together, but if you knew any one of them you could stumble through the others. This is no longer true. FreeBSD’s kernel uses a wholly different locking model than OpenBSD. OpenBSD’s memory protections have no equivalent in FreeBSD. These are not things you can manage with a shim layer or kernel ABI. These are big, complicated, intrusive differences. You can’t tar up one version and dump it in the other’s kernel. It won’t work. If you do a hack job of making it work, it will perform badly.

Yes, you can find “proof” that one PF or the other is faster under particular workloads on specific hardware. I have no doubt that some of them are not only accurate, but honest. There are other workloads, though, and other hardware, and other conditions. Regardless of who wins a particular race, the constant competition to achieve peak performance benefits everyone. I’m not going to link to any of the benchmarks, because I have made my opinions on benchmarking very clear elsewhere.² Pick what you want and roll with it.

Every PF developer is trundling along, doing their best to make things work.

Are features missing from one or the other? Yep. I’m not going to list examples because, as the above links show, each project plucks what they find useful from the other. These things are freely given, with glad hearts, but they take time to integrate. Filling message boards with staunch declarations that my team’s PF is better is not only tedious, it wholly misses the point.

People are working together to improve the world.

And the PF syntax is the most approachable in all of open source Unix.

(Partisan fanboy comments will be mercilessly whacked.)

The new issue of the FreeBSD Journal is out, including my We Get Letters column. Before you click on the browser-based version, know there’s a PDF.

The issue celebrates 30 years of FreeBSD, and has all sorts of articles about the project’s history. By sheer chance, this is my 30th column. I have done my best, but they still won’t fire me. To mark the occasion, I elected to tell the truth.

I’m posting this here because I’m posting it everywhere.

I’ve just sent an email everyone who volunteered to help make BSDCan 2024 happen. I suspect some of you have not received that email. If you haven’t seen it, please check your spam folder. We need to start organizing now to make 2024 go smoothly. Mostly smoothly.

If you’re on the operations team and didn’t get the email, please poke me directly so I can update your email address.

I’ll be doing my talk about OpenBSD filesystems tomorrow night, for mug.org‘s online meeting.

I expect this will be the last time I give this talk, but MUG does a decent job of recording so you can catch it later on YouTube or wherever. But if you show up in person, you can ask inconvenient questions like “when are you going to learn to write?” or “have you considered truck driving school?”

Dan Langille has spent a good part of the last twenty years on BSDCan. We’ve had other BSD conferences in the Western Hemisphere, but BSDCan is the most consistent. Covid interrupted it, but only because Dan coordinated with EuroBSDCon to have a single online conference in 2021.

This is a lot of work. Dan’s life has changed.

Dan is stepping back from organizing BSDCan. I am taking over coordinating 2024.

Note I did not say “running.” Running an international conference is a job best accomplished by a team. A large team. Dan set up BSDCan 2023 with himself and Adam Thompson, and ran it with assistance from Dru Lavigne and Warren Block in registration, and Patrick McEvoy and Andrew Fengler in streaming. I am not nearly that tough.

Instead, we have assembled a team of seventeen people to be the BSDCan 2024 Operations Team. Dan will become our source of knowledge, telling us who to talk to at the University of Ottawa and where to reliably get T-shirts locally and which caterers are most likely to cause indigestion. I am pleased that Adam, Dru, Warren, Patrick, and Andrew all cheerfully agreed to continue in their roles. (Adam’s work of coordinating travel and accommodations for the speakers will be split among a few folks, led by Adam.)

My entire job will be to coordinate the team, help them gather resources, and mediate conflicts between them. Every person on this list is motivated to make BSDCan 2024 happen, but even the best-intentioned folks can disagree. If required I’ll make final decisions, sure, but if my decision makes people unhappy, I have no doubt that the esteemed program committee will tell me I’m being an idiot. They have final say over the con, after all.

This means I need to be staunch on not doing anything myself–although I admit, I might make a call to the Diefenbunker to see how much it would cost to host the closing social event there, and to Pili Pili Chicken for a price quote to cater it. This is mostly for my peace of mind, however. Confirming it’s too expensive will put the idea to rest.

Organizing BSDCan with two folks is a monumental achievement. I have seventeen, which I consider barely adequate for a Redundant Array of Independent Dans. We’ll need folks to handle a variety of smaller tasks, from checking video times to hauling boxes. (If you haven’t hauled a box for the con staff, have you really been to BSDCan?) I intend to make use of the bsdcan-volunteers mailing list to gather those folks. Folks on our team can ask for specific help on that list, whether it be figuring out a balky database or showing up at 7AM opening day with a roll of duct tape and a cattle prod. (“Cattle prod? Really?” Hey, I don’t know the details. Knowing the details would get in the way of me doing my job. I trust the various team members to know what they need and to ask for it.)

One thing that the BSD community has historically excelled at is passing leadership from one generation to the next. BSDCan’s operations team will follow that example. We have old hands taking the lead on parts of BSDCan, but we have at least two people covering each responsibility and at least one of them should be comparatively young.

I don’t intend to coordinate BSDCan for more than a couple years. My goal is to set up a self-perpetuating structure, make sure it runs, and walk away. Normally, I wouldn’t take on anything like this, but BSDCan is important to my people and it deserves my time and attention.

The BSDCan 2024 goal is to largely reimplement BSDCan 2023, but supported by different people. Yes, there’s room to change things. Yes, I have ideas. Many people have ideas. You probably have ideas. Our overwhelming goal is to make the conference happen. Perhaps that’s unambitious but extracting knowledge from Dan’s head, documenting it, and reimplementing it will take time and energy. I don’t want to burn anyone out. I intend to retain the location, the mask policy, the papers committee, the social event, everything, until we have BSDCan down cold.

Dan has done well. He’s earned a break.

We are drafting him to run the auction, however.

I’ll be talking about OpenBSD filesystems at semibug.org‘s online meeting, Tuesday night at 7PM. The jitsi link will be posted to the semibug mailing list late tomorrow.

It’s based on OpenBSD Mastery: Filesystems, of course.

I’ll be teaching a four-hour tutorial on OpenBSD storage management at BSDCan 2023. As you might imagine, it’s based on OpenBSD Mastery: Filesystems.

I am pleased to see BSDCan returning to meatspace. Yes, the pandemic is ongoing, and I don’t blame folks who decide not to attend. The main reason I chose to attend is that the concom (which I’m a member of) has chosen to enforce a stringent mask policy. Yes, I know you have the right to not wear a mask in public. BSDCan is a private event for a community, however, and communities have a responsibility to protect their weakest members. (People who think that your rights outweigh your responsibilities, I will delete your comments.)

I hope to see many of you at the con, if not my tutorial. It will be good to see many old friends. Well, at least their eyes. Faces in one of the many fine outdoor dining establishments Byward Market offers.