Notes on caddy as QUIC reverse proxy with mac_portacl

As I wrote yesterday, I need QUIC for my web sites. The servers I have data on run FreeBSD, because ZFS. I use Apache everywhere, because it’s what I learned back in the 486 Age. My web site is critical to my business, so I must minimize downtime. I chose to implement a Caddy reverse proxy, because it looked easier than Envoy or migrating to nginx. (Nothing against either tool, of course.)

These are my notes, not a tutorial. If they help you, that’s grand. I pillaged Thomas Hurt’s post for this.

QUIC for HTTPS runs on UDP port 443. I suggest you start by opening UDP port 443 on your packet filter. Or you can follow my example, not open it, and spend half an hour staring at the screen shrieking why doesn’t this work? Up to you.

Caddy defaults to running as root, so it can bind to privileged ports. I played “run servers as root” in 1995 and have no desire to get rooted again, so I need to allow an unprivileged user to bind to privileged ports. That’s where FreeBSD’s mac_portacl comes in. It allows unprivileged users to bind to privileged ports according to a policy you set.

I’ve written about mac_portacl before, but my hosting architecture has changed. Instead of VMs scattered around the world, I now rent a single dedicated machine and use VNET jails. It saved me some money and gave me flexibility.

But mac_portacl is not jail-aware. You set rules per UID, but those rules apply across all jails. Individual jails can declare if they use mac_portacl or if they use the traditional scheme. You need to use consistent UIDs across all your jails, meaning that the user www must run whatever’s on port 80 and 443. All services need to run as different unprivileged users, so I’ll need to create a separate user for Apache.

Start on the host.

# pkg install portacl-rc

This gives /etc/rc.conf integration into mac_portacl. So go into /etc/rc.conf.

portacl_users="www"
portacl_user_www_tcp="http https"
portacl_user_www_udp="https"
portacl_enable=yes

Reboot the system and verify that portacl is working.

# sysctl security.mac.portacl
security.mac.portacl.rules: uid:80:tcp:443,uid:80:tcp:80,uid:80:udp:443,uid:80:udp:80
security.mac.portacl.port_high: 1023
security.mac.portacl.autoport_exempt: 1
security.mac.portacl.suser_exempt: 1
security.mac.portacl.enabled: 1

Looks good. Now go to the jail.

Control traditional privileged ports with the net.inet.ip.portrange.reservedhigh sysctl. By setting it to 0, you disable privileged ports.

# sysctl net.inet.ip.portrange.reservedhigh=0
net.inet.ip.portrange.reservedhigh: 1023 -> 0

Make the change permanent in sysctl.conf
net.inet.ip.portrange.reservedhigh=0

mac_portacl now controls access to ports 1 through 1023.

I need a separate user for Apache. Yes, I could run both as www but I survived the “run everything as nobody” era and learned my lesson.

# pw groupadd -n apache -g 81
# pw useradd -n apache -u 81 -g 81 -d /nonexistent -w no -s /usr/sbin/nologin

In theory, I can switch apache to run as this user and it’ll be fine. Reality will have a short sharp shock for me, I’m sure.

Now go to httpd.conf. Bind it to 127.0.0.1 port 8080

Listen 127.0.0.1:8080
User apache
Group apache

I also commented out mod_ssl. Apache will provide everything unencrypted, but only on localhost.

Go into the virtual host config. All those VirtualHost *:443 entries? They need changing to VirtualHost 127.0.0.1:8080

I also comment out all of the TLS entries. We’ll have Caddy manage TLS for us.

Restart Apache. Watch the error logs. It’ll gripe about a few files being owned by www. Change their owner.

Now configure caddy in /usr/local/etc/Caddyfile.

Permit me to put on my old sysadmin hat and shriek: “Don’t start config files with capital letters, people! You know better! WHYYY.” Yes, I know the world has moved on. Come closer so I can smack you with my cane.

Caddy does nothing except get X.509 certificates and forward connections to Apache.

test.mwl.io {
  reverse_proxy localhost:8080

  # Enable logging:
  log {
    output file /var/log/caddy/access.log
    format json
  }
}

That’s it.

Start Caddy. Point the browser at the HTTP site and it gets redirected to the HTTPS site. All this work, and I have replicated what I started with!

So let’s turn on QUIC and HTTP/3.

Sites must inform clients that QUIC is available through an HTTP header. The client makes an initial connection of HTTP/2, sees the header, and switches to HTTP/3 and QUIC. Add the header in the virtual host configuration.

Header set alt-svc "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600

Reload Apache. Set up your packet sniffer to watch UDP port 443. Point your browser at the web page.

So far, I like Caddy. It seems simpler than Apache. It is owned and backed by a commercial firm (ZeroSSL). I am careful going all-in on commercially-backed tools because the Internet’s business model is betrayal. I have other options if that happens.

I’ll deploy this on my main site to get some QUIC experience for the new Networking for System Administrators. QUIC isn’t essential today, but I want to future-proof it. I would be remiss if I didn’t mention that the book is open for sponsorships, for a little while longer at least.

Future path: do I need Apache? For some stuff, probably. But can I serve simple sites straight out of Caddy? Yes.