I just finished the first draft of “Run Your Own Mail Server.” Copies have gone to my volunteer tech reviewers and my sponsors.

When I need to mass-mail my sponsors, I normally can only mail a dozen or so at a time without making Google and Microsoft throw a fit. This time, I mailed all 147 sponsors at once. None of the big providers even looked askance.

Requested feedback by 15 April, just to make tax day extra special. That’ll let me open the Kickstarter by Penguicon.

Open source software has been adjusting its language. In a world without systemic racism, technologists could use words like “master” and “slave” without worries. While the Internet’s primordial developers chose those words without malice¹, we don’t live in that world. Much of the software in Run Your Own Mail Server is older, however. Many people who don’t speak English natively don’t fully understand the implications of “black list” and “white list” and don’t want to go through the annoyance of changing them in large code bases.

Part of my job is to be easily approachable to all readers who connect with my voice.² That means using language correctly. block list.

Another part of my job is to tell the truth. The software calls it black list. No matter how hard you search, you will not find rspamd’s block list.

I’m not going to reject rspamd or postwhite because of their language. To do so would inflict extra pain on my readers. So I’m putting this (raw, unedited) text in Chapter 0.

Today, we use the term “allow list” for entities that are permitted to skip a layer of protections, and “block list” for entities that are categorically refused. Many older programs and some software developed by non-native English speakers, still use the older blacklist and whitelist. This book uses modern language except when configuring those programs. Do please encourage your favorite developers to update their language to the 21st century, however.

This is the same approach I used in the latest Absolute FreeBSD with my beloved blacklistd, but made explicit. Also, blacklistd has been renamed. Even we greybeards can do better.

I much prefer using consistent language throughout, but reality has its own opinions.

Comments defending the old language will be summarily deleted. You also acted without malice? Fine. Now you know better. Do better.

I use bookshop.org to sell print books within the United States. It’s not a direct sale from me–I’m an affiliate, and I make a bit more when you buy from them.

For 11 and 12 July 2023, they are offering free shipping on all books.

So if you want one of my print books and you avoid the Bezosbeast, here’s a sale for you.

I’ve churned through much of the general stuff about email, and am about to dive into specific configurations and examples. In some ways, the protocol background is the hardest part of any book. Orienting the reader to understand the configuration examples and make their own decisions is a pain–though front-loading the hard stuff simplifies writing the rest of the book.

But this means I need to make final decisions on the book’s technology stack.

Postfix (backed with MariaDB) and Dovecot are absolutely in, period. But there are dozens of implementations of DKIM and greylisting and antivirus, of varying sizes and inclusivity. Postfix supports the milter interface, so there’s an overwhelming pool of plugins. Options, options!

I’m leaning towards rspamd, and providing antivirus and dkim through it.

One of the core Unix philosophies is “many small tools that each do a single thing well.” Rspamd is easy, but it certainly isn’t small. How much difference is there between “plug ClamAV and SpamAssassin and greylisting and SPF and OpenDKIM and RBLs into Postfix’s milter interface” and “plug ClamAV and greylisting and SPF and DKIM and RBLs into rspamd’s interface and milter rspamd into Postfix”? Not a huge amount. Either way, you can see all the connections and have source code to all the tools.

Then there’s the web IMAP client. Today’s contenders seem to be Horde and Rainloop. Horde is heavier than necessary. The free version of Rainloop is nice, but it’s commercially backed and I worry about enshittification. I’m not predicting that they will turn on their customers, but including such products in a book has burned me. Many times.

Anyway, that’s where the book is and what I’m doing. The obligatory reminder that you can still sponsor RYOMS.

The title kind of says it all.

I have also made an early draft of Chapter 0 available to help explain what the book is and is not. Before you start sending me corrections, check the disclaimer at the top.

Despite inflation, I have not raised the price of sponsorships. I have passed through the shipping charge for print sponsors, however, so I’ve totally raised the price. But it’s transparent, and relative to where you live. Note that the shipping is per-continent, not calculated live. I can’t calculate shipping for something that doesn’t even exist yet.

And yes, I’m building a Kickstarter into this book’s production plan. That will be more of a pre-order, although I do have some thoughts on how I can use Kickstarter to enhance the book. The Kickstarter will replace a direct pre-order, as an experiment.

Running your own email is a struggle against an opponent you can’t see. That’s why the cover is a take on Battle of Vimy Ridge. You’ll be able to buy Eddie Sharam’s original painting, either through the Kickstarter or direct auction. It will be a full wraparound cover, but here’s the first draft of the front.

Now, if you’ll pardon me, I have to go shell gmail. Uh, figure out why they’re not taking my mails, I mean.

I’m about to open “Run Your Own Mail Server” for sponsorships. This book is a little different than most other books I’ve written, so I’m sharing the introductory chapter.

This is uncopyedited. Unreviewed. It’s what we call a “vomit draft.” It exists to illustrate scope, not for folks to send me corrections. Yet. Before I complete the book, I will probably discard and redraft this chapter. Bridge Out Ahead. Slippery When Wet. No warranty of fitness for purpose. Probably causes cancer, chin hemorrhoids, and alternaria leaf blight. (Updated 2023-04-06)

---

Chapter 0: Introduction

Email is the beating heart of the Internet. Yes, over the last couple of decades online forums and social media and chat systems have blossomed and thrived, but we keep returning to email. Email works everywhere with an Internet connection, and many places without. Unlike chats, where the other people involved see that you’ve received their message and are choosing to ignore them, you answer email when you feel like it. Your correspondents don’t get a notification that you’re typing a reply and watch the screen until you compose words. Email is one of the few surviving asynchronous communication tools, and is available on every platform without vendor lock-in.

The first email-like message was sent in 1965 at MIT, when the Internet was only a drunken dream of comp sci grad students. When those ragtag students finally achieved a network, they developed the primordial Internet protocols. Today’s email looks nothing like theirs—for one, email addresses have those esoteric @ symbols in them rather than nice simple exclamation points. Modern email is built on those primordial protocols and the lessons learned from them, however. That’s not bad—all broadly used protocols evolved the same way. Today, a handful of providers dominate email services. Companies like Google and Microsoft control most email addresses. They’re an Email Empire, while those of us who run out own email form our own ragtag band struggling for independence.

It can be done. The protocols are comprehensible, the software freely available, the debugging tools comprehensible, and the hardware affordable.

The problem with running email has nothing to do with the technology. With the right knowledge, you can set up an email server in moments. The problem is that the email server is only one part of running an email server.

Consider setting up your first web site from scratch. You install the software, start the server, and point your web browser at the host’s IP address. You get the default page and a warm thrill of victory. You have a web server! If you put content on it, you can tell your colleagues “Hey, browse to 203.0.113.99 and you’ll find my new web site!” It’s a thing. Add an entry in the Domain Name Service (DNS) and you can offer folks a link like http://mwl.io. More testing and you add TLS, granting you the precious S in https. Step by step, you build a web site that complies with modern standards.[1]

Email isn’t like that.

Running email in the real world is not configuration problem. It is about citizenship. Once the Email Empire decides that your server is inadequate or untrustworthy, it’s very hard to get them to change their mind. You need to solidly participate in the email society before sending your first email to the outside world. Sure, you can set up a couple of throwaway hosts on expendable domains and and bounce email exclusively between them to see how the software works, but if you want to run a real email server and you want other servers to accept your email, you must quickly establish a good reputation.

Once you establish your system as a good citizen that follows standard practices, however, operating a mail server is comparatively low maintenance. Most of the time I spend on my email server is spent applying security patches, same as any other Internet-facing application. Running a mail server for a large corporation is a full time job, yes, but most of that is spent explaining to users that a bounce message of “The recipient’s email account is over quota” means that the email bounced because the recipient’s email account is over quota.

Run Your Own Mail Server uses common freely available software to illustrate the mechanics of serving email, but the book’s main focus is on establishing that citizenship. You can use any software you like that fulfills the necessary roles.

Why Bother?

If email is such a pain, why run it? Privacy, control, and education.

Service providers often scan email for advertising keywords or saleable personal data. While ad blockers keep the ads from your face, they solve the problem at the wrong end. Yes, if you email someone in the Email Empire address the Empire can tie the content to their records about your email address, but when you correspond with other outsiders they can’t. Email can be intercepted, but that interception takes work.

Privacy is especially important in business. A service provider might promise to not mine your company emails, but one of their sysadmins trying to fix your issues might have to examine your email. A long time ago, at an employer far far away, I had to troubleshoot a server problem caused by a user innocently attaching a word processor document to a mail. That was its own special fun, but—I had to examine the characters in the document to solve the problem. I saw parts of the document. While most of email’s rough edges have been burnished off since, violating privacy during troubleshooting remains a real possibility.

You control your own email. Many service providers offer fancy dashboards that let you add and delete users or set quotas, but when something goes sideways you’re stuck working with their support department. The quickest way to identify a problem is to check the log. Waiting for a support tech to answer your trouble ticket is infuriating. When you run your own email, you control the problem.

Perhaps the least valid but most important reason for many is pure “geek cred.” We want to know how things work. Just as building your own firewall is a great way to learn about networking, running your own email server is the best way to understand this ecosystem. You run your own web server, DNS server, cloud storage and home automation, why not run your own email? You can, and you will learn a lot.

Prerequisites

Operating an email server is not a task for a beginner sysadmin. You must have certain skills before you start.

You must have used email. You must be aware of things like email addresses, attachments, and related concepts.

You must know how to manage your operating system. You need to back up your servers, apply security updates, and in general make your hosts fit to sit on the naked Internet. This book assumes and recommends open source software. I demonstrate everything with Postfix on FreeBSD, but provide the context for all my configuration choices so you can make your own decisions on any platform.

You need at least two test hosts attached to the Internet. These could be minimal virtual machines out in the cloud, hosts in your lab, or whatever. Tests inflict negligible hardware load. These hosts need different but static IP addresses. You must also control the world’s access to those addresses, as email requires opening several TCP/IP ports to and from the servers. Most home Internet providers specifically block these ports.

Those hosts should be in different domains. Yes, you could send email between newyork.solveamurder.org and detroit.solveamurder.org, but your brain absorbs better patterns if you test between the wholly unrelated solveamurder.org and ratoperatedvehicle.com. Domains are inexpensive these days. Get two—or set up private DNS and test between example.org and example.net.

You must control the forward and reverse DNS for your test hosts. Supporting email for a domain requires creating and maintaining several DNS entries in that domain. Similarly, the reverse DNS must match the hostname and the forward DNS. If your host advertises its name as mail.solveamurder.org, but reverse DNS identifies your host as customer87.chicago.bighosting.com, your email will go nowhere. You must have correct reverse DNS on all IPv4 and IPv6 addresses. Follow best practices and have separate authoritative and recursive nameservers. The recursive nameservers should validate DNSSEC.

An understanding of basic public key cryptography is also essential. You don’t need to grind through Diffie-Hellman calculations by hand, but you do need to know the difference between public and private keys and how they’re used. You must routinely protect private keys. Similarly, you need a basic understanding of TLS. Users must authenticate to check their email, and sending usernames and passwords in clear text across the Internet is unwise. The X.509 certificates needed for TLS used to be very expensive, so by tradition SMTP does not require TLS. Certificates prices have plunged to zero, so there’s no longer an excuse to not use TLS. This book uses Let’s Encrypt certificates.

We will also use outside services to support email troubleshooting. These are all services that require writing custom code, a skill orthogonal to running a mail server. I’m certain that immediately after this book comes out, someone will release a package that lets us easily handle these features on our own systems.

If you lack any of these skills, permit me to recommend my books Absolute FreeBSD, 3rd Edition (No Starch Press, 2018), Networking for System Administrators (Tilted Windmill Press, 2015), and TLS Mastery (Tilted Windmill Press, 2021). I also recommend Cricket Liu’s DNS and BIND (O’Reilly, 2006)[2] and, once you’ve finished that, my own DNSSEC Mastery, 2^nd Edition (2022).

From now on, I will toss around terms like network port and X.509 certificate and floccinaucinihilipilification and expect you to either know what they mean or how to look them up. You cannot suffer from hippopotomonstrosesquippedaliophobia and run an email server.

Email Tools In This Book

Email isn’t a server, it’s a system with several components. The most obvious is a “mail server.” The mail server is a host that accepts email from users and exchanges mail with other servers. Most of the system speaks the Simple Mail Transport Protocol (SMTP). This book uses Postfix (https://www.postfix.org) as a reference platform.

Your users would probably like an option to check their email from their desktop. This might use a mail client like Thunderbird, using a protocol like IMAP. We’ll demonstrate clients and IMAP with Dovecot. Other users prefer a web interface, so we’ll discuss <<<pick one that speaks imap>>>. We’ll debug with mutt (https://www.mutt.org).

Once you have the core system working and can send email locally, you’ll need basic antispam protections. We’ll discuss greylisting and DNSBLs.

You can now risk telling the world about your mail system via DNS. This routes us into topics like Sender Policy Framework (SPF), and Domain Keys for Internet Mail (DKIM). With those basics you can look at acronyms like DMARC, BIMI, MTA-STS, and TLS-RPT. The main goal of this book is to teach you about these tools and protocols so that you can improve deliverability of your email.

We will focus on modern email. While some networks still rely on UUCP and bangpaths and POP3, those are vanishingly rare. Don’t deploy them on the modern Internet.[3] We will also ignore proprietary replacements for these protocols. Many mail service providers offer a convenient API that replaces SMTP or IMAP. There’s nothing wrong with using these if you’re willing to accept the vendor lock-in, but this book doesn’t cover them.

Spam

Unsolicited advertising is the scourge of email. While you’ll hear it called things like unsolicited commercial email (UCE) or junk, the most common technical term is “spam” because the Internet was built to a Monty Python soundtrack.

For about thirty years, people considered email a miracle. You could write a message at your desk and hours or even minutes later it would reach the other side of the planet. The early Internet was populated by computer science university students, select computer-related companies, and the US military. Blissfully free of broader commercial interests, SMTP was designed for that open network. Sysadmins ran open mail relays for the convenience of other sysadmins who suffered outages. Everyone worked together to sustain the miracle, with little concern about abuse.

In 1978 Gary Thuerk, an employee of the Digital Equipment Corporation, emailed several hundred people he didn’t know an invitation to a demonstration of the new DEC packet-switching systems.  Nobody had ever tried commercial advertising over email. Reactions were overwhelmingly negative, but reasoned. The broader community agreed that commercial advertisements were unacceptable. Five days after the message went out the chief of the US Air Force’s ARPAnet Management Branch, Major Raymond Czahor, called Thuerk’s boss to tell them to never do it again on pain of disconnection.

This first advertising email sold twelve million dollars of computers.

Thuerk’s email established the “do not advertise in email” precedent. It also sold twelve million dollars of servers, establishing the “random email marketing is highly profitable” precedent. Going forward, the network relied on the social contract and Air Force majors to prevent junk mail. Email protocols remained unchanged.

In 1990, UUnet started building its own commercial IP network. UUnet’s AlterNet could communicate with the Internet, but was not bound by government rules. While people that run businesses might be fine, most corporations are indifferent to feeble social contracts and do not answer to the military. The first spam hit Usenet in April 1994, and spam email followed immediately afterwards.

The first spam hit like an unexpected toxic spill at the kindergarten playground. Mail administrators panicked, implemented short-term fixes, and quickly learned that a protocol optimized for deliverability was unable to stop delivering messages. Sysadmins blocked junk mail domains, so mailers forged their originating domain. Blocking spammers’ networks led the spammers to relay their junk through well-meaning third parties or hacking into unrelated but unpatched mail servers. Despite spending time and money abusing the protocol, spammers made money.

Unchecked spam makes email useless. Almost every change in SMTP in the last thirty years has been propelled by the need to eliminate spam while retaining broad interoperability.

Opt-in email marketing is fine. If you want your favorite creator to tell you when their next big thing escapes,  great! You should get those emails until you opt out. I live and work in Detroit, though, and the dude that keeps trying to sell me courses on South African taxation law can take a long walk out of a short airlock.

If you wonder why email-related software works as it does, or why we have all these add-on protocols, the answer is almost certainly “spam.”

Deliverability

Your users expect their email to reach the intended recipient. Deliverability is how well mail from your system fulfills this expectation. Before spam, deliverability wasn’t a huge issue. In the 1970s and early 1980s you might have to route your email through U of C Berkeley to MIT to the Imperial College of London to TAMK via overloaded phone lines, but it was a well-understood problem. The decade of 1983 to 1993, after the standardization of the Simple Mail Transfer Protocol but before spam, was a golden age of email connectivity.

If mail from your server looks like spam, the destination server will discard it. Your email is not deliverable. As all mail systems have different spam-identification systems configured in different ways, deliverability varies by domain. The goal of this book is to help you make your email as deliverable as possible by configuring protocols that allow recipients to verify your mail’s authenticity.

Even with all of these defensive and authenticy protocols configured properly, you can still trash your deliverability by sending spam.

IP Addresses and Email

“I need full control of a test machine on the public Internet? I’ll go to my favorite provider and spin up a VM!”

Not so fast.

In the email ecosystem, IP addresses have reputations. Addresses previously used by spammers are distrusted. Anyone with an email address and a credit card can create a virtual machine in moments. Spammers have no problem setting up a cheap VM and spewing garbage into everyone’s inbox. While this violates every hosting company’s Terms of Service, spammers don’t care.

One of the tools mail administrators use to block spam is blocking network addresses of known spammers. Hosting provider IP addresses have almost certainly been used before. If a spammer previously used your address, it’s probably on one or more block lists of known bad addresses (Chapter XXX). Almost everyone uses at least one block list. Your email is undeliverable before you start.

With proper automation, a spammer can send millions of emails before the provider’s Network Operations Center can shut them down. A hosting provider will receive dozens or hundreds of complaints within minutes of this flood. While most providers shut down spammers quickly, those complaints cause extra work. Many providers will send lists of the IP addresses that they provide to customers to block list maintainers, so that those systems will proactively block email from those addresses.

Other providers take no action against spammers. Their terms of service declare that spam is not allowed, but they do not enforce it. Block list maintainers quickly identify IP addresses assigned to these providers and add them to their lists. No matter how much you intend to be a good citizen, any email server on these providers will have poor deliverability.

Before choosing a hosting provider for your email systems, ask around to see if other people run email servers on that provider and if that email is broadly deliverable. If you have no contacts who run mail servers, go ahead and try on your favorite provider but be prepared to move. I will not offer recommendations, as they will be obsolete within months.

Perhaps your organization has IP addresses issued by an ISP, or owns its own addresses. A previous owner could have soiled those addresses. Also, some Internet Service Providers block outgoing email at their network border. Check these before starting.

If your addresses have never been used for email, they have no reputation. “No reputation” does not mean that they are trusted, merely that they are not untrusted. You get to make your own reputation. This book will help you do it well.

If you have no control over your test system’s location, you can still educate yourself but remember that your tests will be limited in scope.

Email Components

If you’re accustomed to web-based email, you’ve probably picked up how mail works. When you send an email your web browser talks to the mail server. That mail server talks to the recipient’s mail server. When the human wants to read the email, they open their browser and see it.

If you use a desktop email client you know it’s a little more complex. You compose a message on the client and send it to your mail server. Your mail server sends the message to the recipient’s server. When the recipient opens their email client, the client downloads the email.

Even that’s a shallow view, however. Email was designed by actual computer scientists, and email systems have an actual architecture as documented in RFC 5598. Every email you send flows through the system. Here’s a slightly less shallow view.

Any email user interface is called the Mail User Agent (MUA). The MUA is most often called the mail client. It might be Thunderbird, a web page, or part of an office suite. You can even use a program like netcat to manually send mail to the server. (There are also command-line programs that you can run on the mail server itself, like mutt. They unfairly leverage their location, so we will ignore them for now.) When you send the email, the MUA transmits it to the Mail Submission Agent via SMTP. Usually, but not always, the client copies the message to the Message Store.

The Message Store (MS) is the server’s copy of the user’s emails. Most Message Stores retain a copy of both the ingoing and outgoing messages, at least until the user deletes them. Mail clients use the Internet Message Access Protocol (IMAP) or the older Post Office Protocol (POP3) to synchronize their messages with the server. On Unix systems the Message Store is often in the user’s home directory, something like /home/mwl/Mail.

The Mail Submission Agent (MSA) runs on a server. It accepts mail from mail clients and send it to the Mail Transfer Agent. The MSA communicates with SMTP.

The Mail Transfer Agent (MTA) receives emails from the Message Submission Agent and other Mail Transfer Agents, identifies their destination, and forwards them. The sender’s MTA stores outgoing email in a mail spool or a mail queue and attempts to identify and contact the recipient’s MTA. The two servers have a dialog over SMTP that boils down to “Hi, I have an email for one of your users.” “Yes, that’s my user, please send the mail.” You might hear the MTA called a mail exchange or an MX.

A MTA that receives a message forwards it to the Message Delivery Agent (MDA), sometimes called the Local Delivery Agent (LDA). The MDA accepts the message and stores it safely on disk. If you’re accustomed to managing Unix systems, the MDA places user messages in /var/mail. The MDA also handles tasks like server-side email forwarding, automated out-of-office messages, and filtering email into folders.

When the recipient’s MUA checks for new email and downloads it, it usually moves messages into the recipient’s Message Store. The recipient’s MUA might further sort the messages, delete certain messages, or any other sort of processing configured by the user.

Finally, the user’s mail client downloads the mail. The email is complete.

Out of all these components, which is the “mail server?” None of them, and all. Each component might be on a separate machine, or any of them could be combined with their peers. A single organization might have different MTAs for receiving and sending email. It depends entirely on the system load, the number of users, and which headaches each organization prefers. A user might call the Message Store or the MSA the mail server. Outside organizations might think that your MTAs that receive email from the outside world are your mail servers, unless they’re having trouble receiving emails from you and realize that you have entirely different MTAs for outbound mail. Even if you have only one server that controls all of these roles, stop using the phrase “mail server.” Look more deeply.

This book avoids saying mail server, except when I want to confuse you.

Email Security

Before spam, email clients authenticated only when connecting to the message store. Clients did not need to authenticate to send mail to the mail submission agent. Mail submission agents could dump email straight into the MTA. MTAs did not verify each others’ identity, and could often fling messages at any server that looked suitable. An unprivileged user could trivially forge emails from tax agencies. Everybody knew that email was public, and that someone with a carefully placed packet sniffer could capture messages.

The appearance of spam made everyone say “Maybe we should lock down these other connections?”

The server-to-server protocol is still called Simple Mail Transfer Protocol, but it’s much less simple. It now supports TLS. It uses different TCP/IP ports depending on which piece of the system you’re talking to.  The protocol commands are still plain text, however, and can be replicated by a sysadmin with netcat and knowledge. Even wrapping the connections in TLS only prevents eavesdropping in transit.

As with everything in computing, these protections are not perfect. TLS can be broken, through rogue Certificate Authorities and misconfiguration and good old-fashioned theft of private keys. Clever people can figure out how to violate the underlying protocols. You need to keep up with routine system maintenance.

SMTP itself is not enough to successfully exchange mail with another network, however. Email has added protocols that help secure, restrict, and authenticate email. Most of this book discusses those ancillary protocols. We’ll cover them briefly here to orient you, then delve deep into each.

DNS

The Domain Name Service maps hostnames and IP addresses to one another. You must have solid control of your DNS to successfully run email. Domains publish their MTAs in DNS, and many of email’s ancillary security protocols provide their information in DNS.

This raises the question of DNS security. Like email, DNS was created when the Internet was largely a private network. Also like email, security was integrated afterwards. DNS Security Extensions (DNSSEC) are today’s standard for maintaining the integrity and authenticity of DNS information, but have not yet been widely deployed.

Every domain that receives email should publish a Mail Exchanger (MX) record in DNS, telling remote MTAs the hostnames of the domain’s own receiving MTAs. You’ll need additional DNS records for each protocol you deploy.

Sender Policy Framework (SPF)

Where the MX record tells the world which hosts receive a domain’s email, the Sender Policy Framework (SPF) tells the world which hosts are authorized to transmit mail for a domain. SPF declares the hostnames and networks of outbound MTAs, not clients or mail submission agents. Modern SPF is distributed via DNS TXT records.

SPF syntax lets you declare who can send mail. You can make rules like “only these addresses,” “any of the MTAs for this other domain,” or even “this domain does not send mail.” For example, the SPF record for my domain mwl.io contains a few IP addresses and a hostname.

When a MTA receives a connection claiming to have mail from a particular domain, the MTA checks the SPF against the incoming machine’s IP address. If they don’t match, the connection is dubious and probably should be rejected. If someone sends email from mwl.io to a Gmail user, Gmail’s server checks the SPF record for mwl.io. If the forger’s IP address is not in my domain’s SPF record, Gmail drops the connection and records that the forger’s IP is untrustworthy.[4]

SPF is considered the lowest level of authenticity, and is easy to implement. It requires no maintenance, until you add a new outbound MTA.

DomainKeys Identified Mail (DKIM)

Where SPF declares where email comes from, DomainKeys Identified Mail (DKIM) authenticates the email’s sender. DKIM information         is published in DNS TXT records tied to a _domainkey record.

DKIM lets an organization publish a policy on how to authenticate mail from the domain. You can tell receivers to validate or ignore SPF, to report or ignore DKIM failures, and how to test messages.

DKIM is more difficult to implement than SPF, but not difficult. Best practices say you should rotate your keys regularly, so it does require more maintenance.

Domain-based Message Authentication, Reporting, and Conformance (DMARC)

SPF and then DKIM improved mail authentication. Domain-based Message Authentication, Reporting, and Conformance (DMARC) is the next evolution. It uses SPF to identify legitimate mail sources. It uses DKIM to authenticate email. Its robust policy language lets organizations define how recipients can identify legitimate mail and report failures in those two protocols.

DMARC records are distributed via TXT records, tied to a _dmarc record.

Brand Indicators for Message Identification (BIMI)

If your organization uses a trademark for marketing, you might have an interest in Brand Indicators for Message Identification (BIMI). If the end user has a BIMI-aware email client, it will check for BIMI DNS records and use them to display a digitally signed corporate logo next to the email. BIMI is built on DMARC, so it cryptographically verified.

Deploying BIMI starts with “register an expensive trademark” and “purchase an artificially expensive certificate,” so these two paragraphs are this book’s only coverage of the topic.

Mail Transfer Agent Strict Transport Security (MTA-STS)

Originally, email was transmitted unencrypted. Anyone with a properly placed packet sniffer could eavesdrop on all email transactions. We eventually added encryption to user authentication, securing usernames and passwords, but using TLS between MTAs was optional. Most mail operators who deployed TLS considered large-scale capture programs like Carnivore their primary risk, so self-signed certificates became acceptable even though they do not prevent other attacks.

Mail Transfer Agent Strict Transport Security (MTA-STS) is a declaration that your mail server only accepts SMTP over TLS. Your mail server must offer a policy statement on a web site. MTA-STS is activated via a DNS record, using the _mta_sts identifier.

TLS Reporting (TLS-RPT)

Every sysadmin has seen email bounce messages, declaring that the recipient has no account or that a server is misconfigured or that the receiving MTA is on an unscheduled vacation. These messages go between MTAs via SMTP.

If your MTA uses MTA-STS to declare it only accepts connections over TLS, and it has a TLS problem, those SMTP messages will not go through. Your MTA will continue merrily rejecting connections, leaving you ignorant of the problems. TLS reporting, or TLS-RPT, is how you MTAs exchange reports of TLS problems. If you deploy MTA-STS, you also need TLS reporting.

Like everything else in email, TLS-RPT information is distributed via DNS.

DNS-based Authentication of Named Entities (DANE)

One of the big objections to common X.509 certificates is the existence of Certificate Authorities (CA). While certificate prices have plunged, a compromised CA could still do immense damage to the confidentiality and integrity of its users.

DNS-based Authentication of Named Entities (DANE) is a way for organizations to publish X.509 certificate fingerprints in DNS, as documented in RFC 7671. It allows a domain owner to authenticate both self-signed and CA certificates. DANE is one of those contentious topics in security circles, and bringing it up at a technical conference is a great way to start the kind of argument that makes bystanders call law enforcement. The arguments about certificate authorities will not be settled by myself or (probably) anyone reading this book. Different people have different concerns about different risks and pain points, that’s all.[5] If someone has gone to the trouble of publishing DANE information for their mail servers, we should respect them.

DANE records are published in TLSA DNS records.

Which Do You Need?

These protocols were developed years apart. Some of their functions overlap. Which do you need, and which can you ignore?

Many people deployed the protocols available when they set up their server and never went circled back to add new ones. You’ll see servers with only SPF, or only SPF and DKIM. Which you need isn’t just a question of what you want to offer the world. It’s about what other servers can validate. If an MTA can validate only SPF, but you offer only DMARC, the recipient might consider your mails suspicious.

To maximize deliverability, and to be aware of any problems, implement SPF, DKIM, and DMARC for your outbound mail. To minimize junk mail, verify any and all of SPF, DKIM, and DMARC for your inbound mail.

Do you need MTA-TLS and TLS-RPT? That depends on your threat model. Maybe you don’t care incoming connections are encrypted or not. It’s fairly new, so perhaps your MTA doesn’t yet support it. If you’re deploying a new mail system or taking the time to polish your existing system, however, you might as well experiment with it. Use a very low timeout when testing, so remote servers won’t cache your mistakes for months or years.

What’s In This Book?

Chapter 0 is this introduction.

<<<add in as I write more>>>

[1] Hopefully you stop before you install WordPress and join the dark side. Unlike me.

[2] While I have written books on TLS, DNSSEC, and OpenPGP, as well as the infamous Networknomicon, I retain enough sanity and self-respect to not write a core DNS book.

[3] Unless your network connection involves phrases like “UDP over carrier pachyderm,” “we get phone service every third Sunday,” or “Callisto Colony One.” Then do what you gotta.

[4] If the forger’s address is in my SPF record, I am having a very bad day.

[5] Rule of System Administration #3 applies here. “Nothing is secure. Everything is terrible.”

 

[edit, 17 March 2023: Sponsorships are now open.]

My next tech book will be “Run Your Own Mail Server,” part of the IT Mastery series. I would like a snappier title. “Lay Down All Hope, You That Go In By Me” is accurate to the spirit, but doesn’t inform about the content. Now that the Devotion & Corrosion Kickstarter is over, I could theoretically open sponsorships for the new book. I’m not doing it yet.

I work to be honest in my books. I can’t yet express this book’s truth in a short pithy way suitable for a catalog.

Email is a huge topic. Postfix, exim or Exchange? Dovecot, Cyrus, or Courier? Sendmail or syphilis? What exactly is this book about, anyway? I’m using a couple programs for my reference implementation, but this is not exactly a book about system administration. It is about citizenship and society. A novice sysadmin will not be able to use this book without reading a bunch of other books first. This is mostly about how the system hangs together, and about the less well-known services that help email happen. SPF and DKIM, DMARC, MTA-STS, and TLS-RPT. How not to warm up your IP address. Defeating Google. IPv4 or IPv6?

The easiest way to communicate this is by posting Chapter 0 before I open sponsorships.

Which means that the chapter must be vaguely readable. It will be riddled with errors, but it will discuss what’s further in the book.

A handful of you don’t care what the book is about. You’ll sponsor it anyway. You’d sponsor a book about ls(1). You folks are the best, and I appreciate your eagerness to give me money for no good reason. Most folks are not so enraptured, however, and want to know something about what I’m writing.

Maybe next week? The week after? I write nonfiction in order but not linearly, so it’s taking longer than I hoped.

But book is happening. Eddie Sharam’s cover art is glorious, and it’s an actual painting again. It might be a Kickstarter reward, though I think he’d make more if he put it up for direct auction. We’ll find out.

I do know that email is overwhelmingly dominated by a handful of big companies. An Email Empire, if you will. Those of us who run our own email are a ragtag band of rebels struggling against the Empire, constantly hoping not to be crushed as we carve out our little space. Gmail has altered the email agreement, and we pray they do not alter it further. All we need is a Bigfoot with an energy-crossbow-thingamajig…