Email Alchemy - Exploring Self-Hosted Email Services

By Mark Piper    
"Email: Did that professionally years ago and would rather swallow live scorpions." -- /u/fliberdygibits

Very few technologies have persisted near to their original form as much as email services over time on the internet. It has been over 50 years since Ray decided to use the at symbol to designate a user and destination for electronic mail1. And while it has a few different skins, the fundamental concept and protocols remain largely unchanged to this day.

Given the lack of underlying evolution of email services, combined with the growth of ecosystems, there has been a dramatic shift to a seemingly monopolistic dominance of email services by Google, Microsoft and Apple (via Gmail, M365 and iCloud respectively). One major advantage enjoyed by users of the users of large providers is their price (free) and accessibility from anywhere (web and mobile applications). Combined with enhanced security capabilities and good spam mitigation, it’s completely understandable why we don’t really spend time thinking about non-mainstream email capabilities.

The scale of the larger operators is difficult to estimate, but it’s very safe to say that Google, Microsoft and Apple are responsible for the majority of email services online. As of March 2020, Google had 2 billion MAU alone2.

While Google and others provide easy access to email, it comes at a cost for other providers - to ensure safe delivery and reduction of spam across large user bases, the bar has been raised, and continues to be raised3 to successfully deliver email into each ecosystem.

Having recently had a small side project which required email capabilities whilst simultaneously wishing to avoid adopting any specific ecosystem, I set about building a self-hosted email solution.

I am a strong believer that very few problems that I face in life are unique, so I turned to the good folk at /r/selfhosted4 to find prior art relating to self-hosted email. What I found across dozens of threads relating to email capabilities was a mass of confusion, ad-hoc solutions, dodgy recommendations, delivery nightmares and general complexities intertwined with people who have successfully self-hosted mail for years without issue.

In short - mail hosting looked like a bit of a mess5.

The former systems administrator in me ended up wondering was it as hard as some on the subreddit asserted? Was it as easy as others suggest? Or does it land somewhere inbetween?

For the remainder of this essay, I will explore the basic design, build and deployment of Mail Transfer Agent (MTA) and Mail Delivery Agent (MDA) capabilities to host an informal email solution for a non-enterprise project.

  Scope

Build a self-hosted email server that provides mail services which can reliably deliver emails to the major providers (Gmail, Proton, iCloud, M365) without being canned like spam.

This simple scope is where I began. I must confess that given the informal nature of the project, I didn’t put a lot of forethought into the challenge but was smart enough to pencil the following objectives in a scratch file. In short, the solution should:

Given a list of what’s in scope, one can assume there was plenty considered out of scope. Out of scope items generally encompassed more modern, MUA-implemented features which one will find as a Gmail or Proton user. This includes send-later functionality, AI assisted drafting functionality, native calendar integrations and contacts management. Throughout the project I have also made ad-hoc decisions such as dropping support for legacy protocols (POP3) where it makes sense, but these were not always a deliberate design decision until later. I have also left out of scope building the secondary mail configuration (the secondary mx host in event of primary failure). Finally, I have de-scoped some more advanced tasks such as multi-factor authentication because this is best applied for web MUAs.

With the scope pencilled, there was then the matter of objectives. What was it going to take to get working email services up and running? These included:

30 years of being online has taught me that the internet is an incredibly unreliable place6.

Networks fail. Hard drives fail. Databases drop. Fishing vessels catch undersea cables. Earthquakes buckle racks. Certain governments have been known to just ‘turn it off’. Solar flares flip bits. Hackers hack. Data centres catch fire. Squirrels take out grids. BGP goes boom.

This is far from an exhaustive list, but illustrates that we must be prepared to engineer around a variety of issues.

To be successful, the solution needs to be reproducible and redeployable in the event of an incident, so I decided to automate it with Ansible. Ansible is a powerful configuration management tool which enables solution as code. If you are unfamiliar with Ansible I recommend reading Jeff Geerling’s fantastic book 7. There also exists a number pre-existing Ansible projects in the public domain that deal with building a self hosted email server and anyone looking into a solution should consider using well rounded projects such as Mail-in-a-box8. But for this exercise, I wanted to really understand what it takes and learn some Ansible along the way.

To support recovery, I would deploy with a previously tried and proven backup solution leveraging BorgBackup, an absolutely amazing bit of open source deduplication backup software.

With my sleeves rolled up and looking for a system administration fight, I had two main remaining questions before getting started:

The first of these questions - hosting - turned into an interesting problem statement.

Over the years, as the adoption of email grew, so did the amount of spam. Like all significant issues with exponential growth, this caught the eye of governments with the hope of regulating themselves out of a technological problem. Politicians hate getting spam.

The US government was no exception and ended up with the creatively named Controlling the Assault of Non-Solicited Pornography And Marketing Act9 or CAN-SPAM for short. This rolls-off-the-tongue act outlines a large number of provisions which discourage situations where unsolicited spam goes unchecked. Whilst the act doesn’t explicitly put restraints or regulations on cloud providers, it has created an environment where providers are now extremely cautious to be associated with spam in any way, shape or form.

The net result of this environment is that many cloud providers (including Google, Amazon, DigitalOcean and Linode) deny TCP/25 for ingress, egress or both by default. While you can appeal and request explicit enablement of these services, that is becoming harder and harder and aimed at business users, not individuals or hobbyists.

It is with caution then, one must select their cloud provider to host email services (if not hosting via a dedicated server or hardware under your control). While I found a number of providers who would, upon request, open port 25 for me, very few meet the objective of being price accessible (that is to say - not stealing more than a cup of coffee per month from the good folk of /r/selfhosted).

It was with some trepidation, but out of necessity, I ended up on OVH VPS services who do allow TCP/25 (ingress and egress) by default. The VPS includes a single CPU vCore, 2 GB of Ram and 20GB of SSD disk which should be enough for our needs. All for $5 per month. Perfect.

My trepidation did not only sit with the fact that they have previously had Data Centres catch fire10, or in the fact that much of their address space is tagged as untrustworthy a lot of the time, but more in the fact this meant I didn’t truly understand how the data is handled, or accessed, at rest. Nor could I find an immediate answer to our questions.

Cloud is many things and marketed in many different ways but at its essence, you are renting a computer from someone else. Without a good understanding of that computer, you can never really trust where your data is and how it is handled.

  Data at Rest

Data at Rest matters. If you do not know how your data is handled (encrypted) when sitting on disk, then you may not fully understand the risk of disclosure of that data.

Given I am unsure how OVH will handle my data at rest, I have not used this solution for any personal data I considered ‘sensitive’.

For the purposes of essaying entertainment and on the assumption I am not hosting anything particularly sensitive, I continue on this adventure but please stay aware of the risks if you undertake your own email server build.

Having buried my trepidation in the backyard, I moved onto the second question.

  A note on DNS

When configuring the VPS ensure your DNS pointer record (PTR) matches that of the mail system name you are building (i.e mx.iris.com). Whilst it shouldn't matter for spam mitigation later, many providers still insist on PTR records matching the A (forward) records for successful delivery.

What Operating System?

In technology there exists a few persisting religious wars. Editor of choice and Unix distribution of choice. For this project I considered FreeBSD, OpenBSD, NetBSD, Debian Linux and Fedora Linux. My work history has afforded me production experience in all of the above, but to ensure successful hosting on OVH, I narrowed it down to Debian and Fedora. Both have massive communities of support, are battle tested in some of the toughest environments in the world and are accessible to hobbyists.

While I love Debian, I chose Fedora as the workhorse of choice for this project. I did so for two key reasons:

  1. Prior experience has taught me that there is significant complexity in inter-process communications between the various components of mail services. Applying hardening across these multi-threaded processes while dropping privileges can sometimes be challenging with the likes of AppArmor.

  2. Package and release management - Personal preference has me wanting faster release cycles for non-critical projects. I believe Fedora (and Ubuntu LTS etc) is reasonably stable unless you are running critical projects. Living on the edge is fine for this project.

  On Debian

While my decision is Fedora, before publishing this paper I did take the time to test the general approach on Debian with minor changes (such as packages installed by Ansible) and it all worked fine.

"I'd rather eat broken glass than manage my own email server again." -- /u/ithakaa

Reliable mail transport and delivery is a complex problem. The protocols in use are ageing and in some respects have struggled to keep up with the evolution (volume) of other modern internet services.

To simplify the complexity and improve reliability, message handling is split into two key areas: mail transfer (transporting messages) and mail delivery (shoving messages in the user’s inbox).

Mail Transfer Agents (MTAs) are responsible for interfacing with other email servers (along with message submissions from users). Whether sending or receiving email, the MTA orchestrates the various tasks required to do the job. These tasks include routing, queue management, spam detection, security filtering and bouncing messages. On the surface all seemingly simple tasks but in the real world success relies on layers of complexity having been introduced over time across large code bases.

Universally, the transfer of messages between systems is done over the Simple Mail Transport Protocol (SMTP). This ageing protocol is somehow still responsible for the delivery of email between systems online and given the lack of reliable (and widely adoptable) alternatives, it remains king in the world of electronic mail. The most significant change to SMTP in the last 20 years has been the addition of Transport Layer Security (TLS) support.

A number of free and open source MTAs are available for system operators including Postfix, Exim, QMail and OpenSMTPD. Each comes with their own limitations, quirks and security history. Given SMTPs length of service, there are plenty of MTAs around to choose from.

When making the decision on which MTA to use, I examined the following key attributes:

Based on this very fast and loose list, I ended up selecting Postfix11 for the MTA. It’s free and open source. It’s packaged natively. It has a long history of production deployments. It has a strong history with security. It is actively supported. Everything I could ask for to meet the needs of this project.

While the MTA is responsible for the sending and receiving of email, the Mail Delivery Agent (MDA) is very much obsessed with the delivery of an email message to the end user. Beyond that, the MDA is responsible for other important tasks such as mailbox management (folders) and email filtering (rules). It implements protocols such as IMAP and POP3 for the clients to access their messages.

Similarly to MTAs, there are a number of free and open source MDAs available for you to choose from when building the service. These include Dovecot, Courier Maildrop and Cyrus IMAP Server. Key attributes I looked at during MDA selection largely matched the MTA requirements - with one exception: its interoperability with MTAs. This led to the (almost obvious) decision to leverage Dovecot12 as the MDA.

It turns out, I was not alone in this decision and the Postfix / Dovecot combo is a popular one. This gave me a nice warm fuzzy feeling as I pushed on with this seemingly simple task. There should be no major issues using natively packaged versions, have them interact with each other and do so on limited resources VPS. Furthermore, I should be able to secure them accordingly. As we see below, these assumptions turned out to be accurate.

For the purposes of our configuration, we want Virtual Mail (vmail) users. Traditional Unix MTAs expect the target mail user to be an actual user on that system. Time moves slow in Unixland, and by default even Postfix on Fedora will expect a local user to deliver to.

We don’t want to create a full Operating System user for mail access only (for a variety of reasons including security), so instead we will create a MariaDB database which defines our virtual users for mail services to consume.

With the high level design made, we ended up with the following initial stack:

And the next steps looked something like:

The overall installation, configuration and use of Ansible is out of scope of this work, however. I have provided an example Ansible project13 as a starting point for those working on their own server.

To get started, I wanted a robust foundation for the mail server installation. To achieve this, I created a basic ansible role (base_os) which does the following:

For the purposes of the example Ansible (and examples below), the domain is iris.com, operating mail on mx.iris.com14.

With the base installation done, it was time to get started on the mail specific services. Often, people will start with the MTA followed by the MDA for installation. This makes sense in many cases.

I took a slightly different approach, this is because I wanted Dovecot to act as the authentication provider for users. We achieve this by leveraging the Simple Authentication and Security Layer (SASL) framework. SASL will take care of authenticating users accordingly.

If we walk the dependency chain for installation there are a few prerequisites for Dovecot to work:

To cover all the tasks, I created a new role (mailserver) in Ansible which is available in the example repoistory.

The mailserver task incorporates subtasks for each of the mail components needed in order for the mailserver to work.

tasks/main.yml

--- 
- include_tasks: packages.yml
- include_tasks: vmail-user.yml
- include_tasks: firewall.yml
- include_tasks: letsencrypt.yml
- include_tasks: database.yml
- include_tasks: dovecot.yml
- include_tasks: postfix.yml

As you can see, it configures mail packages, sets up vmail, configures the firewall for immediate effect, creates TLS certificates, creates the database and then configures the Postfix / Dovecot combination.

The `tasks/packages,yml`` task ensures all required packages are installed.

Once packages are installed, it configures the vmail configuration. Whilst the server won’t have a user account for each user on the OS itself, it will still need to store their mail on disk (in the Maildir format). This is done under /var/vmail. The vmail.yml tasks file creates a vmail user (to reduce requirements for privileged users running processes) and ensures the /var/vmail is appropriately created:

tasks/vmail.yml

--- 
# Ensure user / group are created. 
- name: Add Vmail Group 
  group: 
    name: vmail 
    gid: 5000 
    system: yes 
    state: present

- name: Add Vmail User 
  user: 
    name: vmail 
    uid: 5000 
    group: vmail 
    shell: /usr/sbin/nologin 
    system: yes 
    state: present

# Create /var/vmail and assign ownership to vmail user / group.
- name: Create /var/vmail
  ansible.builtin.file: 
    path: /var/vmail 
    owner: vmail 
    group: vmail 
    mode: '0770' 
    state: directory

Next, it ensures FirewallD (the firewall manager for Fedora) enables the following ports with immediate effect:

Port Purpose Service
25/TCP SMTP(s) for receiving email for users. Postfix
80/TCP Let’s Encrypt HTTP challenge. Certbot
465/TCP TLS based email submission by users. Postfix
993/TCP IMAPS (IMAP over TLS). Dovecot

Let’s Encrypt15 is a free, automated and open certificate authority (CA). Previously, securing services with TLS / SSL was in some ways prohibitive due to the cost involved in obtaining a TLS certificate for your website or service. Let’s Encrypt is a public service that removes this limitation and enables any service to ensure robust, secure communications by ensuring a complete certificate chain on most major operating systems.

As outlined in the above table, there are three key services to secure with Transport Layer Security - SMTP with STARTTLS, email Submissions and IMAP access.

The letsencrypt.yml uses the Let’s Encrypt provided certbot utility to request a new certificate for mx.iris.com (as defined in the ansible vars) and complete the host / response (verification) challenge over HTTP.

tasks/letsencrypt.yml

---
- name: Generate certificate for host alias 
  ansible.builtin.shell: 
    cmd: "certbot certonly --standalone --non-interactive --agree-tos --no-eff-email --email {{ certbot_email }} --domains {{ mailserver_alias }}" 
    creates: "/etc/letsencrypt/live/{{ mailserver_alias }}/fullchain.pem"

Next, there is a requirement to create the mailserver database and populate the tables with some initial data. The database task I wrote for this is a little more complex than the others in the mailserver role. In summary, it performs the following:

  1. Create the database mailserver.
  2. Create a database user and associated password for the mailserver database.
  3. Create 3 database tables in mailserver - domains, users and aliases.
  4. For each domain defined in the vars/mailserver.yml file, create a domains table entry.
  5. For each test user defined in the vars/mailserver.yml:
    • Look up their initial password from the ansible vault and hash it.
    • Insert test user into the users table.
  6. For each alias defined in the mailserver.yml, create the aliases table entry.
  7. Finally, we’re going to ensure database backups occur at 3am every day.

The tasks file also relies on some additional data read from the vars/mailserver.yml file. This file defines 3 key items:

If using the example Ansible repository, you’ll want to edit that first.

vars/mailserver.yml

--- 
domains_to_add: 
  - { domain: 'iris.com }
  
users_to_add:
  - { domain: 'iris.com', email: 'user@iris.com', password: '{{ user_mail_password }}' }

aliases_to_add: 
  - { domain: 'iris.com', source: 'root@iris.com', destinations: 'user@iris.com' }
  - { domain: 'iris.com', source: 'postmaster@iris.com', destinations: 'user@iris.com' }

tasks/database.yml

--- 
- include_vars: mailserver.yml

# Ensure MariaDB is running. 
- name: Ensure MariaDB Is Running 
  ansible.builtin.service: 
    name: mariadb 
    state: started 
    enabled: true

# Create the Database.
- name: Setup the mail database 
  ansible.builtin.mysql_db:
    name: mailserver 
    state: present

# Create a user for Postfix / Dovecot to use. 
- name: Create mail user for the database 
  ansible.builtin.mysql_user: 
    name: mailserver 
    password: "{{ mailserver_password }}" 
    priv: "mailserver.*:ALL" 
    host: localhost 
    state: present

# Create 3 tables: 
# - Domains (id, name)
# - Users (id, domain_id, password, email)
# - Aliases (id, domain_id, source, destinations)
- name: Create tables
  ansible.builtin.mysql_query:
    login_user: mailserver
    login_password: "{{ mailserver_password }}"
    login_db: mailserver
    query: 
    - "CREATE TABLE IF NOT EXISTS domains (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, name VARCHAR(50) NOT NULL UNIQUE);"
    - "CREATE TABLE IF NOT EXISTS users (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, domain_id INT NOT NULL, password VARCHAR(106) NOT NULL, email VARCHAR(120) NOT NULL UNIQUE, FOREIGN KEY (domain_id) REFERENCES domains(id) ON DELETE CASCADE);"
    - "CREATE TABLE IF NOT EXISTS aliases (id INT(11) PRIMARY KEY AUTO_INCREMENT, domain_id INT(11) NOT NULL, source VARCHAR(120) NOT NULL UNIQUE, destinations VARCHAR(255) NOT NULL, FOREIGN KEY (domain_id) REFERENCES domains(id) );"
    single_transaction: true 

# For each of our domains, add them to the database.
- name: Add domains to the database 
  ansible.builtin.mysql_query: 
    login_user: mailserver
    login_password: "{{ mailserver_password }}"
    login_db: mailserver
    query: >
      INSERT INTO domains (name)
      SELECT '{{ item.domain }}'
      WHERE NOT EXISTS (SELECT * FROM domains WHERE name = '{{ item.domain }}');      
  loop: "{{ domains_to_add }}"

# For each of our users, generate a password hash and add them to the database.
- name: generate BLF password hashes
  command: doveadm pw -s BLF-CRYPT -p "{{ item.password }}"
  register: hashed_passwords 
  loop: "{{ users_to_add }}"
  no_log: true

- name: Insert user into the database with the hashed password 
  ansible.builtin.mysql_query: 
    login_user: mailserver
    login_password: "{{ mailserver_password }}"
    login_db: mailserver
    query: >
      INSERT IGNORE INTO users (domain_id, password, email)
      SELECT domains.id, '{{ item.1.stdout }}', '{{ item.0.email }}'
      FROM domains
      WHERE domains.name = '{{ item.0.domain }}';      
  loop: "{{ users_to_add | zip(hashed_passwords.results) }}"
  no_log: true

# For each alias, add them to the database.
- name: Add aliases to the database 
  ansible.builtin.mysql_query: 
    login_user: mailserver
    login_password: "{{ mailserver_password }}"
    login_db: mailserver
    query: >
      INSERT IGNORE INTO aliases (domain_id, source, destinations)
      SELECT domains.id, '{{ item.source }}', '{{ item.destinations }}'
      FROM domains
      WHERE domains.name = '{{ item.domain }}';      
  loop: "{{ aliases_to_add }}"

# Ensure backups
- name: Create database backup script 
  ansible.builtin.template: 
    src: mysql-backup-sh.j2
    dest: /usr/local/sbin/mysql-backup.sh
    owner: root
    group: root
    mode: '0750'

- name: Schedule database backup daily at 3am
  ansible.builtin.cron: 
    name: "MySQL Backup" 
    minute: 0
    hour: 3
    job: "/usr/local/sbin/mysql-backup.sh"
    user: root

After successful execution the database mailserver now exists with the following tables:

[root@mx-iris ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1895
Server version: 10.5.22-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use mailserver;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mailserver]> show tables;
+----------------------+
| Tables_in_mailserver |
+----------------------+
| aliases              |
| domains              |
| users                |
+----------------------+
3 rows in set (0.000 sec)

MariaDB [mailserver]> describe domains;
+-------+-------------+------+-----+---------+----------------+
| Field | Type        | Null | Key | Default | Extra          |
+-------+-------------+------+-----+---------+----------------+
| id    | int(11)     | NO   | PRI | NULL    | auto_increment |
| name  | varchar(50) | NO   | UNI | NULL    |                |
+-------+-------------+------+-----+---------+----------------+
2 rows in set (0.001 sec)

MariaDB [mailserver]> describe users;
+-----------+--------------+------+-----+---------+----------------+
| Field     | Type         | Null | Key | Default | Extra          |
+-----------+--------------+------+-----+---------+----------------+
| id        | int(11)      | NO   | PRI | NULL    | auto_increment |
| domain_id | int(11)      | NO   | MUL | NULL    |                |
| password  | varchar(106) | NO   |     | NULL    |                |
| email     | varchar(120) | NO   | UNI | NULL    |                |
+-----------+--------------+------+-----+---------+----------------+
4 rows in set (0.001 sec)

MariaDB [mailserver]> describe aliases;
+--------------+--------------+------+-----+---------+----------------+
| Field        | Type         | Null | Key | Default | Extra          |
+--------------+--------------+------+-----+---------+----------------+
| id           | int(11)      | NO   | PRI | NULL    | auto_increment |
| domain_id    | int(11)      | NO   | MUL | NULL    |                |
| source       | varchar(120) | NO   | UNI | NULL    |                |
| destinations | varchar(255) | NO   |     | NULL    |                |
+--------------+--------------+------+-----+---------+----------------+
4 rows in set (0.001 sec)

  Lock It Down

MariaDB (and MySQL) includes a helpful hardening script called mariadb-secure-installation. Be sure to manually run this script and lock down the MariaDB instance before going into production.

With the foundational services in place, it was now possible to configure Dovecot to act as the MDA. For our Dovecot Ansible task, it generates and writes the configuration files which define the following:

After writing the files, the Dovecot service will restart. After running the task I was able to a) verify sockets for IMAPS, LMTP and AUTH and b) test authentication with my test user.

[root@mx-iris ~]# netstat -anp | grep dovecot
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      77669/dovecot
tcp6       0      0 :::993                  :::*                    LISTEN      77669/dovecot
unix  2      [ ACC ]     STREAM     LISTENING     393918   58849/master         private/dovecot
[...]
unix  2      [ ACC ]     STREAM     LISTENING     496904   77669/dovecot        /run/dovecot/lmtp
[...]
unix  2      [ ACC ]     STREAM     LISTENING     496907   77669/dovecot        /var/spool/postfix/private/dovecot-lmtp
unix  2      [ ACC ]     STREAM     LISTENING     496972   77669/dovecot        /var/spool/postfix/private/auth
unix  2      [ ]         DGRAM      CONNECTED     496869   77669/dovecot
[...]

I validated auth by connecting with OpenSSL’s built in client and sending a login string (a login user@iris.com generate-a-random-password):

[root@mx-iris ~]# openssl s_client -connect 127.0.0.1:993 -quiet
Can't use SSL_get_servername
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = mx.iris.com
verify return:1


* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN] Dovecot ready.


a login user@iris.com generate-a-random-password


a OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged in      

As shown ‘a OK’ (great foresight by the protocol developers of IMAP16) demonstrates successful user authentication via Dovecot SASL. With confirmation auth is working, it means I could now get to the pointy end of the stick and deploy the MTA.

"Shout out to the 11% of people self-hosting email. Brave souls." -- /u/essjay2009

Any systems engineer will tell you that it’s often a wonder that anything works at all. As we’ve previously discussed, the internet is a chaotic, messy and dangerous place. It’s with some surprise then, as to how generally reliable email routing and inter-system delivery really is. For this, we thank good MTAs and the SMTP protocol which, while butchered and extended, remains unchanged in its nature for as long as SMTP has existed.

In establishing the MTA, I wanted Postfix to achieve a number of key tasks, including:

  • Listening for inbound email from other systems for our domain.
  • Determine if a (virtual) user exists.
  • Queueing incoming messages for spam processing and delivery to the MDA.
  • Queue processing via spam mitigation capabilities.
  • Rejecting the bad ones.
  • Securely accepting outbound emails from users MUAs and queueing them for processing.
  • Determine where to send the user originated message.
  • Strip certain heads to preserve privacy of the user.
  • Securely deliver the message to the target desitnation.

There are a lot of moving parts and there’s plenty that can go wrong. To assist with reducing mistakes, I focused on keeping it as simple as possible and make as few modifications to the base configuration as possible.

The Postfix configuration matches the default file structure as outlined by the Fedora distribution with some modifications as required. It roughly looks like the following:

  • master.conf: Define the service listeners and processes.
  • main.conf: Define the internal configuration, mail delivery instructions and TLS/SSL configuration.
  • mysql-{aliases,domains,users}.conf: SQL instructions for the virtual users.
  • smtp_header_checks: Regex file to strip headers for privacy.

Similarly to the Dovecot configuration, rather than outline the entire configuration in the Ansible role, the following are some of the key configuration directives:

postfix/master.conf

# Define our listeners for SMTP
smtp      inet  n       -       y       -       -       smtpd
smtpd     pass  -       -       y       -       -       smtpd

# Secure SMTP service configuration
smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_tls_security_level=encrypt

# Submission service configuration
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_tls_auth_only=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=$mua_client_restrictions
  -o smtpd_helo_restrictions=$mua_helo_restrictions
  -o smtpd_sender_restrictions=$mua_sender_restrictions
  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

# Define our listeners for LMTP
dovecot   unix  -       n       n       -       -       lmtp

postfix/main.conf

# Server configuration 
myhostname = {{mailserver_alias}}
mydomain = {{mailserver_domain}}
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, localhost.$mydomain, localhost
alias_maps = hash:/etc/aliases

# Configure our virtual mail accounts 
virtual_transport = lmtp:unix:private/dovecot-lmtp
virtual_mailbox_domains = mysql:/etc/postfix/mysql-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-users.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-aliases.cf

virtual_mailbox_base = /var/vmail
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
virtual_mailbox_limit = 512000000

# Configure Dovecot as our SASL authentication provider. 
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_tls_auth_only = yes
disable_vrfy_command = yes

# Ensure hardened TLS/SSL services with valid certificates. 
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_use_tls = yes
smtp_tls_protocols = !SSLv2, !SSLv3
smtpd_tls_protocols = !SSLv2, !SSLv3
smtp_tls_ciphers = high
smtpd_tls_ciphers = high
smtp_tls_exclude_ciphers = aNULL, MD5
smtpd_tls_exclude_ciphers = aNULL, MD5
smtp_tls_note_starttls_offer = yes
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_exclude_ciphers = aNULL, MD5
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1, !TLSv1.2

smtpd_tls_cert_file=/etc/letsencrypt/live/{{mailserver_alias}}/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/{{mailserver_alias}}/privkey.pem
smtpd_tls_CAfile=/etc/letsencrypt/live/{{mailserver_alias}}/chain.pem

# Configure restrictions on email sending (prevent open replay)
smtpd_sasl_security_options = noanonymous
smtpd_tls_auth_only = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
smtpd_client_restrictions = permit_mynetworks

smtpd_helo_required = yes
smtpd_helo_restrictions = 
    permit_mynetworks, 
    permit_sasl_authenticated, 
    reject_invalid_helo_hostname, 
    reject_non_fqdn_helo_hostname, 
    reject_unknown_helo_hostname

smtpd_recipient_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_invalid_hostname,
    reject_unknown_recipient_domain,
    reject_unauth_destination,

Finally, the Ansible playbook undertakes one additional step which is to remove Received and X-Originating-IP headers from outbound messages. These headers often contain potentially sensitive information for some users including their RFC1918 IP and external origin IP when sending email via Postfix.

The role task writes /etc/postfix/smtp_header_checks and then calls postmap /etc/postfix/smtp_header_checks which creates an indexed version (/etc/postfix/smtp_header_checks.db) for processing by Postfix:

postfix/smtp_header_checks

/^Received: .*/     IGNORE
/^X-Originating-IP: .*/     IGNORE

With everything going well during playbook execution, I ended up with a smooth set of running Postfix services:

[root@mx-iris ~]# netstat -anp | grep -i master
tcp        0      0 0.0.0.0:465             0.0.0.0:*               LISTEN      179551/master
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      179551/master
tcp6       0      0 :::465                  :::*                    LISTEN      179551/master
tcp6       0      0 :::25                   :::*                    LISTEN      179551/master
unix  3      [ ]         STREAM     CONNECTED     879468   179551/master
[...]

[root@mx-iris ~]# netstat -anp | grep -i postfix
unix  2      [ ACC ]     STREAM     LISTENING     704606   167617/dovecot       /var/spool/postfix/private/dovecot-lmtp
unix  2      [ ACC ]     STREAM     LISTENING     704673   167617/dovecot       /var/spool/postfix/private/auth

With Postfix running I could then verify user authentication by calling the SMTP Submission service as the test user. AUTH PLAIN expects the format [null byte][username][null byte][password]. To test, first generate this string on the command line and then provide it during the session. For example:

[root@mx-iris ~]# echo -ne '\0user@iris.com\0generate-a-random-password' | base64
AHVzZXJAaXJpcy5jb20AZ2VuZXJhdGUtYS1yYW5kb20tcGFzc3dvcmQ=

We can now send the base64 string via the AUTH PLAIN statement:

$ openssl s_client -connect localhost:465 -crlf -quiet
Can't use SSL_get_servername
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = mx.iris.com
verify return:1
220 ESMTP - Notice: This is an experimental service on iris.com
EHLO localhost
250-mx.iris.com
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-AUTH PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250-SMTPUTF8
250 CHUNKING
AUTH PLAIN AHVzZXJAaG9zdC5jb20AZ2VuZXJhdGUtYS1yYW5kb20tcGFzc3dvcmQ=                                                                                              
235 2.7.0 Authentication successful               

With authentication confirmed, I could now test sending email to myself. I configured Mozilla Thunderbird with a fresh profile (manual) and the following server settings:

IMAP Server:

  • Server Type: IMAP Mail Server
  • Server Name: mx.iris.com
  • User Name: user@iris.com
  • Port: 993
  • Connection Security: SSL/TLS
  • Authentication Method: Normal password

SMTP Server:

  • Description: user@iris.com
  • Server Name: mx.iris.com
  • Port: 465
  • Connection Security: SSL/TLS
  • Authentication Method: Normal password
  • Username: user@iris.com
"As someone who self-hosts email, my advice is to not self-host email." -- /u/Prawny

With a functional mailserver up and running a number of the original objectives had been met. However, I still had the significant challenge of spam mitigation to address.

It’s hard to measure via exact figures, but spam and phishing attacks make up a significant portion of emails online17. The job now was to try and reduce that for the end users I’m hosting.

When it comes to spam mitigation, no single approach works best. Spam killing is generally achieved through a variety of methods including community domain blocklists, policy enforcement (DMARC, SPF and DKIM), content scanning (fuzzy hashing and regular expressions) and statistical analysis (Bayes classification and machine learning).

I wanted to leverage as many of the approaches as possible to maximise the chances of catching both spam and phishing attempts against users. When looking for open software to do the job two primary contenders stood out - SpamAssassin and Rspamd.

I would love to say I ran an objective comparison of the two capabilities but truth be told in a prior life I was responsible for administration of SpamAssassin and didn’t enjoy the experience. As a result, Rspamd18 became the solution of choice for this project.

  Complex features means complex code

Complex features almost always means a complex code base. Rspamd brings a huge number of powerful features to administrators but at a cost: a complex code base. Be careful when enabling features and where possible understanding each feature is expanding the potential attack profile of your services.

At time of writing, there really has not been enough volume of traffic to truly understand the effectiveness of the anti-spam measures. The observed spam making it through generally via compromised mail accounts of legitimate domains and users or popular mail delivery platforms such as Amazon SES or Mailgun. For these instances, the mail comes through with SPF and DKIM pass results. I have deployed the Bayes classification and neural network features provided by rspamd which are currently in training and hope they help alleviate the success rate of spam into users inboxes.

In addition to checking spam inbound, rpsamd is configured to DKIM sign outbound messages to increase the likelihood of delivery into major provides such as Google.

Like the Dovecot and Postfix configuration, rspamd supports splitting configuration files and configuration file overrides. The configuration is written to the local.d/ configuration folder for the bits we wish to customise. The configuration roughly looks like the following:

  • dkim_signing.conf: Configure DKIM signing.
  • dmarc.conf: DMARC configuration.
  • milter.conf: Define the mail filter (milter) configuration for Postfix to call.
  • milter_headers.conf: Inject spam score information into mail headers.
  • multimap.conf: Configure the X-Spam-Flag header.
  • neural.conf: Define the neural learning function.
  • rbl.conf: The realtime blackhole list configuration.
  • redis.conf: Configure redis for Neural and Bayes functions.
  • worker-controller.inc: Configure passwords for the worker process.

The majority of rspamd features will work well out of the box or with minor tuning to the configuration. The primary exception to this is the DKIM signing functionality.

When it comes to policy frameworks for email, there are three key frameworks which most mail providers will acknowledge:

  • Sender Policy Framework (SPF)19 is used to reduce the likelihood of spoofing being successful for mail domains. It can be considered as a kind of server authorisation protocol which is defined via DNS.

  • DomainKeys Identified Mail (DKIM)20 leverages public / private key infrastructure to sign emails. If we consider SPF to be authorisation, DKIM is the authentication of messages. DKIM signed messages can be validated via DNS records.

  • Domain-based Message Authentication, Reporting, and Conformance (DMARC)21 is not just a long name. DMARC let’s mail operators define how mail servers should behave in the event of a SPF or DKIM failure (someone spoofing mail). This includes not only the behaviour but where to report violations to. Like all good policy frameworks, this too is configured via DNS records that we will look at below.

In order to deliver email to other mail providers without being marked as spam, I ideally needed to implement all three policy frameworks with the mail server.

For SPF, it’s as simple as defining a DNS TXT record for each domain such as the following:

v=spf1 a:mx.iris.com ip6:2402:dead:beaf:4242::0000 mx -all

In short, this record defines a SPF version 1 record and defines the MX DNS entries, the host mx.iris.com and a specific IPv6 address as authorised to send mail on a iris.com’s behalf. If any attempt doesn’t match those 3 defined methods, the SPF fails.

Having SPF pass will work for the majority of providers to avoid getting marked as spam, however, DKIM is now increasingly required. During initial testing of the server I found numerous instances where my mail was marked as spam even though SPF clearly passed. To resolve this, I decided to bite the bullet and implement DKIM for the outbound domains.

To support DKIM, there are generally three key steps required:

  1. Generate a private key for the domain.
  2. Configure the MTA to sign messages as they egress the system.
  3. Configure a DNS record with the public key information so remote systems can verify the signature on a piece of message.

This is a rather in depth process, but one that rspamd makes very accessible. In fact, really simple. First, I generated a DKIM key for each domain using the rspamadm utility. The three bits of detail we need to provide are the domain (iris.com), the RSA key size and the selector. The selector is just an identifier and can be anything, for the purposes of my setup I used sig1 as the selector:

[root@mx-iris ~]# rspamadm dkim_keygen -d iris.com -k sig1 -b 2048
mail._domainkey IN TXT ( "v=DKIM1; k=rsa; "
        "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2DtOYvGbJRugjJ/WbGzbxHyqXG6duvlhUeBhU/hOn16bTABlcamwKzV/Tmcv84YgxyaCI7NXJBOYn87i1bzJwfnosPD55gDrdYZ/meqKBYN7a1brrCC3V8piQ3fjoeR3ejfbR9Rd/OBYSAnIKahSHqigKRWTo9kdBe5LIeDPrgGV98pEXmQpoauLqFy5fEqMZEoSHJOkiPGGI3+LK"
        "t0yLjOjqzcEHh0tWdH2T8Z7lrZY4ugMXcLN0vtTng/pa49tS+q4RPa27B/dw8qRiKKR5edv2eSoVQWVMJRY/58r7316NVeqwEHqo8JR6bgR0V2/NjCFZ+sAo63X6sFDa4E32QIDAQAB"
) ;

With the key generated, it’s then important to ensure the sig1._domainkey.iris.com TXT DNS record is configured correctly. For example:

[root@mx-iris ~]# dig +short sig1._domainkey.iris.com TXT
"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2DtOYvGbJRugjJ/WbGzbxHyqXG6duvlhUeBhU/hOn16bTABlcamwKzV/Tmcv84YgxyaCI7NXJBOYn87i1bzJwfnosPD55gDrdYZ/meqKBYN7a1brrCC3V8piQ3fjoeR3ejfbR9Rd/OBYSAnIKahSHqigKRWTo9kdBe5LIeDPrgGV98pEXmQpoauLqFy5fEqMZEoSHJOkiPGGI3+LK" "t0yLjOjqzcEHh0tWdH2T8Z7lrZY4ugMXcLN0vtTng/pa49tS+q4RPa27B/dw8qRiKKR5edv2eSoVQWVMJRY/58r7316NVeqwEHqo8JR6bgR0V2/NjCFZ+sAo63X6sFDa4E32QIDAQAB"

Two further notes on DKIM. The first, you will note that the DNS record is split in two. Like SMTP, DNS is an extremely old protocol and when developed it was envisioned that it would not have to store more than 255 bytes in a TXT record. This doesn’t work well with lengthy public keys, so the clever solution was to split the records and reassemble when required for signing validation.

Secondly, it should be noted that I deliberately have not automated with Ansible. DKIM is finicky at the best of times and I believe it is a task best done by hand to ensure clean key generation, public key capture, DNS record creation and ultimately timely testing of any changes with third-party systems.

The last step is to configure Postfix to pass messages to the rspam provided milter upon message recept via SMTP. To do that, I added the following configuration directives to the Postfix main.cf:

# Configure rspamd for spam checking 
smtpd_milters = inet:localhost:11332
non_smtpd_milters = $smtpd_milters
milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen}
milter_protocol = 6
"I've been hosting email on our friends+family server for twenty-three years, and agree, it's not for the faint of heart." -- /u/ttkciar

At this point things were looking pretty good. I had a mail server providing MTA and MDA services for virtual users. Inbound and outbound message delivery was working. Rspamd was checking inbound messages with a variety of anti-spam and anti-phishing functions as well as signing our messages for DKIM on the way out the door.

Not bad for a $5/month VPS.

As happy as I was, I still had one final hardening step to undertake - brute force mitigation. Whilst there is no webmail access to target with phishing, any place where authentication occurs is a great place to undertake brute force attacks to guess a user’s password.

Fail2ban22 is a free and open source project which enables brute force attack mitigation via log monitoring. In short, it monitors log files for ongoing authentication attacks and when it detects one, it blocks the attacking IP by leveraging IPTables firewalls. To leverage fail2ban I had ansible create two new configuration files:

jail.d/01-dovecot.conf

[dovecot]
enabled = true
port    = imap,imaps
filter  = dovecot
logpath = /var/log/maillog
maxretry = 5

jail.d/01-postfix.conf

[postfix]
enabled = true
port    = smtp,ssmtp
filter  = postfix
logpath = /var/log/maillog
maxretry = 5

The server has been in operation with ‘production’ workloads for a few weeks now and in general, it is performing very well.

The choice of Fedora has applied SELinux contexts to our key processes by default (Dovecot, Postfix, MariaDB, Rspamd) and I have basic firewalling and are leveraging fail2ban to prevent brute forcing of user accounts via our SMTP and IMAP services.

Spam mitigation appears to be effective while still being trained to tackle some of the harder edge cases and messages from the server are successfully arriving in inboxes on services such as Gmail, iCloud, ProtonMail and M365.

Finally, the majority of the solution (with exception to DKIM signing) implemented as code via Ansible which makes the solution reproducible in the event of catastrophic VPS issues.

At the start of this essay I said I wanted the following capability:

Build a self-hosted email server that provides mail services which can reliably deliver emails to the major providers (Gmail, Proton, iCloud, M365) without being canned like spam.

I am very happy to report that I met this objective and in a reasonably stable state.

I feel compelled to point out that the purpose of this essay has not been to convince you to self-host email, nor convert you to unsupported open source platforms due to idealism. It was simply to see if it is achievable for those who want to give it a go for their own reasons and have some fun along the way.

If you do decide to pick up the challange of self-hosting email then good luck on your self-hosting journey! Please be sure to let me know if you found this essay helpful at all in your adventures.

"Yes, it's possible. But it's not very practical." -- /u/swayuser

Updates

  • 2023-11-30: This hit the front page of Hacker News! Thank you for all the comments and feedback.
  • 2023-12-03: Updated with the PTR note.

Acknowledgements

I would like to thank all those (Vanessa, Justin, Aaron and Mark) who took the time to review an early pass of this essay. It’s a lot of work for a niche project so I really appreciate it.

To the /r/selfhosted community: stay curious! While often seen as a silly or time wasteful effort, I believe that the importance of open systems matters and find your projects inspirational. Taking control of your own data, learning and playing is powerful and should be celebrated. Keep fighting the good fight.

To Jeff Geerling for his excellent book.

Finally, to the open source developers, writers and community members for all the software mentioned in this essay (Fedora, Postfix, Dovecot, MariaDB, Rspamd, fail2ban, SELinux) my heartfelt thank you for keeping software open.

Footnotes