How many times have you faced a technical challenge only to solve it by following someone’s tutorial? I’m sure many of us have done it. I know I have.
That’s all well and good when you’re trying to figure out how to do something personal. But how do you know if the instructions are right? I’ll tell you a secret, not all of them are correct and others are mostly correct but have some suggestions that can cause problems.
Understanding technology means not having to blindly follow technical recommendations without really understanding everything you’re doing. You can actively decide what you’re doing and make the most appropriate decisions for your marketing and your business.
Deliverability Basics
There are two primary components to getting mail to the inbox: sending mail the user wants and expects and sending mail in a technically compliant manner. Both conditions are necessary to reach the inbox, but neither on their own are sufficient,
Many email marketers rely on their ESPs, technical staff and deliverability consultants to handle all the technical bits of email. But they often have their own biases and points of view that aren’t clear to the marketer.
On the ESP end, most of their guidance for SPF and DKIM are designed to be easy to implement and get customers up and running as soon as possible. This works for most senders and doesn’t cause delivery problems. But quick deployment isn’t always the best for deliverability. Some of the better ways to authenticate email require coordination between the ESP and the folks handling the corporate domain.
Think about it, marketers spend hours thinking up copy and testing subject lines and colors. They fret about the text to image ratio and responsive design. And then they let their ESP drive authentication.
Much of what I try and do is demystify the technical end of email. Give non-technical folks a framework to work with technical folks. I often describe myself as a translator from geek to marketer and back. I believe marketers should have enough of an understanding of the technical underpinning of email to be able to make informed decisions about mail.
Basics of MailStructure
Understanding authentication and technical sending of email starts with the email itself.
Email takes a lot of its structure from postal mail. Every email has an envelope, with a from and a to address. The envelope from is the address that mail is returned to if it’s undeliverable. Email has the same thing, an envelope from.
Inside the envelope is the message. This message also has a to and a from address. These can be different, similar to or identical to the addresses on the envelope.
Email is a little more complicated than postal mail because we aren’t just transmitting a message, we also transmitting information about the message. Inside the envelope we have two parts the body of the message and information about the message.
The body of the message is what the user sees. It’s not that vital for deliverability, so we’re not going to focus on that.
Most of what is hidden from the user is in the headers. They contain information about the email, where came from, what systems handled it, how it was authenticated and so on. Every server that handles a message can add to the headers.
- Headers can be added by the sending server. These include the envelope from, list unsubscribe links, subject lines, identifiers and on and on.
- Headers can be added by the receiving server.These include the authentication results, spam scores and internal forwarding and handling.
- Headers can be added by intermediary servers. Often these are simply received lines.
Authentication
When email was initially invented, most of the people on the internet got their access through work or school. There was a social contract that users would behave in certain ways. This meant the email protocol did not incorporate a lot of security. People who did bad things would be disciplined by their administrators. As internet access spread to more and more people, there were more opportunities for malicious behavior and less control.
Something needed to be done to authenticate email.
Over the last decade a number of technologies have been tried to improve the security of email. Three of them are in common use today: SPF, DKIM, DMARC. All of the information needed to authenticate these messages is contained in two places: the email headers and DNS.
From the email headers we need a few pieces of information.
- The IP address the mail came from - this is added by the receiving server.
- The DKIM signature (specifically the d= and i= value) - this is added by the sending server
- The Envelope from - this is communicated by the sending server and recorded by the receiving server.
- The From in the body of the message - this is added by the sending server.
To authenticate with SPF we take the Envelope From address, do a DNS lookup on it and get a list of IP addresses. These IP addresses are compared to the IP address that sent the message. If the sending IP is in the list of IP addresses, then the message is SPF authenticated.
To authenticate with DKIM, we take the d= value in the signature, and do a DNS lookup for a public key. We then use that public key to check the email. If the check validates, the message is DKIM authenticated.
DMARC compares the domain from in the body of the message to the Envelope From domain and the d= value. If the domains match, then the mail is considered DMARC aligned. If either SPF or DKIM authenticate, then the message is DMARC authenticated.
So there are 3 different kinds of authentication: SPF, DKIM and DMARC. All of these methods of authentication rely on DNS.
DNS
DNS really is the center of the internet. It is a way to broadcast information about a hostname. This information can be an IP address, but there are a lot of other things we’re using DNS for these days, including providing domain ownership and authenticating email.
When we’re talking email authentication, the information we’re looking for contained in TXT records. These records are published by the domain owner and can be queried by anyone from anywhere.
Each type of record has it’s own syntax and conventions. This lets the checking server distinguish between a SPF record and a DKIM record and other types of records.
A blog post is a little too short to go into as much detail as I’d like about the different records, what the different parts of the records mean and how to use them, so I’ll just provide a few rules to remember.
SPF:
- SPF authenticates the envelope from address. You do not need to publish SPF for the message from.
- The spec has a limit of 10 DNS lookups. If you have too many non-IP addresses in your record (A, MX, PTR, and include:) then you may have unexpected SPF failures.
- Don’t use ?all, use either ~all or -all.
- Regularly check your SPF record for errors.
DKIM
- You don’t need to sign with DKIM and DomainKeys. DomainKeys was replaced by DKIM so you can just sign with DKIM.
- If you want different reputations for different mail streams, sign with different subdomains in the d= (transaction.example.com, marketing.example.com).
- You should be regularly rotating DKIM keys.
DMARC:
- DMARC has two independent processes.
- Reporting: receivers send you email when ever an email fails DMARC authentication
- Policy requests: you can request a receiver handle mail that fails authentication in a particular way.
- You should monitor DMARC reports before making policy statements to verify your mail is all correctly authenticating.
- Turning on a policy of reject may, and often does, cause legitimate mail to be rejected.
- Not every service provider can support DMARC as they don’t have the ability to make the d= or the Envelope From match the from domain
Authentication in general
- Authenticate with both SPF and DKIM. Because they use different mechanisms to authenticate mail it’s unlikely that both will fail at the same time.
- Some ISPs are checking for authentication alignment even in the absence of a DMARC record. Aligned mail is better than unaligned mail.
- Hacked DNS records can let a malicious actor take your reputation. Talk to your IT department to use DNSSec.
- Turning on DMARC affects more than just your bulk mail, it affects all mail coming from your domain. Deploy DMARC policy requests with care or risk deliverability problems.
- Most receivers request but do not require authentication for mail sent over IPv4.
- Most receivers require authentication for mail sent over IPv6.
That is by no means all the ins and outs of authentication. I could do a full day course on authentication and the ins and outs and how to set up your records and verify them. These hints and tips should help you at least understand the technical end of authentication well enough to double check the recommendations you get from your ESPs and technical team.
For more insights into understanding more about the technical end of email, about authentication come see me speak at the Innovations conference next week.