Understanding SPF, DKIM, and DMARC: A Simple Guide

I manage IT at a mid-size business. At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated and doesn't include all the email services they actually use. (What this really tells me is how many small businesses are out there running with no in-house IT expertise or support of any kind.)

I don't do whitelisting. Instead, I always reach out and offer to help the other party correct their SPF record.

It happens often enough that I wrote a script in Racket that will generate the email for me and paste it into the clipboard [1]. The email tells them exactly what they need to change, and links to docs from their current email provider (so they don't have to trust me about edits to their DNS).

[1]: https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72e...

Neat! I must try this.

What would you say the normal reception you receive from this email template is?

I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.

Well if you read the template, you'll see I start out with a non-technical explanation, advise that they forward the email to an IT type person, and offer to help in any way I can. Then I put in a "More info" heading further down with all the details and instructions.

Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.

That is really awesome. It can be easy to miss setting up SPF on every new tool.

>Well if you read the template

I read the template, that's how I spotted the acronyms that weren't spelled out. Like DNS on the second line, before you recommend forwarding.

>Overall I'm pleased with how well this approach works.

Interesting, I definitely would have thought it'd be ignored more often than not, but I might have to look into rolling out something similar. Thanks for the idea.

I think you're being a bit unfair here.

> If you do not have access to your company’s DNS records, please forward this email to someone in an IT role.

If you don't even know what DNS records are, I'd imagine you'd assume you don't have access to them and so forward them to the IT person as suggested. But sure, maybe he could also add ", or don't know what they are" to this line.

I'm not trying to be unfair or critical or anything. My first question was genuine. I intended my note about acronyms to be just that: a side note. The response I got was "If you read the template [...]", which it should have been pretty obvious I did. Then I got an explanation of the template as if I hadn't read it, which was a bit patronizing.

I think it's a good idea (and said so twice!). I was curious what the reception was like.

Sorry if my comment about acronyms was too much. It is a pet peeve of mine to see acronyms not spelled out, especially technical ones in a document intended for non-technical people. I didn't intend it to derail the conversation. Obviously it was taken in a way more critical way than I had intended -- my fault.

Sorry I didn’t pick up that you had read the template. I was just trying to give context for my answer without assuming or requiring anyone who might read it to have scrolled through all the code.

Cool stuff! BTW, what's up with the license?

Including the whole license in the gist was probably overkill, but I particularly like Blue Oak for permissive licensing and would like to see it more widely used.

https://writing.kemitchell.com/2019/03/09/Deprecation-Notice...

Hey, English is not my first language so I'm surely missing something here but:

> Copyright

> Each contributor licenses you to do everything with this software that would otherwise infringe that contributor's copyright in it.

This sounds like the license specifically allows you to infringe on the contributor's copyright.

> If anyone notifies you in writing that you have not complied with [Notices](#notices), you can keep your license by taking all practical steps to comply within 30 days after the notice. If you do not do so, your license ends immediately.

I'm not sure I like it. Like, what if the notification of notice was incorrect? You lose your license anyway?

Does the script handle macros in SPF?

I've had a couple of other-company-IT-admins tell me that my MX is jacked because I use hosted SPF via proofpoint, and when they look up my SPF it looks like this:

"v=spf1 include:%{ir}.%{v}.%{d}.spf.has.pphosted.com ~all"

A surprising number of mail admins don't understand SPF macros.

I was on the receiving end of an automated version of this. However, when I looked into it, it seemed the problem (SPF record required more than 10 DNS lookups) was fairly common. I don't seem to have any other deliverability problems and my email and DNS is managed by some big hosting company so I assume it's not a real problem and didn't fix it.

It's a permissive license. The licensor is saying “I own copyright in this work, which gives me the right to give you permission to do whatever you want with it.”

If I’m reaching for the script, it's because I’m already in a scenario where Proofpoint has quarantined legitimate email for failing SPF checks (we use Proofpoint too). So the script itself doesn’t do any analysis of the existing SPF record. It just shows them the existing record and tells them how to fix it based on the sender's IP for the email in question.

By default under most open source licenses, particularly permissive ones, violating the terms even by accident ends your license instantly, with no notice whatsoever.

Including a fair, common sense path to forgiveness severely limits legal risk for users, and is one of the things I like about the Blue Oak license.

It’s a real problem. The solution is just to isolate each service on its own subdomain. The only thing that should be listed in your top level SPF record is the corporate email for the domain.

In defense of those who haven’t read the RFCs personally — I can count on one hand the number of times I’ve seen SPF macros in the wild, which holds true if I included yours.

Interestingly all Proofpoint customers too.

I’ve seen it more common to isolate services to subdomains and specify subdomain SPF records rather than use macros. This is my preferred approach.

I’m not hating on the macros. They’re just seemingly very rarely used. I know they’re on the table but I haven’t found a compelling use case in my own deployments.

Awesome, mind if I send you an email solely to test if it gets through or if I get to be the recipient of your awesome script? ;-)

> At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated

And at the same time, I regularly get Spam/Phishing with perfect SPF, DKIM, DMARC, etc. The domains and IPs they use might get blocked within a day, but of course, these people have no problem getting others.

And although I have set up my MTA perfectly, my mail gets refused by MS/t-online/etc., because I don't have enough "sender reputation". In e-mail, we have an oligopoly of a few big mail providers, and in the end, they decide which mail gets delivered and which isn't, and to me it looks like they give a rat's ass about SPF and DKIM, and probably rightfully so, because most spammers are probably better at configuring MTAs than your average mail admin.

t-online uses a global whitelist, which is pretty stupid for e-mail. sometimes it helps contacting them, other times they refuse to resolve it for arbitrary reasons (not because of actual spamming)

t-online told me I needed an imprint on the website that's reachable under my domain. Seems to be some misunderstanding of German law (German commercial websites need an imprint, legally, but t-online also apply this requirement to private domains).

Last month they unblocked me even though the website is blank.

I work at an IT provider - we see this daily. Have to whitelist to keep the customer happy, usually the other end is a 1 or 2 person business with an old hosted Yahoo system or similar

> happy

or misguided?

(I know the pain.)

> ignored in most cases

I see no problem here.

Is it osi approved?

If we care about keeping open protocols like HTTP and SMTP alive, we need to overhaul DNS.

Or at least create a simplified common abstraction layer.

It’s the most inherently user-hostile thing I’ve ever encountered - and I’m only just now starting to understand it, even though I’m almost 20 years into dealing with it.

My favourite are queries "why are you rejecting my email?"

Ehh... Because your dmarc policy told us to?

On some level I can see it being a benefit to the big providers to only accept email from other big providers, as it would incentivise people to buy email services from them, because only email sent from the big providers would "work".

It is more complicated than that. There are more criteria for when you need an imprint:

(1) any kind of journalistic content on your site

(2) any kind of financial gain from showing ads or making ads

(3) organizing any kind of group of people active on German territory

(4) running a business website

There might be more, but those are the ones I remember from reading the paragraphs a while ago.

And these are, of course, vague, which means that even something like "my favorite restaurants in Berlin" could be considered an ad, or any kind of comment on politics might be considered a form of journalism.

I dislike these rules, because they basically kill German blogging scene. Not so many people want to run a blog and have every idiot on the Internet know their personal address. And few bloggers want to rent a digital office or actual office, that will send mail to them (an indirection). The German law in this respect is terrible and working against a free Internet and against freedom of voicing your opinion. It works greatly in favor for tech giants, because people resort to putting their blogging on Facebook, Instagram and other disservices. It is very anti-decentralization.

How do I make use of the gist?

I do

    (load "spf-fail.rkt")

But afterwards none of the definitions are available in the Racket REPL. Maybe I need to (require ...) something? I do not see a module definition, that I would need to import and

    (require spf-fail)

fails.

T-Online has a simple whitelist approach, and it is usually enough to just drop them an email. I did that back in 2014, asking for my private mail server to be added to that whitelist, and I received a positive answer within a few hours.

Maybe that was possible 10 years go, they now require that you put up a web page for your domain with a valid German imprint (most importantly: your full contact information).

To be fair, I had that in place 10 years ago (and if you already have a mailserver, it's trivial (as in: MUCH easier than to set up a mail server) to host a small imprinted HTML page).

Yeah, the only thing you need for simple SMTP services is:

- with DNS, the simplest as possible SPF record.

- Without DNS, (aka pure IPv[46] SMTP), your have implicit SPF: instead of querying the DNS and parsing the SPF record, you parse the mail header to check "reply-to/from/etc" fields (the appropriate fields) for the sending SMTP IPv[46] address, that to perform spam scoring.

If you’ve just run "racket" from the command line to get a REPL, you would use

    (require "spf-fail.rkt")

to import all the bindings from this file (assuming it's in the same folder you ran racket from). The “Module Basics” chapter of the Racket is a good quick explainer of how this stuff works in Racket: https://docs.racket-lang.org/guide/module-basics.html

Another method would be to open the file in DrRacket (or VSCode or Emacs or whatever editor you have set up with a Racket plugin/lang server) and just "run" it in the REPL.

There are comments at the top that explain how to use it once it's loaded:

    ;; Generate a form email to let someone know their SPF records are misconfigured for their current email provider.
    ;;
    ;; Run (fill-report "domain.com" "1.2.3.4") where the 2nd arg is the sending email server's IP address.
    ;; It will copy the completed report to the clipboard for you.
    ;;
    ;; Only works on Windows for now.

Personally, I have this file incorporated into a larger package (not published anywhere) for producing canned responses. With that package installed I can do this at a command prompt:

    raco canned spf domain.com 1.2.3.4

Sure, I think?

The trouble is, it's full of stuff I don't understand because I'm not managing it, such as: include:_spf.qualtrics.com include:_spf.salesforce.com include:sparkpostmail.com include:spf.mailjet.com include:spf.protection.outlook.com include:_spf.myorderbox.com include:eig.spf .a.cloudfilter.net include:spf.websitewelcome.com

> most importantly: your full contact information

Note that unless T-Online has additional requirements here this doesn't need to be your home address but only a valid mail address through which you can be reached.

> have every idiot on the Internet know their personal address

It doesn't have to be your personal address though.

I'm somewhat mixed about the details of the law, but requiring businesses to make it clear who you are dealing with makes sense to me.

In Germany, you might be required by law to publish your address for everyone to see.

Original German text of the law: https://www.gesetze-im-internet.de/ddg/__5.html

Google translation to English: https://www-gesetze--im--internet-de.translate.goog/ddg/__5....

Of course, if you run a business. But then you might even have an actual physical office address for work related stuff and not have a problem giving that to random strangers on the Internet. If you are a private person simply wanting to run a blog and talk about whatever you like, then it sucks. It does not have to be your home address you are sharing, but the other options will cost you money. And no, a postbox will not be sufficient, as decided in previous court cases. It must be a "ladungsfaehige" address. There are some businesses selling services to have a virtual office, with an actual address, which then send your mail further to your actual address and notify you via e-mail and all that. Whether those are really an acceptable option when it comes to the law is a bit unclear.

Yep, each one of those should be isolated to their own subdomain rather than being in the top level SPF record. Should have 8 total SPF records including the top level.

Easiest thing to do is to start by moving just 1 include to its own subdomain.