✧✦Catherine✦✧
there exist several pieces of folk wisdom:
- "you cannot run your own mail server in 2025, this is too hard and time consuming" (completely false, i've done this since ~2010 with minimal ongoing maintenance)
- "you can do it but gmail will sort your mail to spam" (partially true and what i want to talk about here)
recently, my hand was forced: i had to migrate my mail server across providers and regions. it's unimportant why but important what the result is.
1/2
✧✦Catherine✦✧
in the end, i went:
- from running mail.whitequark.org on DigitalOcean (extremely poor reputation of every IP block) to Hetzner Cloud (coin flip, but you can get clean IPs with minimal effort)
- from running roundcube+postfix+dovecot on ansible+Debian (ongoing suffering) with spamassassin to NixOS with rspamd (one-time suffering)
in the end i have clean IP reptuation, SPF pass, DKIM pass, DMARC pass. does gmail accept my email? yup, cold emailing people just works. with everyone but M365.
2/3
✧✦Catherine✦✧
by setting the entire thing (https://codeberg.org/whitequark/mail.whitequark.org) up from scratch on a never-seen-before IP using a completely fresh server in about a day on new-to-me NixOS, i want to put the folk wisdom of "you can't host email and even if you did gmail won't accept it" to rest. evidently, it can be done.
Office 365 however apparently will hate your email no matter what. you learn to live with it
3/3
✧✦Catherine✦✧
_should_ you self-host your email? i dunno, i'm not your mom, pick your own poison. i do it because i need independence: throughout most of my life i would had been, for various reasons, under significant risk of losing control of corporate hosted email in one way or another. doing it myself solves that.
_can_ you self-host your email? Absolutely.
✧✦Catherine✦✧
i think the myth of "you can't self-host email" persists because while it can evidently be done, basically all of the software involved is ancient, baroque, inconsistently documented, requires a PhD in Bullshit to correctly configure, and is almost actively hostile to observation. but this is _annoying_, very different from _impossible_, and fortunately mostly solvable by delegating the annoying bits to an expert using something like NixOS to make it reliably work
David Chisnall (*Now with 50% more sarcasm!*)
I wonder how much this worked because you had a clean reputation for the domain. I’ve been using the same domain for my primary email for 20+ years, so by now the big providers have it on their ‘probably not a spammer’ list. When I added some new domains. They used the same DKIM key and same sender IP, which probably helped. I didn’t have any problems with sending to Google or MS domains (when I was at MS, I tried sending a mail from my mail server pretending to be from a microsoft.com email address and it arrived too!).
But I’ve heard people setting up a new mail server have a much worse experience.
✧✦Catherine✦✧
@david_chisnall this is possible! I haven't done a proper study of the factors that go into it so I can't claim otherwise. but, anecdotally:
- until I cleaned up my IP reputation (which has been awful for almost a decade) Gmail refused to deliver to anything but spam
- I always had clean SPF but never bothered with DKIM or DMARC
- I kept sending email under these conditions for years
✧✦Catherine✦✧
@david_chisnall given this, what kind of reputation would I have? clearly not good enough to mark my email as ham; which makes it hard to believe it was somehow enough to result in a near-180 turn in deliverability after just a day? I'm skeptical
✧✦Catherine✦✧
@futuresprog they do actually deliver it after someone from the org marks it as non-spam (and, especially, replies to it)
Future Sprog
Confirmed: Microsoft won’t even deliver email you want to receive!
✧✦Catherine✦✧
@doragasu I do not have professional ops background either, I just dabble really
doragasu
@whitequark Awesome, I have wanted to do that since forever. Unfortunately I don't have that much devops knowledge and always read that my emails would always go to spam, so in the end never tried. Maybe this is what I needed to read to finally give it a try.
✧✦Catherine✦✧
@futuresprog yep I believe so, assuming I understood your terminology right
Future Sprog
365 tenants are keeping their own known domains list?! Well that’s an interesting insight.
Public Outlook/Hotmail would send even well formed, with DKIM and SPF, and solicited email to Spam all the time
Glyph
@whitequark another reason that it persists is that even the public IP reputation scores are inscrutable as hell, and the secret bonus reputation scores that Microsoft, Google, Apple, and Yahoo all keep internally are even worse. As you are apparently discovering with Microsoft right now (sorry).
✧✦Catherine✦✧
@glyph yeah I think the combo of "roll the IP gacha a few times" + "let it sit for 8 months while the VM idles" probably did me a lot of good here, though I can't prove it!
Glyph
@whitequark yeah, the spooky magic parts suck.
HOWEVER.
while I would not encourage most people to do this, I also wouldn't want to unnecessarily scaremonger about the process, so there is a very important coda to this:
many people see the undocumented complexity, want to "keep things simple", and thus avoid DKIM, SPF, DMARC, and SMTP/TLS.
The key is to set up all the complicated shit on day 1. You can't control IP reputation but you CAN control these "security" markers, and they all help
✧✦Catherine✦✧
something interesting I discovered is that the combination of
- greylisting (telling senders that are suspicious but not obviously spammers to retry in a few minutes)
- rejecting non-well-formed HELO hostnames
- rejecting clients that pipeline message data without even listening to your greeting
has reduced the amount of spam i receive (at the MTA) to essentially zero. i got one single message since i set this up. normally i get ~50 per day!
✧✦Catherine✦✧
these functions are now either built-in to postfix (smtpd_helo_restrictions, smtpd_data_restrictions) or come built-in to rspamd, so are very cheap to set up; I'm amazed that I receive all legitimate mail (personal and transactional) instantly but junk just stops coming. I almost felt like setting up DNSBL and Bayesian filtering was a waste of time
Glyph
@whitequark the more you have this stuff consistently running clean (and don't get out over your skis with overly harsh SPF policies, especially if you ever forward mail anywhere; just *have* a policy, don't worry about making it strict) the more you are likely to be considered clean early on in the process.
(Ironically, the other thing that you want to do is… maintain a certain level of sender volume. So ideally do this for you and 100 friends not just yourself.)
✧✦Catherine✦✧
@glyph also (a note to past catherine), DKIM doesn't involve regular key rotation so setting it up is way easier than say Let's Encrypt; you run a single CLI command, update DNS and kinda forget about it forever. or until your key gets compromised at least lol
Volker Stolz
@whitequark Tagging this as #ryoms.
✧✦Catherine✦✧
@ben I managed to divine out just enough to know who and when to ping via other channels to get mail through; I don't think I'll ever get it to go through when it's unsolicited
Ben Zucker 🍰
@whitequark
Just like many others already mentioned: don't waste your time with Microsoft and emails. You will most likely end up getting insane or with a major headache at least.
✧✦Catherine✦✧
@yannsionneau I was migrating an existing server and wanted to keep it roughly 1:1
Yann Sionneau
@whitequark about "ancient baroque undocumented" software. Have you tried opensmtpd?
I kind of like it, it's more modern than postfix definitely.
But you have then to learn yet another config if you wanna migrate to it.
✧✦Catherine✦✧
also Roundcube is pretty amazing, they have a nice responsive new theme that works well on desktop and mobile and is as nice to use as the Thunderbird Android app if not more
one of the few PHP codebases i'd run in production
✧✦Catherine✦✧
@bert_hubert try me? whitequark@whitequark.org
bert hubert 🇺🇦🇪🇺🇺🇦
@whitequark you'd miss out on my email though, depending on your definition of "non-well-formed HELO messages'.
dram🎀
@whitequark as both a free outlook email user and an (rarely, but sometimes) organization m365 email user, i can confirm that i just blame microsoft for throwing anything unfamiliar into spam and learn to live with it,
the september discord *tos update* email was sent to my spam folder. i'm pretty sure there are anecdotes out there of microsoft sending their own notices into spam too.
Russell
@whitequark I recently went through a similar change. My old mail server was getting long in the tooth and, rather than upgrade the distro and potentially break all the configs in subtle ways I've forgotten, I searched for mail servers written in Rust and switched to Stalwart. It's been ... fine? ¯\_(ツ)_/¯ They make setting up all the things fairly easy (spf, dkim, dmarc, tls) with decent docs so I'm hoping future version upgrades will also be simple and painless.
John Regan
@whitequark @david_chisnall > - I always had clean SPF but never bothered with DKIM or DMARC
This is something I see really often, and I think nowadays DKIM and DMARC are more of a requirement.
I see some organizations implement DMARC reject policies with SPF as their only mechanism. I don't think people realize how many mail forwarding services are out there that will always break SPF, DKIM is really crucial.
✧✦Catherine✦✧
@bert_hubert received!
bert hubert 🇺🇦🇪🇺🇺🇦
@whitequark done
ocdtrekkie
@whitequark For the second largest email provider, Microsoft appears to be deeply incompetent at handling mail. From my experience Outlook autoresponders from 365 accounts *always* fail DMARC and most annoyingly, Microsoft doesn't handle error messages correctly: Your server sends a standard "message size exceeded" and Microsoft tells the sender it was rejected as spam.
✧✦Catherine✦✧
@jernej__s @jpm oh that's super clever
Joel Michael
@whitequark oh hell yes these 3 SMTP tricks definitely cut out about 90% of spam. One more I do is to wait 5 seconds between SMTP client commands and server responses including the initial HELO because that will catch even more pipeliners while not affecting delivery of legitimate email at all.
Jernej Simončič �
@jpm @whitequark Right, I've got this implemented, too.
permit_mynetworks, sleep 2, permit_sasl_authenticated, sleep 2, reject_invalid_helo_hostname, check_policy_service inet:127.0.0.1:10031, permit
✧✦Catherine✦✧
@twipped yup! I run too much infra to have capacity for this but other people can step in!
Jocelynephiliac :reclaimer:
@whitequark seems like a good opportunity for a docker image with a solid config readme.
✧✦Catherine✦✧
@twipped I'm exaggerating a bit; I sat on the IP for 8 months before setting up the server (for reason of being tired)
Jocelynephiliac :reclaimer:
@whitequark there are IPs that have never been used before?
✧✦Catherine✦✧
@HeyQui I greylist only email that is suspicious in first place (via rspamd); tokens and stuff go right thru
Henry Isoppo ◕‿◕✿ | ⭐
@whitequark 💞greylisting. It cuts down on so much spam it's like witchcraft.
A side effect that drives me nuts tho is 2FA via email.
It seems that 2/5 minutes tokens are becoming more common-place with each passing day. These stupid a*holes treat email as if it was instant messaging.
Sometimes I need to request 3 tokens to get a valid one to go through grey in time
My brother in Christ you're not a well known service and your tokens don't even come from your own domain. Chill the f down.
✧✦Catherine✦✧
@zimzat let's test deliverability?
✧✦Catherine✦✧
i'm seeing that a lot of people know about greylisting but have various issues stemming from the added delay
my setup doesn't have this problem! rather than greylist at the MTA (Postfix), i greylist at the filter (rspamd); only email that is suspicious in first place gets greylisted, while email that's "known good" or "50/50" goes through
this doesn't noticeably reduce the efficiency of greylisting, but makes the mail server wayy more usable, especially with "magic login links"
defer
@whitequark I've been running pretty much that exact setup for over a decade, and moving it to nix like that for a declarative config sure looks appealing. I've only played with nix a little, so reading a more complex config was educational. I'm definitely stealing that "load the config variables from a TOML file" bit next time I touch it.
✧✦Catherine✦✧
@defer I don't really like the Nix language (while fully appreciating its sheer power!) so I design my systems in a way where merely operating them doesn't need editing Nix files. it also makes for a very nice conceptual split
✧✦Catherine✦✧
@jernej__s @jpm which section do you put these commands in?
René Seindal
There are several shrink wrapped FOSS mail server packages around, which will do all the heavy lifting.
I use Mail-in-a-box for my mail server, and I spend less than an hour a week keeping in going.
✧✦Catherine✦✧
@seindal a hour a week seems _incredibly high_
René Seindal
Less than ...
Also, I use it for work.
✧✦Catherine✦✧
@hallunke23 highly recommend greylisting
hallunke23 🇺🇦:nona:
I have all of this and my spam detection rate is way lower than yours. HELO catches some of my spam, pipelining catches nothing these days because all of my spammers seem to know they're not supposed to pipeline. For greylisting I'm not sure how effective it is.
Looks like I have to implement additional anti-spam defences like DNSBL or SPF.
✧✦Catherine✦✧
@Dragon this is exactly why I use NixOS, which lets me restore a system from a backup in around 10 minutes
Dragon
@whitequark I used to do it until the raid controller in the machine that was hosting it on shat a brick (Technical term) and it took me several days to get the provider to replace the faulty hardware (It was a dedicated server) and get everything restored from backups. Particularly as I couldn’t focus on it full time due to needing to do the dayjob
it was at that point I decided it wasn’t worth it for me.
✧✦Catherine✦✧
@Dragon right, yeah
i used to be way more conservative about putting everything on one single machine until the monthly cost of running a server didn't get comparable to monthly cost of renting an IPv4
Dragon
@whitequark The problem was more the provider being slow replacing the server.
Thesedays I’d just spin up another VM, this was quite a few years ago where “cloud” was less common.
Also for personal stuff operating at the budget end of the market didn’t help.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.