We're having problems with link previews / rich media and there's nothing we can do about it. I suspect Mastodon instances have the same problem but just haven't noticed yet.

Originally we had issues with link previews to social media sites like Twitter, Facebook, etc. The OpenGraph/TwitterCard meta tags were not there unless you had 'Bot' in your User Agent string or in some cases you literally needed 'Twitterbot' in there. This is how Telegram works actually -- their User Agent is 'Telegrambot (like Twitterbot)'.

This doesn't work anymore. I see a ton of 403s to normal websites.

I try with curl, change user agent, etc. Always a 403. The error message says you need to enable javascript and disable your ad blocker. It also returns an HTTP header that indicates they think you're a web scraper.

But how does this work for Telegram etc then? Well, their IP ranges must be on a whitelist at the CDN level.

I've confirmed Fastly and CloudFlare are doing this but I don't know how it works. It's probably an opt-in anti-abuse feature that companies are enabling.