The approach Mastodon has gone with is to have each instance generate link previews for each incoming link. This means that while having a link shared on Twitter or Facebook might give you one automated pageview for the preview, on Mastodon it gives you one for each instance the link is federated to. For a link shared by a widely-followed account this might mean thousands of link-preview requests hitting a server, and I'd also now consider this traffic to be from an automated agent.
I'm not sure what the right approach is for Mastodon. Making the link preview fetcher respect robots.txt (Issue 21738) would be a good start. [1] Longer term I think including link previews when composing a post and handling abuse with defederation seems like it should work as well as the rest of Mastodon's abuse handling.
From the blog post: