Sounds to me like this is not about robots.txt. you can DM me if you want. Anyway you don't have to add one. You can, it won't hurt. Don't overthink it.
Sounds to me like this is not about robots.txt. you can DM me if you want.
Anyway you don't have to add one. You can, it won't hurt. Don't overthink it.
robots.txt is definitely optional but search engines do give a shit. Not from a ranking perspective, just that the majority of bots honor it. It can be really useful to reduce excess hits from...
robots.txt is definitely optional but search engines do give a shit. Not from a ranking perspective, just that the majority of bots honor it. It can be really useful to reduce excess hits from misbehaving bots. No real harm in not using it until you need it though.
SEO is a rabbit hole of diminishing returns and increasing madness that you don't need to stress about when the majority of your traffic is from a subreddit.
SEO is a rabbit hole of diminishing returns and increasing madness that you don't need to stress about when the majority of your traffic is from a subreddit.
Good search engines do give a shit but it's more of a polite hint than obligation.
What are you trying to achieve?
Sounds to me like this is not about robots.txt. you can DM me if you want.
Anyway you don't have to add one. You can, it won't hurt. Don't overthink it.
robots.txt is definitely optional but search engines do give a shit. Not from a ranking perspective, just that the majority of bots honor it. It can be really useful to reduce excess hits from misbehaving bots. No real harm in not using it until you need it though.
SEO is a rabbit hole of diminishing returns and increasing madness that you don't need to stress about when the majority of your traffic is from a subreddit.
Your angst reminds me of the robot policy on Cheapskate's Guide.