8 votes

I am becoming increasingly frustrated and beating my self up over this damn website. Let's start with something simple.

I need to stew about the real issue for a few more days...processing my emotions around this is tough right now.

However, to help break the ice and possibly get me out of my mental funk, I have a question.

Is there, really, any need to have a 'robots.txt' file in this day and age? I just have this gnawing feeling that search engines just don't give a shit.

What do you think?

9 comments

  1. [4]
    Adys
    Link
    Good search engines do give a shit but it's more of a polite hint than obligation. What are you trying to achieve?

    Good search engines do give a shit but it's more of a polite hint than obligation.

    What are you trying to achieve?

    6 votes
    1. [3]
      suspended
      Link Parent
      Currently, I don't have a 'robots.txt' file for my website and I'm not sure that I, really, need one. The purpose of this post was to put some 'feelers' out about my general technological/social...

      Currently, I don't have a 'robots.txt' file for my website and I'm not sure that I, really, need one.

      The purpose of this post was to put some 'feelers' out about my general technological/social frustration. It's just that right now I'm not in what I believe is a good 'head space' to go into more details.

      Because I believe that I am stuck, and I don't know where else to turn, I will come clean about everything in the next several days.

      Thank you for responding.

      3 votes
      1. [2]
        Adys
        Link Parent
        Sounds to me like this is not about robots.txt. you can DM me if you want. Anyway you don't have to add one. You can, it won't hurt. Don't overthink it.

        Sounds to me like this is not about robots.txt. you can DM me if you want.

        Anyway you don't have to add one. You can, it won't hurt. Don't overthink it.

        4 votes
        1. suspended
          Link Parent
          I, probably, will. Thanks.

          you can DM me if you want.

          I, probably, will. Thanks.

          1 vote
  2. [2]
    post_below
    Link
    robots.txt is definitely optional but search engines do give a shit. Not from a ranking perspective, just that the majority of bots honor it. It can be really useful to reduce excess hits from...

    robots.txt is definitely optional but search engines do give a shit. Not from a ranking perspective, just that the majority of bots honor it. It can be really useful to reduce excess hits from misbehaving bots. No real harm in not using it until you need it though.

    6 votes
    1. suspended
      Link Parent
      Thanks. I appreciate that.

      Thanks. I appreciate that.

      1 vote
  3. [2]
    heady
    Link
    SEO is a rabbit hole of diminishing returns and increasing madness that you don't need to stress about when the majority of your traffic is from a subreddit.

    SEO is a rabbit hole of diminishing returns and increasing madness that you don't need to stress about when the majority of your traffic is from a subreddit.

    5 votes
    1. suspended
      Link Parent
      I wasn't really worried about SEO to begin with and thank you for responding.

      I wasn't really worried about SEO to begin with and thank you for responding.