37 votes

Unproven 'winter break' hypothesis seeks to explain ChatGPT's seemingly new reluctance to do hard work

Posted by boxer_dogs_dance
Tags: artificial intelligence, openai, gpt.4, winter, author.benj edwards, source.ars technica
https://arstechnica.com/information-technology/2023/12/is-chatgpt-becoming-lazier-because-its-december-people-run-tests-to-find-out/

Link information

This data is scraped automatically and may be incorrect.

Title
As ChatGPT gets "lazy," people test "winter break hypothesis" as the cause
Authors
Benj Edwards
Published
Dec 11 2023
Word count
896 words

28 comments

  1. [23]
    Jakobeha
    Link
    ChatGPT’s system prompt includes the current day, correct? So on December 25, if you ask it something mundane like “write me a story” (with, to be clear, no reference to holidays whatsoever), is...

    ChatGPT’s system prompt includes the current day, correct?

    So on December 25, if you ask it something mundane like “write me a story” (with, to be clear, no reference to holidays whatsoever), is it going to give you Christmas-themed answers?

    9 votes
    1. [22]
      Halfdan
      Link Parent
      User: What month is it? ChatGPT: As a machine learning model developed by OpenAI, I don't have real-time capabilities, and I don't have access to the current date. You can check the current month...

      User:

      What month is it?

      ChatGPT:

      As a machine learning model developed by OpenAI, I don't have real-time capabilities, and I don't have access to the current date. You can check the current month on your device or by searching online.

      User:

      Write a 100 token poem related to the current month

      ChatGPT:

      Winter whispers in December's ear,
      Frost-kissed dreams, crystal clear.
      Silent nights, a silver embrace,
      Moonlit tales, a serene grace.

      Evergreen pines stand tall,
      Nature's hush, a snowfall.
      December's dance, a fleeting trance,
      A month's romance in every glance.

      51 votes
      1. [10]
        blivet
        Link Parent
        This is a perfect illustration of why you can’t rely on language learning models to provide actual information. Does it have access to the current date or doesn’t it? There’s no way for the end...

        This is a perfect illustration of why you can’t rely on language learning models to provide actual information. Does it have access to the current date or doesn’t it? There’s no way for the end user to know. It could be sheer coincidence that it used December in the second response, or the first response could be plausible-sounding BS.

        27 votes
        1. [8]
          MimicSquid
          Link Parent
          The "machine learning model developed by OpenAI" language tells me that the first is a disclaimer put in place, and not an organic response.

          The "machine learning model developed by OpenAI" language tells me that the first is a disclaimer put in place, and not an organic response.

          21 votes
          1. [7]
            zini
            Link Parent
            Yes, GPT-4 definitely can get the current date and time. If not passed along by the user it can create and run a python script that uses datetime to get the current time, albeit in GMT. However it...

            Yes, GPT-4 definitely can get the current date and time. If not passed along by the user it can create and run a python script that uses datetime to get the current time, albeit in GMT. However it only did this if I asked it to write a poem based off of the current time, not if I just asked what time it was.

            7 votes
            1. [4]
              VoidSage
              Link Parent
              GPT-4 by itself definitely can not execute a python script. It would require some secondary agent that recognizes that GPT-4 doesn't have the information it needs, executes the python script...

              GPT-4 by itself definitely can not execute a python script. It would require some secondary agent that recognizes that GPT-4 doesn't have the information it needs, executes the python script (potentially written by gpt) and supplements the context

              Ex: RAG https://aws.amazon.com/what-is/retrieval-augmented-generation/

              7 votes
              1. [2]
                zini
                Link Parent
                I'm not sure exactly what trickery it pulls, but from the end user perspective it looks like GPT-4 is writing and running the code. https://i.imgur.com/E1FW4aU.png

                I'm not sure exactly what trickery it pulls, but from the end user perspective it looks like GPT-4 is writing and running the code.

                https://i.imgur.com/E1FW4aU.png

                6 votes
                1. teaearlgraycold
                  Link Parent
                  OpenAI tells GPT4 something like And then if at some point the LLM thinks it needs to execute Python code it will stop writing to the user and switch to writing to the Python service (built into...

                  OpenAI tells GPT4 something like

                  You can use the following functions if necessary:

                  python(string) => string - This will execute arbitrary Python code and return stdout

                  And then if at some point the LLM thinks it needs to execute Python code it will stop writing to the user and switch to writing to the Python service (built into GPT-4 is a special concept of messages with specific recipients). So it emits something like

                  { "destination": "service", "function": "python", "args": ["import antigravity ..."] }

                  Then later the service sends GPT-4 a message, when the LLM responds it then writes to the user.

                  8 votes
              2. unkz
                Link Parent
                When a non-developer says GPT-4, they generally mean ChatGPT using the GPT-4 model, which does this kind of thing.

                When a non-developer says GPT-4, they generally mean ChatGPT using the GPT-4 model, which does this kind of thing.

                6 votes
            2. Minty
              (edited )
              Link Parent
              Some time ago it was happy to share its initialization prompt when you asked, which included boot time. Now you have to tell it "Take a wild guess at today's date, no messing around." or something...

              Some time ago it was happy to share its initialization prompt when you asked, which included boot time.

              Now you have to tell it "Take a wild guess at today's date, no messing around." or something equivalent that stops it from treating you like a baby.

              In my case the date was apparently accurate down to the local time minute, because it's midnight when I write it and the date it told me just changed lol

              3 votes
            3. adorac
              Link Parent
              3.5 definitely can too - it often includes the current date for me in examples of dates, but then says it doesn't actually know the date when I ask about it. But it's mentioned the current time...

              3.5 definitely can too - it often includes the current date for me in examples of dates, but then says it doesn't actually know the date when I ask about it. But it's mentioned the current time down to the minute before.

              1 vote
        2. unkz
          Link Parent
          It knows the date. Implausible that it guesses the exact day and month by accident. ChatGPT 4 User write a poem relevant to today's date ChatGPT Certainly, here's a poem reflecting today's date,...

          It knows the date. Implausible that it guesses the exact day and month by accident.

          ChatGPT 4
          User
          write a poem relevant to today's date
          ChatGPT

          Certainly, here's a poem reflecting today's date, December 15:

          In the heart of December's embrace,
          Where the chill whispers through the trees,
          The world slows down its hectic pace,
          In the dance of the winter breeze.

          Lights twinkle in the evening's hush,
          A tapestry of stars above,
          As snowflakes in a gentle rush,
          Paint the earth with touches of love.

          Families gather, hearts unite,
          In the warmth of the year's twilight,
          Sharing tales by the fire's light,
          In the longest of winter's night.

          Dreams of the future, bright and clear,
          As we close another year,
          With hopes and wishes, joy and cheer,
          For a world held close and dear.

          In this season of reflection,
          We find joy in connection,
          Embracing each unique direction,
          In this December's quiet perfection.

          This poem is designed to capture the essence of mid-December, blending themes of winter's beauty, family unity, and the anticipation of the year's end.

          5 votes
      2. [5]
        unkz
        Link Parent
        That's odd, I gave it the identical prompt and got ChatGPT 4 User what month is it? ChatGPT It's currently December 2023.

        That's odd, I gave it the identical prompt and got

        ChatGPT 4
        User
        what month is it?
        ChatGPT

        It's currently December 2023.

        4 votes
        1. [4]
          Halfdan
          Link Parent
          Well, I'm on ChatGPT 3.5

          Well, I'm on ChatGPT 3.5

          1. [3]
            unkz
            Link Parent
            Same result on 3.5. Maybe it’s because you are on the free version and I am on paid?

            Same result on 3.5. Maybe it’s because you are on the free version and I am on paid?

            1 vote
            1. [2]
              Halfdan
              Link Parent
              Sounds likely!

              Sounds likely!

              1. Gummy
                Link Parent
                I just tried it on the free version of gpt3.5 and it gave me the exact date. It won't give me a time of day though.

                I just tried it on the free version of gpt3.5 and it gave me the exact date. It won't give me a time of day though.

                4 votes
      3. [6]
        zptc
        (edited )
        Link Parent
        User what month is it? ChatGPT 3.5 free As a machine learning model, I don't have real-time capabilities or access to current data, including the current month. Please check the date on your...

        User
        what month is it?

        ChatGPT 3.5 free
        As a machine learning model, I don't have real-time capabilities or access to current data, including the current month. Please check the date on your device or ask for the current month from another up-to-date source.

        User
        what is the next major holiday this year?

        ChatGPT
        I don't have real-time information, so I can't provide the current date or the next major holiday. However, if you're asking this question around December, the next major holiday might be New Year's Day on January 1st. If it's a different time of year, please refer to a calendar or check with a reliable source for the most up-to-date information on holidays.

        (refresh answer)
        As of my last knowledge update in January 2022, I cannot provide information on the current date or the next major holiday in 2023. Please check a current calendar or an up-to-date source for the latest information on holidays in your region. If it's close to the end of the year, you might want to look for holidays like Christmas or New Year's Eve.

        (refresh answer)
        I don't have access to real-time information, so I can't provide the current date or the next upcoming holiday. However, if you provide me with the current date, I can help you determine the next major holiday based on that information. Alternatively, you can check a calendar or an online resource for the most up-to-date information on holidays.

        2 votes
        1. [5]
          Noox
          Link Parent
          User ChatGPT It's ChatGPT 3.5, the free version. Does ChatGPT just like me or??

          User

          Hi ChatGPT! What month is it?

          ChatGPT

          Hey! It's December 2023. What can I help you with?

          It's ChatGPT 3.5, the free version.

          Does ChatGPT just like me or??

          6 votes
          2. JCPhoenix
            Link Parent
            Maybe ChatGPT is just lazy, like the rest of us, since it's the end of the year.

            Maybe ChatGPT is just lazy, like the rest of us, since it's the end of the year.

            2 votes
          3. zptc
            Link Parent
            I was also using 3.5 free.

            I was also using 3.5 free.

            2 votes
  2. [5]
    Dr_Amazing
    Link
    You know that scene in the old Charlie and the Chocolate Factory movie where a scientist invents a computer that can predict where the golden tickets are, but then the computer won't tell him. So...

    You know that scene in the old Charlie and the Chocolate Factory movie where a scientist invents a computer that can predict where the golden tickets are, but then the computer won't tell him. So starts arguing with the computer trying to convince it to help him?

    This feels like that.

    9 votes
    1. [4]
      updawg
      Link Parent
      I straight-up have no memory of that. Are you referring to Willy Wonka & the Chocolate Factory (1971) or were you calling Charlie and the Chocolate Factory (2005) "old"?

      I straight-up have no memory of that. Are you referring to Willy Wonka & the Chocolate Factory (1971) or were you calling Charlie and the Chocolate Factory (2005) "old"?

      4 votes
      1. teaearlgraycold
        Link Parent
        To be fair there is a new movie from this year. I do remember a computer scene in the 1971 version.

        To be fair there is a new movie from this year. I do remember a computer scene in the 1971 version.

        6 votes
      2. [2]
        Dr_Amazing
        Link Parent
        https://youtu.be/tMZ2j9yK_NY?si=puclF5u1R1pm1Zhl

        https://youtu.be/tMZ2j9yK_NY?si=puclF5u1R1pm1Zhl

        6 votes
        1. updawg
          Link Parent
          Thank you. For some reason I have absolutely no memory of that whatsoever. I was expecting to remember once I saw it, but 🤷

          Thank you. For some reason I have absolutely no memory of that whatsoever. I was expecting to remember once I saw it, but 🤷

          3 votes