12 votes

Randomized trial shows AI tutoring effective in Nigeria

8 comments

  1. [6]
    scarecrw
    Link
    I've been eagerly waiting to see how some of these AI-assisted educational programs perform, especially those focused on broadening access to education. A lot of the overly-rosy perspectives tend...

    I've been eagerly waiting to see how some of these AI-assisted educational programs perform, especially those focused on broadening access to education. A lot of the overly-rosy perspectives tend to compare outcomes to students who received no treatment whatsoever, which is a disingenuous way to report an approach's effectiveness, so it was very confidence inspiring to see that this also outperformed the majority of comparable interventions.

    I was also pleased to see from their videos how they seemed to have made the experience a social activity, involving interaction with the teacher and other students. A lot of failed attempts at using technology in education seem tailored for some mythical, emotionless, entirely self-motivated student.

    There's some draw to the idea that, if only the explanations were perfect and practice exercises just-so, every student would breeze through any topic. Maybe the AI optimists are correct, and we'll see some revolution coming, but from my experience with students in the present, a lot of the effectiveness of educational approaches stems from fostering an environment conducive for learning and building students' confidence in their own abilities. While current LLMs seem far from providing this on their own, I hope more efforts like this one find ways to use them in conjunction with more human approaches.

    7 votes
    1. skybrian
      Link Parent
      In this case, they're comparing to what schools normally do in Nigeria. Some of them might not be so great? But it also means that AI tutoring might be especially useful there.

      In this case, they're comparing to what schools normally do in Nigeria. Some of them might not be so great? But it also means that AI tutoring might be especially useful there.

      5 votes
    2. [4]
      OBLIVIATER
      (edited )
      Link Parent
      Yeah, I imagine something is better than nothing, but I would worry on an AI tutor possibly giving false information to a student. I guess if you think about it though, that's also a concern with...

      Yeah, I imagine something is better than nothing, but I would worry on an AI tutor possibly giving false information to a student. I guess if you think about it though, that's also a concern with real teachers haha. A lot of the things I learned growing up turned out to be not true or exaggerations which is always a bit of a jolt when you realize something you've thought was true for 20 years was just some random teachers daydream.

      I think its fair to compare the outcomes between an AI and nothing because in a lot of these situations the alternative to it would be nothing. They do need to be clear about the comparisons though as to not implicitly mislead people. I think there is space in the world of education for tools like this as long as they prove to be somewhat effective. Good teachers are in very short supply, even in countries that actually pay them living wages. The number of young people going into teaching careers is cratoring worldwide and likely isn't going to get any better any time soon.

      2 votes
      1. [2]
        stu2b50
        Link Parent
        Well, they're mostly learning English, according to the article. IMO foreign language is the best case scenario for chatGPT-based tutoring. Language is inherently fuzzy and soft.

        Well, they're mostly learning English, according to the article. IMO foreign language is the best case scenario for chatGPT-based tutoring. Language is inherently fuzzy and soft.

        6 votes
        1. Inanimate
          Link Parent
          Agreed. I've had to attend PD shops and the like where they're trying to sell us on AI tutoring, but as a math teacher, it's still abysmal. Any time I've tried to stress test it during these...

          Agreed. I've had to attend PD shops and the like where they're trying to sell us on AI tutoring, but as a math teacher, it's still abysmal. Any time I've tried to stress test it during these workshops, it takes less than 5 minutes for it to spit out nonsense. I expect it'll be far easier to adapt into English and other language classes first and foremost, with History or Science being the next best contender, I suppose?

          3 votes
      2. unkz
        Link Parent
        This kind of thing is best done when paired with RAG and a well curated textbook, and clear guidelines about straying from the source material.

        This kind of thing is best done when paired with RAG and a well curated textbook, and clear guidelines about straying from the source material.

        2 votes
  2. skybrian
    Link
    From the article: ... See the article for a chart. Just from looking at it, it is pretty clearly an improvement. I don't know about "overwhelming," but they only used it for six weeks. They seem...

    From the article:

    The results of the randomized evaluation, soon to be published, reveal overwhelmingly positive effects on learning outcomes. After the six-week intervention between June and July 2024, students took a pen-and-paper test to assess their performance in three key areas: English language—the primary focus of the pilot—AI knowledge, and digital skills.

    Students who were randomly assigned to participate in the program significantly outperformed their peers who were not in all areas, including English, which was the main goal of the program. These findings provide strong evidence that generative AI, when implemented thoughtfully with teacher support, can function effectively as a virtual tutor.

    Notably, the benefits extended beyond the scope of the program itself. Students who participated also performed better on their end-of-year curricular exams. These exams, part of the regular school program, covered topics well beyond those addressed in the six-week intervention. This suggests that students who learned to engage effectively with AI may have leveraged these skills to explore and master other topics independently.

    Moreover, the program benefited all students, not just the highest achievers. Girls, who were initially lagging boys in performance, seemed to gain even more from the intervention, highlighting its potential to bridge gender gaps in learning.

    ...

    The more sessions students attended, the greater their gains. As discussed in the first blog, attendance was challenging for many students due to factors like flooding during the rainy season, teacher strikes, and after-school work commitments. Using a robust monitoring system developed for the program, we tracked attendance carefully. Each additional day of attendance resulted in significant improvements in learning outcomes. Importantly, the benefits did not taper off as the program progressed. This suggests that a longer program might lead to even greater gains.

    See the article for a chart. Just from looking at it, it is pretty clearly an improvement. I don't know about "overwhelming," but they only used it for six weeks.

    They seem impressed, though:

    The learning improvements were striking—about 0.3 standard deviations. To put this into perspective, this is equivalent to nearly two years of typical learning in just six weeks. When we compared these results to a database of education interventions studied through randomized controlled trials in the developing world, our program outperformed 80% of them, including some of the most cost-effective strategies like structured pedagogy and teaching at the right level.

    4 votes
  3. OBLIVIATER
    (edited )
    Link
    I haven't read the article yet, but I plan to when I get home. I work in Ed tech and our old CEO was obsessed with trying to get some form of AI tutoring system online during the last few years....

    I haven't read the article yet, but I plan to when I get home. I work in Ed tech and our old CEO was obsessed with trying to get some form of AI tutoring system online during the last few years. It ultimately ended up crashing and burning and they laid off most of the team last week. I think they're still trying to do something with it but at a much reduced capacity.

    I think most of the difficulty was training models on good data without violating COPPA, I had to do a lot of manual data anonymizing which was a huge pain in the ass

    3 votes