Alpha says that its students show “2x growth” if they meet certain criteria on an adaptive standardized test called MAP Growth. Alpha projects each student’s expected growth in the next testing period — for example, a third grader who is a 50th-percentile reader might be expected to grow 7 points from fall to winter. That number is a median; of all the third graders who are 50th percentile readers, half will grow more than 7 points, and half will grow less. If a student grows 14 points, Alpha calls this 2x growth.
The school offers students huge prizes, payouts, and incentives to perform well on those tests. That the resulting scores are higher than at other schools isn’t shocking and doesn’t say much.
But this can produce extremely odd results, particularly in high school, where term-to-term academic growth, as measured by the MAP tests, is far slower than in the early years; the expected gains for high schoolers on the MAP growth tests are close to nil as a result. For example, a ninth-grade student who is a 90th-percentile reader is expected to gain only 0.23 points on the reading MAP test from fall to spring. But this means that a ninth grader gaining two points, which is within normal random variation, would count as 9x growth!
[...]
One other obstacle to evaluating Alpha’s claims is the difficulty of disentangling them from selection effects: In general, the overwhelming majority of the differences between schools in student performance are driven by the self-selection of involved parents and motivated students. It stands to reason that a school serving the children of people who can afford to pay $75,000 a year is going to have better outcomes than any school that doesn’t, even if instructional quality is no better or worse.
Because Arizona’s Unbound Academy is a free public charter school based around the Alpha online platform, that particular school’s performance promises a window into the method’s effectiveness that is less distorted by selection bias.
The first year of results from Unbound, which came out earlier this month, are not encouraging on that front. Alpha had told the state of Arizona that it hoped to get 65% of the school’s students proficient in English language arts and 60% proficient in math. (The statewide averages are 42% and 34%, respectively.)
Instead, after one year, just 28% were proficient in English language arts and 10% in math.
Now, that doesn’t mean that Unbound failed to teach its students anything. The student body seems to have arrived at Alpha performing well below average to begin with. If you run Alpha’s usual MAP Growth analysis, the school can report 1.8x growth.
I don’t really find it plausible that these students in any meaningful sense “learned at 1.8x” in their year at Unbound. If Alpha really had taken a bunch of incredibly low-performing students and were helping them improve, then this would be a success story — but it seems far more likely that it took a bunch of weak students who almost entirely remained weak students. They improved their standardized MAP test scores a bit more than expected because Alpha teaches to the MAP test so directly. Overall this is obviously a failure on Alpha’s own terms.
[...]
Alpha, insofar as it has gotten results, has done so by hosting kids in an environment where they actually reliably use the edtech. That is the true magic.
[...]
It makes more sense to think of Alpha schools as the incentive system and environment that gets the kids to use the Timeback package properly — meaning, if there is an effective intervention, it is not Timeback itself. Just by getting kids to actually do the practice that the apps offer, Alpha seems to have made some real achievements, if not the miraculous numbers they claim.
[...]
But one of the most interesting things to me, as I’ve reported on Alpha School, is the program’s enduring appeal among parents who are well aware that the headline numbers are kind of meaningless.
“I love the school,” Naimoli wrote in a review that I would otherwise be inclined to call a scathing debunking of Alpha. “Alpha kids are increasing their MAP scores faster than kids at traditional schools, but the number is not 2.6 times faster. 2.6 is an erroneously computed and meaningless number.”
[...]
Toward that end, one of my favorite Alpha critiques is from the education writer Michael Pershan, who wrote: “I just watched like three dozen promotional videos from Alpha School. I saw young kids fencing, knitting, rock climbing, biking, jogging, petting a horse, talking to Dr. Phil, golfing, painting, training with Navy SEALs, playing piano, playing guitar, doing pullups, squatting, giving TED talks, kayaking, lighting stuff on fire, baking, running in a parking lot, and recording a podcast. But I struggled to find footage of a student reading a book.”
Alpha, he pointed out, has a very specific viewpoint on the purpose of education: They are trying to minimize time spent on core instruction while maximizing performance. This is simply not something that most schools — especially not most fancy private schools that serve the kinds of kids who can afford Alpha — are trying to do. Most schools will tell you that they are trying to build holistic thinkers and citizens or foster a love of learning. Alpha is trying to get school out of the way as fast as possible.
From the article:
[...]
[...]
[...]
[...]
[...]