Emily has recently noticed that if she takes 400 mg magnesium in the morning, it’s much less likely she’ll have a migraine that afternoon. This isn’t a sure thing — she still gets migraines on days when she takes the magnesium — but it seems like it’s less than 80%.
One way for Emily to get some support for this hypothesis would be for her to run a simple AB self-experiment. She could take no magnesium for two weeks, then take 400 mg magnesium every morning for two weeks, and see if it makes a difference for her migraines. If she gets migraines 80% of the time in weeks where she’s taking no magnesium and only 40% of the time in weeks where she’s taking 400 mg magnesium, that seems like evidence that the magnesium is helping.
And it is evidence, but it’s on the weak side. Depending on how you slice it, the effective sample size here is two — just one fortnight with magnesium, and one fortnight without. You shouldn’t draw strong conclusions here for the same reason that you shouldn’t draw strong conclusions from a study with one person in the experimental group and one person in the control group. There’s just not that much evidence.
[...]
To account for these problems, Emily can just go ahead and get a larger sample size. She can use a random number generator to randomly assign days to either take magnesium or not take magnesium, and then follow that random assignment.
With the addition of these two small steps, she can now use a normal within-subjects experimental approach. Let’s imagine she finds that there’s an 80% chance of developing a migraine on days when she takes no magnesium, and a 30% chance of developing a migraine on days when she takes 400 mg magnesium. She can demonstrate this difference to an arbitrary level of precision, just by running the trial for more days.
[...]
We should emphasize that N = 1 studies falsify a very specific kind of null hypothesis: that an intervention cannot work. If the intervention works for you, that just shows that the intervention can work.
[...]
Even with better randomization, however, these designs still have a lot of limitations.
For a start, they’re limited by the speed of your research cycle. For example, repeated-measures studies won’t work very well for studying obesity, because people tend to lose and gain weight pretty slowly. It may take months to lose and then regain weight, so it’s hard to study weight gain with this method. If you have to randomize periods of months, it will take you a full year to get a sample size of 12. In comparison, headaches would be easier to study, since they come and go daily or even hourly, and you could randomize your treatment on much shorter timescales.
[...]
If a cure is too powerful, or has long-lasting effects, that actually makes it harder to study. If magnesium cures Emily’s migraines for a month, she’ll have to wait a month between randomization cycles, and it will take her years to get a decent sample size.
From the blog post:
[...]
[...]
[...]
[...]