The Lindy Effect

This is a paper about estimating lifespans from ages. How long will some entity last, given its age?

The author recommends thinking in terms of the "hazard rate". A simple example is the exponential decay of a radioactive particle. It has a constant hazard rate. Radioactive decay is a memoryless process, meaning that past history of a particle doesn't affect its future history. This is why we measure the radioactivity of a material in terms of its half-life.

You can get a Lindy effect for radioactive decay even though the hazard rate is constant, provided that you don't know which radioactive material you have. The longer it takes for a particle to decay, the more likely it becomes that it has a long half-life, assuming the right distribution of possible radioactive materials:

For example, the physical properties of a radioisotope (of unknown half-life) are not changing as time proceeds — it remains equally robust — though your expectation of its future lifespan keeps increasing. Its observed or estimated robustness is increasing, but this is because you are gaining information that it is more likely to be a robust specimen if it has made it this far. There is an evidential increase in its robustness as time passes without failure, but no causal increase. The change is in the observer, not in the object being observed.

He discusses a trick by Richard Gott for producing estimates out of thin air by assuming you know exactly nothing about an entity other than its age:

In 1993, Richard Gott presented a remarkable series of observations about predicting future lifespans by applying the Copernican principle. This is a generalisation of the idea that humanity should expect to find itself on a typical planet, rather than a special one. The Copernican principle suggests that unless there is evidence to the contrary, we should expect to find ourselves in typical situations rather than special ones, and the principle is frequently deployed in astronomy and cosmology. Gott’s argument achieved notoriety (and the name ‘The Doomsday argument’) when he applied it to estimating the future lifespan of Homo sapiens, but he has also applied it (with considerable success) to more mundane topics like the future lifespans of Broadway plays and the future reigns of world leaders (Tyson et al. 2016, pp. 436–7).

Apparently this trick works mathematically because it produces estimates that are well-calibrated, but they're very poor estimates anyway, if you know more about the problem. For example, it's well-calibrated to assume that a human will live to twice their current age. But all that means is that it's equally likely to result in very bad estimates in either direction. Assuming a one-year old will live to two is a bad estimate, and so is assuming that an 90-year old will live to 180. It greatly underestimates the lifespans of young people and overestimates them for old people, and being well-calibrated comes from arranging so that these errors cancel out, provided that you choose a person at random in a certain way.

It's absurd because we aren't ignorant about human lifespans. It's only pretending we know nothing, like an alien who knows nothing about Earth.

This puts pressure on the idea that the real-world Lindy effect is generated by things being antifragile — becoming more robust in light of stresses they face. While our beliefs may still shift towards them being more and more robust as the stresses mount up, in some cases this does not reflect any improvements in the intrinsic nature of the entity itself. We may even know that the entity is getting more and more likely to fail over each successive year — it is just that this is being outweighed by our updates towards it being among the more robust objects of that type. Antifragile things can be Lindy, but so can fragile things.

Se we should distinguish between Lindy effects that come from how we mathematically model our ignorance, versus those that come from understanding lifespans:

It is worth noticing that the forms of Lindy effect that we’ve discussed have come in two types. There is an empirical Lindy effect, where the effect appears in the observed distribution of lifespans, which can directly be shown to be heavy tailed. For the empirical Lindy effect, one can ask whether various entities have lifespans that display the effect and confirm the results experimentally. But there is also an epistemic Lindy effect, where it is our rational estimates for remaining lifespan that increase with observed lifespan, and thus our credence distribution over lifespans is heavy tailed. In this case, it isn’t really open to experimental refutation. Instead, a better question would be to ask which combinations of priors, models, and evidence give rise to the effect.

Here's the conclusion:

For this epistemic form, we saw that the Lindy estimate can be a good starting estimate when you have little information — perhaps even optimal if the current age is your only relevant information. And the Lindy estimate is also good in the common situation of uncertainty over the size of a slowly varying hazard rate. As more information comes in about the shape of the survival curve for the entity in question, the Lindy estimate may become less accurate than other ways of estimating. But it will always remain well calibrated.