I don't know that I care much for this substack post. More generally, I'm not a fan of pseudo science being used to justify racist ideologies, and tend to feel frustrated when I read flawed...
Exemplary
I don't know that I care much for this substack post. More generally, I'm not a fan of pseudo science being used to justify racist ideologies, and tend to feel frustrated when I read flawed rebuttals. I dislike the whole conversation, the need for rebuttals, and dislike critiquing the rebuttals.
I'm not going to say much, but I feel the article confuses the known genetic associations with measured heritability. And if they get that wrong, right out of the gate, it makes me skeptical of the parts I don't have time to run to ground and decrease my desire to dig through sources.
Here is the first paragraph and title of their argument, emphasis mine:
IQ is much less heritable and more confounded than height
We’ll start with the specific point made in The Atlantic article (and the Vox article that it is citing) that, unlike height, adding up all of the genetic variants only predicts a small fraction of IQ score. This is just objectively true: the largest genetic analysis of IQ scores built a predictor that had an accuracy of 2-5% in Europeans, depending on the target cohort [Savage et al. (2018) - Table S9] whereas the largest genetic analysis of height build a predictor that had an accuracy of 45% in Europeans [Yengo et al. (2022) - Figure 4A]. I’m quite certain that 45% is larger than 5% though perhaps the race scientists will come up with some new math to debunk this.
From the opening paragraph of the cited article (Savage):
Intelligence is highly heritable1 and a major determinant of human health and well-being2. Recent genome-wide meta-analyses have identified 24 genomic loci linked to variation in intelligence3–7, but much about its genetic underpinnings remains to be discovered.
Those are the first two sentences in the cited article, not buried somewhere easy to miss. And while the figure of 5.2% cited is correct when you look at the supplemental figures, it doesn't seem to convey the meaning of the paper. There are many better ways to incorporate this papers findings.
The substack was not written targeting lay people given the reading level, etc, so I treated it like a scientific article and clocked through to sources as I got to major claims. So it's disappointing to see this after investing time reading through much of the work.
I wish the argument was better and more sound because I agree with the underlying message.
Edit: not going to be online for a couple hours, but happy to discuss if I see replies later.
I do think that specific section is unnecessary. The big draw of the Yengo et al. 2022 is that it found genetic variants that account for nearly the total heritability of height, and it is odd to...
I do think that specific section is unnecessary. The big draw of the Yengo et al. 2022 is that it found genetic variants that account for nearly the total heritability of height, and it is odd to compare that with Savage et al. 2018 which clearly lacks a similar power. However, doesn't the rest of the article stand fine without it? The point there is that heritability of "intelligence" is notably lower than height. Howe et al. estimated the population heritability of cognitive ability as 23%, which is right in the ballpark of the heritability estimate given by Savage et al. 2018 (h2 = 19-22%). Both of these estimates are still substantially smaller than the heritability height at 38%, and even smaller when comparing the within-sibship heritability (14% vs 34%).
I think asking if the article stands without that part is an excellent question. To me, the question is: What is the thesis of the post? What is the core statement that all of the facts and...
I think asking if the article stands without that part is an excellent question. To me, the question is: What is the thesis of the post? What is the core statement that all of the facts and figures cited support? And that is a bit too vague for me to say. However, the way I read it in its entirety is that it is really dismissive of any contribution of genetics to IQ.
Let's take the following at face value (meaning I haven't checked the underlying source):
As expected, they find that the population heritability for height (37%) is much higher than for IQ (23%) or for educational attainment (12%). [...] What was surprising was the novel estimate of direct heritability. For height, 38% and statistically indistinguishable from the population estimate3. But for IQ, the direct heritability dropped to 15% (with a wide error bar) and for educational attainment all the way down to 4% (with a narrow error bar).
Two things:
This was following over a paragraph of dense circumlocution that boils down to "As the number of SNPs (pronounced snips, which are gene sequences) that we include in our sample increases, the current theory holds that their cumulative contribution to IQ plateaus and asymptotically approaches an upper limit. Using the best version of these methods available, we find that the upper limit to the contribution of genes to height, IQ, and educational attainment are as follows:" I point this out because at times this post feels like a "proof by intimidation" which as a mathematician by training, I like to joke about, but rubs me the wrong way in practice and is a red flag.
Even at face value, 15% as an upper limit to the genetic contribution to IQ, with or without large error bars, doesn't feel as insignificant as the language and rhetoric of the post would suggest. But I'm biased here because when I publish under my real name I take a very formal tone all the time. I would also say that with the wide error bar, as we identify more SNPs, that number could go up just as well as it could go down. We really can't say; that's the nature of error bars.
Lastly, I think it would be interesting to redo some of this augmented by the lens of non-genetic inheritance through multiple epigenetic mechanisms. Instead, I'm going to take a step back and advance the outline of what I feel is an even better argument to counter pseudo-science racists:
We know that the direct contribution of genetics on IQ, however we choose to measure it, is limited to ~15% based on our most up-to-date techniques. We posit that despite this notable level of heritability, regression to the mean would prevent sustained genetically driven differentiation of IQ in heterogeneous populations. Additionally, even in the case of homogenous populations, environmental factors, including place of residence, physical activity, family income, parental education, and parents' occupation, have all been demonstrated to have a more significant role in predicting IQ than genetics.
Further, while a combination of genetic and environmental factors may be associated with IQ distributions at the population level and can support policies to ameliorate disparities caused by environmental sources, they can not and should not be used to make decisions at an individual level. Individuals deserve equal opportunities regardless of their environmental background or genetic composition.
And I would more or less turn those assertions into their own paragraphs, with citations as appropriate.
But reasonable people can disagree on the approach. Personally, I liked the Vox article cited by the substack post more than the substack post because, as far as accessible communication goes, I think it does better. In the substack, I don't care how some of the cited works are presented, and some of the rhetoric and circumlocution cut to the line of what I find kosher in scientific work.
I could also be a cantankerous goat who has spent too many hours this week in policy meetings and wants to fly home now. :)
Edit: As a thought experiment, let's say we had high quality studies that showed eating breakfast and lunch contributed 15% to assessed IQ in K-12 students. I think we would rightly say that's significant and use it as ammo for free meal programs in schools. And if 15% is meaningful coming from a single environmental source, then up to 15% from genetics also seems like it shouldn't be completely hand waived away. Rather, we should simply explain why it doesn't mean anything at the population level.
[…] unlike height, adding up all of the genetic variants only predicts a small fraction of IQ score. This is just objectively true: the largest genetic analysis of IQ scores built a predictor that had an accuracy of 2-5% in Europeans, depending on the target cohort [Savage et al. (2018) - Table S9] whereas the largest genetic analysis of height build a predictor that had an accuracy of 45% in Europeans […]
…
Population heritability includes direct genetic influences on the trait that we typically think of as “genetics” but it also includes a lot of other correlated stuff that we typically think of as confounding, like cultural influences on the trait from relatives or prior generations (including the effects of parenting or dynastic advantages) or biases due to population stratification. In contrast, “direct heritability” is a measure of the specific genetic influences that are acting within individuals, and it is estimated in a large number of families. Direct heritability is immune to many sources of environmental confounding …
What was surprising was the novel estimate of direct heritability. For height, 38% and statistically indistinguishable from the population estimate But for IQ, the direct heritability dropped to 15% (with a wide error bar) and for educational attainment all the way down to 4% (with a narrow error bar). These substantial decreases are the result of some mix of cultural influences, assortative mating, and population structure.
…
After confounding is removed within families to derive an approximately causal estimate, the gap between IQ/education and height grows even further. For educational attainment — the one objectively measurable cognitive outcome we can actually interpret — the causal contribution of genetics is a piddling 4%.
…
Though we’ve known theoretically that the causal arrow between genes and culture could go both ways, these new molecular findings are a clear demonstration of cultural forces shaping and mimicking genetic processes — and the lack of similar forces on height serves as an important negative control.
I think these bits are very relevant to include as well What does IQ measure? Does IQ measure the same thing each time, even after it is re-normed? I also appreciated an earlier note that access...
I think these bits are very relevant to include as well
[T]he mechanism by which an IQ score measures intelligence is not understood at all. Different tests can get substantially different estimates and need to be normalized against some population. The estimates across populations change dramatically with each generation, as does the structure of the subtests that comprise the IQ test (see [Nisbett et al. (2012)]). As a consequence, IQ tests regularly become obsolete and need to be perpetually revised or re-normed (see [Flynn (2009)] for literal life and death consequences of test norms). Unlike height, the thing the IQ score is measuring is unseen; it could be a single mechanism like processing speed, or it could be samples from thousands of different processes, or the emergent property of a complex interactive network. No one knows for sure and the debate has been going on for over a century (see [Clapp Sullivan et al. (2024)] for the most recent perspective).
The bottom line is, while there are many measured IQ scores in different settings, and those scores are often transformed into g factors from a myriad of factor models, there is zero consensus on how these scores or factors map to actual intelligence - the thing we have yet to observe or agree on a model of. Nor, as we saw above, how they map to causal genetic effects.
What does IQ measure? Does IQ measure the same thing each time, even after it is re-normed?
I also appreciated an earlier note that access to the studies themselves are often gatekept by education level. We've done way too much research on psychology on college psych majors who have to get X required hours. Meaning most of our data in that field has historically been on white middle/upper-class, cis, white, able-bodied men.
Nearly a decade of genetic studies of cognitive outcomes were done under the assumption that the effects being identified were direct when that was largely not the case. Prior to that, over a century of race science argued that intelligence is just like any other biological trait, with individual differences explained by simple genetic causes that are easily quantifiable and culturally immutable. It turns out molecular genetics has (for lack of a better word) thoroughly debunked that view. But when this cold fact is pointed out the response is to simply deny it ever happened.
In my experience the response is to wait a few years, rename scientific racism something new and make the same claims all over again.
Yeah. I always cite the study that found studying before an IQ test boosts scores by 15 points. It's less a test of intelligence and more of a test of memorization and what I would call "public...
Yeah. I always cite the study that found studying before an IQ test boosts scores by 15 points. It's less a test of intelligence and more of a test of memorization and what I would call "public knowledge".
Honestly, I don't put much stock in trying to figure out what makes humans "intelligent". While I'm sure it may help educating the next generation, I see too much scientific racism in a lot of these efforts, and I feel like we have bigger fish to fry as a planet. Though I do appreciate this study showing how bad we are about predicting intelligence.
Another study I saw many years ago showed that offering subjects a cash reward for getting a high score on an IQ test caused people who previously had low IQs to suddenly have higher IQs. It had...
Another study I saw many years ago showed that offering subjects a cash reward for getting a high score on an IQ test caused people who previously had low IQs to suddenly have higher IQs. It had little effect on people who already had high IQ scores, though, suggesting that IQ tests may in part be measuring motivation to get a high score. You can imagine how culture can play a huge role in motivation on tests, and it could potentially also explain the Flynn effect (in populations where academic achievement is increasingly highly prized).
This does not mean IQ tests have no value, though. They are measuring something. For example, when replacing lead pipes in a community causes children living there to score higher on IQ tests, that does strongly suggest that consuming lead in drinking water does something to whatever it is that IQ tests measure (whether that's academic motivation, ability to focus, thinking speed, logical prowess, or something else).
I can't remember. It was a very long time ago and I can't find the original study I read. However, this metastudy might be of interest to you. In Study 1 of this paper, they looked at number of...
I can't remember. It was a very long time ago and I can't find the original study I read.
However, this metastudy might be of interest to you. In Study 1 of this paper, they looked at number of different experiments where test subjects were rewarded for earning higher IQ scores. I haven't looked through the papers cited, but I would be surprised if the many of the experimenters gave test subjects time to study. I imagine most of these studies probably just split their test subjects into two equivalent groups and offered rewards to just one group.
Yeah, I was concerned this author would not make those notes but am glad they did and wanted to include them. Especially in any conversation about "IQ" I always harken back to the "cup -> saucer"...
Yeah, I was concerned this author would not make those notes but am glad they did and wanted to include them. Especially in any conversation about "IQ"
I always harken back to the "cup -> saucer" example, where to one person it's obvious that those two objects go together but to another they've never seen a cup and saucer combination because their family doesn't have porcelain teacup sets, or because their teacups don't have handles, etc.
I'd love it if we as a society could just go "huh it's probably not measuring what we think and it's probably ignoring many other cultural paradigms of knowledge and wisdom in exchange for a very narrow sort of view of "intelligence"
But no, it'll be whatever the new term for scientific racism is: "Human Biodiversity," neo-eugenics, or whatever else they've come up with.
Yeah, I've been especially worried with a lot of the big tech founders / c suite guys talking eugenics and engineering smarter children. It's the same old racist talk with a new coat of paint. I...
Yeah, I've been especially worried with a lot of the big tech founders / c suite guys talking eugenics and engineering smarter children. It's the same old racist talk with a new coat of paint.
I also think we need to stop overestimating general intelligence in the sense that a "smart person" can know everything about multiple fields. I see too many of these smart billionaires trying to stick their hands in too many pots, wasting a lot of government tax breaks and grants in the process.
The issue here is that in my experience scientists in the field of psychometrics don't make any of those claims, those are made by laypeople, neoreactionary bloggers etc. Should we throw out...
I'd love it if we as a society could just go "huh it's probably not measuring what we think and it's probably ignoring many other cultural paradigms of knowledge and wisdom in exchange for a very narrow sort of view of "intelligence"
The issue here is that in my experience scientists in the field of psychometrics don't make any of those claims, those are made by laypeople, neoreactionary bloggers etc. Should we throw out something that seems to be quite useful in some limited areas because the public tends to not understand it? Personally I don't think we should.
I honestly think the harm caused by the rich history of scientific racism in regards to I, in addition to modern works like The Bell Curve parroting and advancing those same backwards ideas...
I honestly think the harm caused by the rich history of scientific racism in regards to I, in addition to modern works like The Bell Curve parroting and advancing those same backwards ideas founded on poor science, is greater than whatever benefits there are to using IQ in appropriate ways in science. The fact that IQ is so generally inconsistent anyway doesn't help.
I'd say give it a different name and don't pretend they know what it measures really, since it seems we don't. Especially if you're still ignoring the other paradigms out there. But the history of...
I'd say give it a different name and don't pretend they know what it measures really, since it seems we don't. Especially if you're still ignoring the other paradigms out there.
But the history of scientific racism does not lack actual scientists in it either. And I think scientists today must proceed with the knowledge of that past and attempting to avoid the repetition of it.
These sort of biases are well-known nowadays, so I’d expect modern IQ tests to have fewer of them, and less obvious. The cup-and-saucer thing sounds more like a cautionary tale from older tests?...
These sort of biases are well-known nowadays, so I’d expect modern IQ tests to have fewer of them, and less obvious. The cup-and-saucer thing sounds more like a cautionary tale from older tests?
There are also very abstract IQ tests with no explicit cultural knowledge that are just based on pattern-matching, but they’re biased towards people who are willing to try when given abstract, meaningless puzzles. A familiarity with playing games like that might help?
They literally have to norm the tests every time to every group, and we don't know what they measure. It doesn't map to any tangible thing. It's just "IQ" not necessarily intelligence or reasoning...
They literally have to norm the tests every time to every group, and we don't know what they measure. It doesn't map to any tangible thing. It's just "IQ" not necessarily intelligence or reasoning or spatial thinking, etc.
The cup and saucer is illustrative of the point. What seems obvious to one person or within one culture or one educational system is not necessarily so to everyone.
This is another source of bias -- someone who grew up taking standardized tests is going to have an advantage taking a standardized test over someone who has never taken a standardized test...
This is another source of bias -- someone who grew up taking standardized tests is going to have an advantage taking a standardized test over someone who has never taken a standardized test before, even if the material is somehow completely lacking in bias towards one student, just being experienced with the format and experience of test-taking is a big advantage.
When i was a kid, i loved taking online tests, including IQ tests (which, i am aware are even more useless than the "real ones" but hear me out) So anyway, there i was, like 14? years old, and I'd...
It's less a test of intelligence and more of a test of memorization and what I would call "public knowledge".
When i was a kid, i loved taking online tests, including IQ tests (which, i am aware are even more useless than the "real ones" but hear me out) So anyway, there i was, like 14? years old, and I'd been homeschooled my whole life, by pretty strict christian parents. And i remember taking one of these tests, and just feeling so stupid because a solid third of the test was pop culture knowledge that I'd had no exposure to, a third was like. idk algebra? that i didnt understand because my mom gave up on teaching me by the time i was like 12, and the rest was like. word problems that were super hit or miss (shocking how much cultural knowledge gets passed along in schooling lol) and idk spatial reasoning type stuff that i did ok on.
This test claimed that it was super similar to real IQ tests, and i remember feeling personally slighted because if that was true, I'd be dumb forever, because not only was i not part of the dominant culture that was being tested and i didn't have any of the requiste cultural knowledge, i also lacked an actual y'know. Education. That experience stuck with me, so i guess the test was good for something, even if it wasnt good for testing my alleged intelligence lmao
I understand the dislike of IQ testing as a pushback against people who overestimate its importance, though imo that is usually done by laypeople and not by people actually in the field of...
I always cite the study that found studying before an IQ test boosts scores by 15 points. It's less a test of intelligence and more of a test of memorization and what I would call "public knowledge".
I understand the dislike of IQ testing as a pushback against people who overestimate its importance, though imo that is usually done by laypeople and not by people actually in the field of psychometrics. But this just goes way too far to the opposite side, to the point of being nonsensical.
You can argue that something like Raven's progressive matrices are subject to some cultural bias, I wouldn't trust it to test someone who did not go through what we perceive as reasonably normal educational system (plus it's one of the less accurate tests, but I cannot remember anything else that's this abstract, psychometrics is not exactly a field of interest for me), but it certainly is not a test of memorization or public knowledge and studying similar problems generally does not help. And the results of highly abstract tests like Raven's progressive matrices and something like WAIS, which is more culturally dependent, tend to strongly correlate. How much studying affects IQ scores has also been tested numerous times for different tests.
There is also a myriad of indirect evidence that IQ score as tested certainly means something, things in tune of mathematicians and physicists having high IQ on average and people with low IQ struggling in those fields despite high effort.
Imo the notion that IQ tests are a test of motivation instead of memorization has more merit and it might explain for example the average IQ gap between white and black people in the US, but that gap is likely connected to culture within the US as it does not exist in the UK for example. There's not a lot of evidence that either explanation is true in general, in other societies.
I'm taking an astronomy course specific to life in the universe, and it highlights what "intelligent life" is in a very basic way. It describes it as beings with the ability to communicate, learn,...
Honestly, I don't put much stock in trying to figure out what makes humans "intelligent".
I'm taking an astronomy course specific to life in the universe, and it highlights what "intelligent life" is in a very basic way. It describes it as beings with the ability to communicate, learn, and understand things.
By this metric, nearly every human meets the criteria for intelligence. (The only exception would be those with severe disabilities.)
But I also understand the pursuit of understanding levels of intelligence. Job interviews are designed to assess your intelligence in some way, both generally and related to skills required to perform the job.
However, the concept of scoring someone based on the general concept of intelligence is pointless, in my opinion. Value people for their actions and goals.
Generalizing away from people seems even more fraught. When we say a dolphin or an octopus is very intelligent, what we are really talking about are some remarkable behaviors. And then there’s the...
Generalizing away from people seems even more fraught. When we say a dolphin or an octopus is very intelligent, what we are really talking about are some remarkable behaviors. And then there’s the question of what computers are doing. We shouldn’t expect tests created for humans to be particularly meaningful benchmarks, though it’s interesting when computers do well on them.
For computer benchmarking, I think it’s better to test on the task that you actually care about, but it requires deciding what that task is.
I'm only on a 10 minute break at work rn, but I found this abstract that goes into it a bit. I'll keep digging for my original source. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7709590/
I'm only on a 10 minute break at work rn, but I found this abstract that goes into it a bit. I'll keep digging for my original source.
People can raise their IQ, but it's really hard? It seems like a good news, bad news sort of thing.
[W]ith the properly defined measures of fluid and crystallized intelligence, the experimental group showed a 15 IQ points higher increase than the control group. We concluded that prolonged intensive training in creative problem-solving can lead to substantial and positive effects on intelligence during late adolescence (ages 18–19).
People can raise their IQ, but it's really hard? It seems like a good news, bad news sort of thing.
Although it’s true that the results are still biased, This looks like a different problem than what happens in inexpensive studies of undergrads. Following the link, here’s a quote from the cited...
Although it’s true that the results are still biased, This looks like a different problem than what happens in inexpensive studies of undergrads. Following the link, here’s a quote from the cited paper:
Such ‘healthy-volunteer bias’ is well documented in the UK Biobank (UKBB), one of the most widely used resources for biomedical research. Of the nine million people invited to take part in the UKBB, only 5.5% (~500,000) participated in the study—a sample of volunteers with healthier lifestyles, higher levels of education and better health than the general UK population.
…
In case of the UKBB, the target population is middle-aged to older adults of recent European ancestry living in the United Kingdom, which is not the same as the general UK population.
The UK Biobank resource aims to include 500,000 people from all around the UK who are currently aged 40-69.
By comparison with family-based or retrospective case-control studies, much larger numbers of people need to be recruited into a prospective study and careful follow-up needs to continue for many years until sufficient numbers of any particular disease have developed.
It seems this is a very large-scale, expensive study. How did they recruit people? Apparently they started by sending them letters?
After eight years of planning, criticism, and debate, UK Biobank has begun recruiting its intended 500 000 volunteers. The first batch of 10 000 invitation letters began arriving last week at the homes of people aged 40-69 living in the Manchester area.
Maybe a better comparison is to the difficulties of doing telephone surveys. They need to recruit people somehow, and the recruitment process inherently has a biased response rate. Telephone surveys select for people who answer the phone, and sending letters selects for people who read them and respond.
“Gatekept” doesn’t seem like quite the right word when they’re trying to recruit people rather than keep them out.
There are other things a very well-funded study could do in theory, like going door-to-door in a similar way to the US Census. But perhaps a better contrast is with studies of people who aren’t volunteers. Some countries can do studies using data they collect for other reasons from their national health care system, or they have data from required physicals that young men are required to have for military reasons.
I’m guessing that universal genetic testing doesn’t happen in any medical system, though?
I'm referencing the article. The Manchester area may not be representative and getting to the Manchester area is a gate. Edit: and yes it's a different bias, but it's a similar one where the...
Another emerging source of bias is the act of participating in a genetic study itself. We sometimes forget this point, but people have to jump through hoops to end up in a genetic analysis: they need to know about the study and be able to access it (which typically means living near a major hospital or university and having the resources to engage with it), provide informed consent, fill out the relevant questionnaires, etc. Each of these steps will ascertain for a certain subset of individuals, and unsurprisingly that type tends to be highly educated and do relatively well on IQ tests.
I'm referencing the article. The Manchester area may not be representative and getting to the Manchester area is a gate.
Edit: and yes it's a different bias, but it's a similar one where the paradigm, in this case being a professor in a university full of white male students, creates a set of results that can't be easily universalized
Which professor, university, and set of results do you mean? Or were you speaking generally? Looking again at the UK Biobank study, it wasn't gated on people getting to the Manchester area. They...
Which professor, university, and set of results do you mean? Or were you speaking generally?
Looking again at the UK Biobank study, it wasn't gated on people getting to the Manchester area. They started out by reaching out to people who already live there, but then they went on to do other UK regions, too.
It's of course limited to people in the UK, so if you want to know about American minorities you're out of luck. And that's inherent - many studies are only done in one country, so you need to be careful generalizing, and often people aren't; they will talk in universal terms based on a single-country evidence base. It's a problem any study of people has that physics doesn't have.
(On the other hand, for someone who wants to know about UK genetics, a one-country study is better.)
It doesn't look like there was any bias towards male participation. Here's a bit more about demographics:
UK Biobank participants were more likely to be older, to be female, and to live in less socioeconomically deprived areas than nonparticipants. Compared with the general population, participants were less likely to be obese, to smoke, and to drink alcohol on a daily basis and had fewer self-reported health conditions.
Definitely very white, in part because it's the UK we're talking about:
In the UK Biobank cohort, 94.6% of participants were of white ethnicity, which was similar to the national population of the same age range in the 2001 UK Census (94.5%) but somewhat higher than in the 2011 Census (91.3%; Table 1).
The general criticism you quote might very well be even more true of other studies he didn't cite. But the point I'm trying to make is that bias will still be there even in very large, well-done studies. The data is inherently messy, which is annoying for people who want to make universal statements.
I'm wary of making universal claims about people at all these days. Too difficult! It means sampling the whole world.
Speaking generally about the history of psychological experiments as an example of gatekeeping access to surveys and thus creating results you aren't even aware could be a problem. As those...
Which professor, university, and set of results do you mean? Or were you speaking generally?
Speaking generally about the history of psychological experiments as an example of gatekeeping access to surveys and thus creating results you aren't even aware could be a problem. As those professors weren't.
Intelligence Quotient (IQ) tests are deeply flawed and deeply biased. They have been, for ages, and it's a reflection of the desire to measure something which we can't even define. While...
Exemplary
Intelligence Quotient (IQ) tests are deeply flawed and deeply biased. They have been, for ages, and it's a reflection of the desire to measure something which we can't even define. While methodology for this and thought has progressed since the early days (newer tests are more readily replicated, are more standardized, and do a better job at predicting something), calling it an intelligence quotient is frankly wording we should move away from. I have much less of an issue with and I think we should focus on specific aspects that we understand and can measure - for example having and measuring a logic quotient, a visual processing quotient, or other aspects typically rolled up into an IQ score is much more reflective of what we actually understand and can measure.
Since someone will inevitably come in here and argue that IQ is "good enough", here's a few studies to chew on as to why IQ is in fact, not good enough, and we need to rethink both how we frame this as well as how much bias actually seeps into this.
Large meta-analysis reveals that IQ tests in the UK shows that whites and Asians are smarter
IQ tests frequently show that men and women perform differently on IQ tests- of note, this difference used to be much larger and continually decreases with refinements (this is in fact a false reflection of reality influenced heavily by social pressures but also not reflective of actual real-world performance; this is discussed in great detail in the fantastic book Delusions of Gender)
Education improves intelligence scores meaning that at least some portion of IQ is actually just a reflection of education and has nothing to do with one's ability to abstract, apply logic, understand things, be self-aware, their emotional intelligence, reasoning, planning, creativity, critical thinking, or problem-solving (all measures commonly rolled into IQ)
Nearly all IQ testing research is biased - a large meta-analysis revealed common issues with IQ research
Retest bias shows that an IQ test can increase when testing again, although this tends to plateau at around 3 retests.
A meta-analysis of religiosity as compared to IQ found that high religiosity results in lower IQ. Given that most of the world is religious, this likely reflects a bias in what, exactly, is being measured. There may be some bias towards "free thinking" or some other concept which folks who are less religious unconsciously insert into their testing.
I want to be clear that I post all of this not to say that there is no point in measuring the various facets which are generally associated with intelligence or to say that IQ tests serve absolutely no purpose in society, but rather that we need to think more deeply about what an intelligence quotient is, and to think more deeply about what a score even means for each individual person. When sub-populations such as race results in very different standard deviations of score, one has to question how reliable a study really is. When sub-populations also result in different scores, we should be examining whether this test is biased and not jumping to the conclusion that it means that these sub-populations are less smart or whatever we believe the test to be measuring. This very article exists to highlight just how little we really know what we are measuring and how it has close to no predictive power of anything we can measure.
Perhaps most importantly, I believe the word intelligence is rather charged and something we should try to avoid. For example, do IQ tests measure street smarts? Is street smarts something we would consider a form of intelligence? What about emotional intelligence? How about more abstract ideas rarely considered in research such as sensual intelligence? If I can determine what a substance is by smell, is that not a form of olfactory intelligence akin to visual intelligence? Why is visual intelligence measured in an IQ test but olfactory intelligence not? We need to reconsider what we are measuring and move away from broad terms which have broad definitions, because we almost invariably are introducing untold number of biases based on our own perception of the world and upbringing. What one society values as "intelligence" is going to differ from another, and is ultimately a reflection of societal values. Shifting towards terms we have more precisely defined, such as "visual-spatial intelligence" is a step in the right direction, but we need to simultaneously let go (at least a little) the idea that we can reliably measure anything so complex without large amounts of bias seeping in.
After thinking about it a while I think I largely agree, though with different emphasis. A meta-analysis showing that whites and Asians are “smarter” in the UK is a surprising result that needs to...
After thinking about it a while I think I largely agree, though with different emphasis.
A meta-analysis showing that whites and Asians are “smarter” in the UK is a surprising result that needs to be explained. The first thing to ask is whether it’s a bug of some sort. With a meta-analysis, it’s possible that some studies included are fraudulent, or poorly done; the selection criteria matters.
Even if the studies are well done, there are other possible selection effects. For Asians, it might be a selection effect - smarter people are more likely to do well academically and study abroad, and sometimes they don’t go back. Or maybe it’s not academics, but smarter people are more likely to find work overseas? And there will be downstream family effects. There will be cultural effects - maybe, for some immigrants from some countries, going back isn’t really an option, so they stay regardless. That’s another selection effect.
All sorts of possibilities. “The test is biased” is one of many. And I’m just speculating. Questions like this require deep investigations to understand.
For those of us following along at home, it makes sense to be wary of news reports of scientific studies. When reading a headline about a nutritional study, my first guess is that it’s probably spurious, because they’re hard to do well and there are many badly done studies, and even if it’s a good one, they’re often badly summarized.
So I don’t really blame anyone for deciding that reading about such things isn’t worth their time.
But I think people go a little far sometimes and assume there is no scientific progress in a field because there have been a lot of blunders. My take on this blog post is that there is progress, that some scientists understand genetics now better than they used to? Including understanding some of the things that can go wrong, and being more hesitant about drawing conclusions?
IQ is a summary statistic, and it does seem like a better understanding will come from not depending so much on summary statistics. It doesn’t seem like there’s much consensus on how to break it down, though?
Yes I have the same takeaway. Progress is incremental. I just felt the need to highlight that this progress is happening in a field where some of the basic premises are inherently flawed by...
My take on this blog post is that there is progress, that some scientists understand genetics now better than they used to?
Yes I have the same takeaway. Progress is incremental. I just felt the need to highlight that this progress is happening in a field where some of the basic premises are inherently flawed by design. It is impossible to have an objective measure of intelligence because the very concept of intelligence itself is based on a combination of personal and cultural values. It's not impossible, however, to have an objective measure of the components which we decide to include in an intelligence quotient, and for something which most hold in high regard, we need to be extremely cautious about what we are saying. IQs have been historically used as arguments for hateful ideologies and atrocities and even when they are used to argue for good it's very easy for them to end up worsening disparities because we select for and promote that which we know and understand which is ultimately a function of what we value.
Including understanding some of the things that can go wrong, and being more hesitant about drawing conclusions?
I think at least from a scientific perspective we've learned to be more hesitant about drawing conclusions, but the problem is that science doesn't exist in a vacuum. This science is used by folks who don't understand science as justification for real world action like in politics.
There's two fields I'd like to highlight as reasons why framing is vitally important and why broader changes are necessary - climate change and AI ethics. In the case of climate change, no amount of scientific rigor helped the world meet the many deadlines we've missed with regards to slowing global warming. In fact, the entire field had to learn to shift its presentation of data drastically as a response to climate change deniers cherry picking the data they wanted to push forward their political agenda. I'd encourage you to read the summary section of modern climate science papers and compare it to those published 20 or 30 years ago and note not just the change in tone, but the change in what is stated in the summary and what words they use to present their findings. In the case of AI ethics, there's clear scientific consensus on the massive harm potential, yet you will no doubt have noticed that there is little deceleration in the rapid adoption of AI in fields where AI ethics researchers have already cast warnings about. It would not surprise me to see a similar shift in tone and wording happen over the next 20 years as humanity continues to ignore researchers in this field.
When it comes to what we can learn from these and how it applies to IQ - I once again urge for a push to change the language we use and for us to focus on specific measures. We need to be extremely cautious about any use of quotients such as IQ and even more wary about ideas such as a general intelligence such as g precisely because science doesn't exist in a vacuum and folks are going to misinterpret scientific findings.
It doesn’t seem like there’s much consensus on how to break it down, though
That's correct, but I think the idea is we should focus less on how to define an IQ or a g or any other general factor and instead focus on what we can measure. I personally believe that if we wish to combine some of these more specific measures into something more general, we should avoid the use of the word intelligence at all because of its charged nature.
In the context of this article, I think the author is trying to highlight some of this by focusing in on how 'heritable' it is as a way to attack the historical framing of intelligence being an innate quality and to show just how much it is environmental. I think that the author fails to capture the nuance of how the environmental portion is not just a reflection of the test taker but also of the values of the test creator. Perhaps more importantly, however, I think that the author also fails to spend adequate time or focus breaking apart the social concept of intelligence from the academic one and this is why I highlighted climate science earlier - in certain scientific fields we need to pay much closer attention to this divide between the social and academic when we summarize and talk about findings if we wish for both the science and the world to benefit from this knowledge.
Yes, what we value is often a cultural decision, although sometimes it bottoms out in things that would be valued in any culture, like food and shelter. (The particular kind of food and even...
Yes, what we value is often a cultural decision, although sometimes it bottoms out in things that would be valued in any culture, like food and shelter. (The particular kind of food and even what's considered edible is culturally determined, so there is still substantial cultural influence.)
Similarly, I think there is something related to what we call intelligence that is meaningful in respects to who is capable of various jobs more effectively, though again, the specifics are mostly determined by culture, often in ways that are counterproductive. This is similar to how we might say someone is more athletic in general even if they're not the best at all sports.
When you know what you want, asking a more specific question is going to get better results. For example, SAT's are intended to predict academic performance in college and are fairly good at that, but they're just as controversial for similar reasons. Also, using coding interviews for programming jobs is controversial, in part because they're often done badly, and their relationship to the actual work is a bit muddy. It's a more relevant test, but is it relevant enough? Unclear.
So I suspect that the causation goes the other way, too. Some things are controversial because they matter so much? We don't see quite as much controversy around the big five personality traits, even though they're even more suspect. They don't seem to be used for anything important?
I think it would be interesting to read what Gusev has to say about alternatives to IQ.
While I agree with your general premise that we can measure effectiveness in particular roles, we should avoid hyper-reliance on them as the only weighted factor and we should challenge ourselves...
Similarly, I think there is something related to what we call intelligence that is meaningful in respects to who is capable of various jobs more effectively
While I agree with your general premise that we can measure effectiveness in particular roles, we should avoid hyper-reliance on them as the only weighted factor and we should challenge ourselves as to whether we can even measure this. KPIs are a good example of how picking and choosing can have its pros and cons, and you need look no further than service desk culture to see how malicious these can be and how they've contributed to the enshittification of literally every service number ever (if you can even find them nowadays, they're often hidden behind AI chatbots that refuse to connect to you). How many tickets you close per hour is one way to measure efficiency, but it doesn't result in a product that consumers rate highly. Too much customer rating reliance can also result in companies pushing hard to force consumers to leave a review and encourage fake reviews and handouts in response to high values being given.
A relevant anecdote from my own work history which taught me to think more outside the box was a coworker I had at a previous job. It seemed like this guy's whole family worked at the same company - his brother was in the c-suite, he had a sister on another team, he even had cousins in other departments. He had a similar role to the one I held, but I felt like a lot of his work was seriously amateur... I'd be putting together multifunction dashboards and complex analyses and this dude would often roll up with excel spreadsheets and real simple bar graphs and numbers. He seemed to be on the phone, endlessly. He was the kind of guy who greet everyone, and talk with them in depth about their family- you know the kind of person who seemingly treated job like a fun social space. In short, I wondered whether he was in his position due to nepotism and the quality of his work didn't matter. Over the years, however, I noticed that time and time again when we ran into roadblocks and red tape that this guy would silently remove them. He'd get on the phone, drop by the guy's desk, or talk with them at other social places like their church and win them over. By every metric I ever thought to consider important in my job, he was a terrible worker. But his soft skills combined with his deep social ties in the company and the fact that he was just genuinely a nice guy, made him a huge asset. His effectiveness in his position was minimal, but his effectiveness in the organization was unmatched and irreplaceable. My experience with this guy taught me to think a lot more broadly about how everyone fits into the larger picture and how sometimes value can be packaged in unexpected ways.
We don't see quite as much controversy around the big five personality traits, even though they're even more suspect. They don't seem to be used for anything important?
I mean, I personally have a huge bone to pick with any personality test and have been deeply annoyed by their integration into the workplace in some effectiveness programs, but yes I agree that how controversial something is will be directly influenced on how much it can affect someone's life or humanity. Using IQ to argue for/against resources to groups can mean real-world consequences of opportunity, and is therefore something we need to pay critical attention to.
I don't know that I care much for this substack post. More generally, I'm not a fan of pseudo science being used to justify racist ideologies, and tend to feel frustrated when I read flawed rebuttals. I dislike the whole conversation, the need for rebuttals, and dislike critiquing the rebuttals.
I'm not going to say much, but I feel the article confuses the known genetic associations with measured heritability. And if they get that wrong, right out of the gate, it makes me skeptical of the parts I don't have time to run to ground and decrease my desire to dig through sources.
Here is the first paragraph and title of their argument, emphasis mine:
From the opening paragraph of the cited article (Savage):
Those are the first two sentences in the cited article, not buried somewhere easy to miss. And while the figure of 5.2% cited is correct when you look at the supplemental figures, it doesn't seem to convey the meaning of the paper. There are many better ways to incorporate this papers findings.
The substack was not written targeting lay people given the reading level, etc, so I treated it like a scientific article and clocked through to sources as I got to major claims. So it's disappointing to see this after investing time reading through much of the work.
I wish the argument was better and more sound because I agree with the underlying message.
Edit: not going to be online for a couple hours, but happy to discuss if I see replies later.
I do think that specific section is unnecessary. The big draw of the Yengo et al. 2022 is that it found genetic variants that account for nearly the total heritability of height, and it is odd to compare that with Savage et al. 2018 which clearly lacks a similar power. However, doesn't the rest of the article stand fine without it? The point there is that heritability of "intelligence" is notably lower than height. Howe et al. estimated the population heritability of cognitive ability as 23%, which is right in the ballpark of the heritability estimate given by Savage et al. 2018 (h2 = 19-22%). Both of these estimates are still substantially smaller than the heritability height at 38%, and even smaller when comparing the within-sibship heritability (14% vs 34%).
I think asking if the article stands without that part is an excellent question. To me, the question is: What is the thesis of the post? What is the core statement that all of the facts and figures cited support? And that is a bit too vague for me to say. However, the way I read it in its entirety is that it is really dismissive of any contribution of genetics to IQ.
Let's take the following at face value (meaning I haven't checked the underlying source):
Two things:
This was following over a paragraph of dense circumlocution that boils down to "As the number of SNPs (pronounced snips, which are gene sequences) that we include in our sample increases, the current theory holds that their cumulative contribution to IQ plateaus and asymptotically approaches an upper limit. Using the best version of these methods available, we find that the upper limit to the contribution of genes to height, IQ, and educational attainment are as follows:" I point this out because at times this post feels like a "proof by intimidation" which as a mathematician by training, I like to joke about, but rubs me the wrong way in practice and is a red flag.
Even at face value, 15% as an upper limit to the genetic contribution to IQ, with or without large error bars, doesn't feel as insignificant as the language and rhetoric of the post would suggest. But I'm biased here because when I publish under my real name I take a very formal tone all the time. I would also say that with the wide error bar, as we identify more SNPs, that number could go up just as well as it could go down. We really can't say; that's the nature of error bars.
Lastly, I think it would be interesting to redo some of this augmented by the lens of non-genetic inheritance through multiple epigenetic mechanisms. Instead, I'm going to take a step back and advance the outline of what I feel is an even better argument to counter pseudo-science racists:
We know that the direct contribution of genetics on IQ, however we choose to measure it, is limited to ~15% based on our most up-to-date techniques. We posit that despite this notable level of heritability, regression to the mean would prevent sustained genetically driven differentiation of IQ in heterogeneous populations. Additionally, even in the case of homogenous populations, environmental factors, including place of residence, physical activity, family income, parental education, and parents' occupation, have all been demonstrated to have a more significant role in predicting IQ than genetics.
Further, while a combination of genetic and environmental factors may be associated with IQ distributions at the population level and can support policies to ameliorate disparities caused by environmental sources, they can not and should not be used to make decisions at an individual level. Individuals deserve equal opportunities regardless of their environmental background or genetic composition.
And I would more or less turn those assertions into their own paragraphs, with citations as appropriate.
But reasonable people can disagree on the approach. Personally, I liked the Vox article cited by the substack post more than the substack post because, as far as accessible communication goes, I think it does better. In the substack, I don't care how some of the cited works are presented, and some of the rhetoric and circumlocution cut to the line of what I find kosher in scientific work.
I could also be a cantankerous goat who has spent too many hours this week in policy meetings and wants to fly home now. :)
Edit: As a thought experiment, let's say we had high quality studies that showed eating breakfast and lunch contributed 15% to assessed IQ in K-12 students. I think we would rightly say that's significant and use it as ammo for free meal programs in schools. And if 15% is meaningful coming from a single environmental source, then up to 15% from genetics also seems like it shouldn't be completely hand waived away. Rather, we should simply explain why it doesn't mean anything at the population level.
From the article:
…
…
…
I think these bits are very relevant to include as well
What does IQ measure? Does IQ measure the same thing each time, even after it is re-normed?
I also appreciated an earlier note that access to the studies themselves are often gatekept by education level. We've done way too much research on psychology on college psych majors who have to get X required hours. Meaning most of our data in that field has historically been on white middle/upper-class, cis, white, able-bodied men.
In my experience the response is to wait a few years, rename scientific racism something new and make the same claims all over again.
Yeah. I always cite the study that found studying before an IQ test boosts scores by 15 points. It's less a test of intelligence and more of a test of memorization and what I would call "public knowledge".
Honestly, I don't put much stock in trying to figure out what makes humans "intelligent". While I'm sure it may help educating the next generation, I see too much scientific racism in a lot of these efforts, and I feel like we have bigger fish to fry as a planet. Though I do appreciate this study showing how bad we are about predicting intelligence.
Another study I saw many years ago showed that offering subjects a cash reward for getting a high score on an IQ test caused people who previously had low IQs to suddenly have higher IQs. It had little effect on people who already had high IQ scores, though, suggesting that IQ tests may in part be measuring motivation to get a high score. You can imagine how culture can play a huge role in motivation on tests, and it could potentially also explain the Flynn effect (in populations where academic achievement is increasingly highly prized).
This does not mean IQ tests have no value, though. They are measuring something. For example, when replacing lead pipes in a community causes children living there to score higher on IQ tests, that does strongly suggest that consuming lead in drinking water does something to whatever it is that IQ tests measure (whether that's academic motivation, ability to focus, thinking speed, logical prowess, or something else).
Fair enough. I definitely didn't think of the lead pipes / leaded gasoline situation where the metrics are useful for health reporting.
Do you know if the students were given an opportunity to study between the cash offer and the test?
I can't remember. It was a very long time ago and I can't find the original study I read.
However, this metastudy might be of interest to you. In Study 1 of this paper, they looked at number of different experiments where test subjects were rewarded for earning higher IQ scores. I haven't looked through the papers cited, but I would be surprised if the many of the experimenters gave test subjects time to study. I imagine most of these studies probably just split their test subjects into two equivalent groups and offered rewards to just one group.
Yeah, I was concerned this author would not make those notes but am glad they did and wanted to include them. Especially in any conversation about "IQ"
I always harken back to the "cup -> saucer" example, where to one person it's obvious that those two objects go together but to another they've never seen a cup and saucer combination because their family doesn't have porcelain teacup sets, or because their teacups don't have handles, etc.
I'd love it if we as a society could just go "huh it's probably not measuring what we think and it's probably ignoring many other cultural paradigms of knowledge and wisdom in exchange for a very narrow sort of view of "intelligence"
But no, it'll be whatever the new term for scientific racism is: "Human Biodiversity," neo-eugenics, or whatever else they've come up with.
Yeah, I've been especially worried with a lot of the big tech founders / c suite guys talking eugenics and engineering smarter children. It's the same old racist talk with a new coat of paint.
I also think we need to stop overestimating general intelligence in the sense that a "smart person" can know everything about multiple fields. I see too many of these smart billionaires trying to stick their hands in too many pots, wasting a lot of government tax breaks and grants in the process.
The issue here is that in my experience scientists in the field of psychometrics don't make any of those claims, those are made by laypeople, neoreactionary bloggers etc. Should we throw out something that seems to be quite useful in some limited areas because the public tends to not understand it? Personally I don't think we should.
I honestly think the harm caused by the rich history of scientific racism in regards to I, in addition to modern works like The Bell Curve parroting and advancing those same backwards ideas founded on poor science, is greater than whatever benefits there are to using IQ in appropriate ways in science. The fact that IQ is so generally inconsistent anyway doesn't help.
I'd say give it a different name and don't pretend they know what it measures really, since it seems we don't. Especially if you're still ignoring the other paradigms out there.
But the history of scientific racism does not lack actual scientists in it either. And I think scientists today must proceed with the knowledge of that past and attempting to avoid the repetition of it.
These sort of biases are well-known nowadays, so I’d expect modern IQ tests to have fewer of them, and less obvious. The cup-and-saucer thing sounds more like a cautionary tale from older tests?
There are also very abstract IQ tests with no explicit cultural knowledge that are just based on pattern-matching, but they’re biased towards people who are willing to try when given abstract, meaningless puzzles. A familiarity with playing games like that might help?
They literally have to norm the tests every time to every group, and we don't know what they measure. It doesn't map to any tangible thing. It's just "IQ" not necessarily intelligence or reasoning or spatial thinking, etc.
The cup and saucer is illustrative of the point. What seems obvious to one person or within one culture or one educational system is not necessarily so to everyone.
This is another source of bias -- someone who grew up taking standardized tests is going to have an advantage taking a standardized test over someone who has never taken a standardized test before, even if the material is somehow completely lacking in bias towards one student, just being experienced with the format and experience of test-taking is a big advantage.
When i was a kid, i loved taking online tests, including IQ tests (which, i am aware are even more useless than the "real ones" but hear me out) So anyway, there i was, like 14? years old, and I'd been homeschooled my whole life, by pretty strict christian parents. And i remember taking one of these tests, and just feeling so stupid because a solid third of the test was pop culture knowledge that I'd had no exposure to, a third was like. idk algebra? that i didnt understand because my mom gave up on teaching me by the time i was like 12, and the rest was like. word problems that were super hit or miss (shocking how much cultural knowledge gets passed along in schooling lol) and idk spatial reasoning type stuff that i did ok on.
This test claimed that it was super similar to real IQ tests, and i remember feeling personally slighted because if that was true, I'd be dumb forever, because not only was i not part of the dominant culture that was being tested and i didn't have any of the requiste cultural knowledge, i also lacked an actual y'know. Education. That experience stuck with me, so i guess the test was good for something, even if it wasnt good for testing my alleged intelligence lmao
I understand the dislike of IQ testing as a pushback against people who overestimate its importance, though imo that is usually done by laypeople and not by people actually in the field of psychometrics. But this just goes way too far to the opposite side, to the point of being nonsensical.
You can argue that something like Raven's progressive matrices are subject to some cultural bias, I wouldn't trust it to test someone who did not go through what we perceive as reasonably normal educational system (plus it's one of the less accurate tests, but I cannot remember anything else that's this abstract, psychometrics is not exactly a field of interest for me), but it certainly is not a test of memorization or public knowledge and studying similar problems generally does not help. And the results of highly abstract tests like Raven's progressive matrices and something like WAIS, which is more culturally dependent, tend to strongly correlate. How much studying affects IQ scores has also been tested numerous times for different tests.
There is also a myriad of indirect evidence that IQ score as tested certainly means something, things in tune of mathematicians and physicists having high IQ on average and people with low IQ struggling in those fields despite high effort.
Imo the notion that IQ tests are a test of motivation instead of memorization has more merit and it might explain for example the average IQ gap between white and black people in the US, but that gap is likely connected to culture within the US as it does not exist in the UK for example. There's not a lot of evidence that either explanation is true in general, in other societies.
I'm taking an astronomy course specific to life in the universe, and it highlights what "intelligent life" is in a very basic way. It describes it as beings with the ability to communicate, learn, and understand things.
By this metric, nearly every human meets the criteria for intelligence. (The only exception would be those with severe disabilities.)
But I also understand the pursuit of understanding levels of intelligence. Job interviews are designed to assess your intelligence in some way, both generally and related to skills required to perform the job.
However, the concept of scoring someone based on the general concept of intelligence is pointless, in my opinion. Value people for their actions and goals.
Generalizing away from people seems even more fraught. When we say a dolphin or an octopus is very intelligent, what we are really talking about are some remarkable behaviors. And then there’s the question of what computers are doing. We shouldn’t expect tests created for humans to be particularly meaningful benchmarks, though it’s interesting when computers do well on them.
For computer benchmarking, I think it’s better to test on the task that you actually care about, but it requires deciding what that task is.
There are a wide variety of IQ tests and I’m wondering which ones they used in that study. Could you find the study you read?
I'm only on a 10 minute break at work rn, but I found this abstract that goes into it a bit. I'll keep digging for my original source.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7709590/
People can raise their IQ, but it's really hard? It seems like a good news, bad news sort of thing.
Although it’s true that the results are still biased, This looks like a different problem than what happens in inexpensive studies of undergrads. Following the link, here’s a quote from the cited paper:
Participation bias in the UK Biobank distorts genetic associations and downstream analyses
…
To go into more detail on that:
UK Biobank: Protocol for a large-scale prospective epidemiological resource
It seems this is a very large-scale, expensive study. How did they recruit people? Apparently they started by sending them letters?
UK Biobank gets 10% response rate as it starts recruiting volunteers
Maybe a better comparison is to the difficulties of doing telephone surveys. They need to recruit people somehow, and the recruitment process inherently has a biased response rate. Telephone surveys select for people who answer the phone, and sending letters selects for people who read them and respond.
“Gatekept” doesn’t seem like quite the right word when they’re trying to recruit people rather than keep them out.
There are other things a very well-funded study could do in theory, like going door-to-door in a similar way to the US Census. But perhaps a better contrast is with studies of people who aren’t volunteers. Some countries can do studies using data they collect for other reasons from their national health care system, or they have data from required physicals that young men are required to have for military reasons.
I’m guessing that universal genetic testing doesn’t happen in any medical system, though?
I'm referencing the article. The Manchester area may not be representative and getting to the Manchester area is a gate.
Edit: and yes it's a different bias, but it's a similar one where the paradigm, in this case being a professor in a university full of white male students, creates a set of results that can't be easily universalized
Which professor, university, and set of results do you mean? Or were you speaking generally?
Looking again at the UK Biobank study, it wasn't gated on people getting to the Manchester area. They started out by reaching out to people who already live there, but then they went on to do other UK regions, too.
It's of course limited to people in the UK, so if you want to know about American minorities you're out of luck. And that's inherent - many studies are only done in one country, so you need to be careful generalizing, and often people aren't; they will talk in universal terms based on a single-country evidence base. It's a problem any study of people has that physics doesn't have.
(On the other hand, for someone who wants to know about UK genetics, a one-country study is better.)
It doesn't look like there was any bias towards male participation. Here's a bit more about demographics:
Definitely very white, in part because it's the UK we're talking about:
The general criticism you quote might very well be even more true of other studies he didn't cite. But the point I'm trying to make is that bias will still be there even in very large, well-done studies. The data is inherently messy, which is annoying for people who want to make universal statements.
I'm wary of making universal claims about people at all these days. Too difficult! It means sampling the whole world.
Speaking generally about the history of psychological experiments as an example of gatekeeping access to surveys and thus creating results you aren't even aware could be a problem. As those professors weren't.
Intelligence Quotient (IQ) tests are deeply flawed and deeply biased. They have been, for ages, and it's a reflection of the desire to measure something which we can't even define. While methodology for this and thought has progressed since the early days (newer tests are more readily replicated, are more standardized, and do a better job at predicting something), calling it an intelligence quotient is frankly wording we should move away from. I have much less of an issue with and I think we should focus on specific aspects that we understand and can measure - for example having and measuring a logic quotient, a visual processing quotient, or other aspects typically rolled up into an IQ score is much more reflective of what we actually understand and can measure.
Since someone will inevitably come in here and argue that IQ is "good enough", here's a few studies to chew on as to why IQ is in fact, not good enough, and we need to rethink both how we frame this as well as how much bias actually seeps into this.
I want to be clear that I post all of this not to say that there is no point in measuring the various facets which are generally associated with intelligence or to say that IQ tests serve absolutely no purpose in society, but rather that we need to think more deeply about what an intelligence quotient is, and to think more deeply about what a score even means for each individual person. When sub-populations such as race results in very different standard deviations of score, one has to question how reliable a study really is. When sub-populations also result in different scores, we should be examining whether this test is biased and not jumping to the conclusion that it means that these sub-populations are less smart or whatever we believe the test to be measuring. This very article exists to highlight just how little we really know what we are measuring and how it has close to no predictive power of anything we can measure.
Perhaps most importantly, I believe the word intelligence is rather charged and something we should try to avoid. For example, do IQ tests measure street smarts? Is street smarts something we would consider a form of intelligence? What about emotional intelligence? How about more abstract ideas rarely considered in research such as sensual intelligence? If I can determine what a substance is by smell, is that not a form of olfactory intelligence akin to visual intelligence? Why is visual intelligence measured in an IQ test but olfactory intelligence not? We need to reconsider what we are measuring and move away from broad terms which have broad definitions, because we almost invariably are introducing untold number of biases based on our own perception of the world and upbringing. What one society values as "intelligence" is going to differ from another, and is ultimately a reflection of societal values. Shifting towards terms we have more precisely defined, such as "visual-spatial intelligence" is a step in the right direction, but we need to simultaneously let go (at least a little) the idea that we can reliably measure anything so complex without large amounts of bias seeping in.
After thinking about it a while I think I largely agree, though with different emphasis.
A meta-analysis showing that whites and Asians are “smarter” in the UK is a surprising result that needs to be explained. The first thing to ask is whether it’s a bug of some sort. With a meta-analysis, it’s possible that some studies included are fraudulent, or poorly done; the selection criteria matters.
Even if the studies are well done, there are other possible selection effects. For Asians, it might be a selection effect - smarter people are more likely to do well academically and study abroad, and sometimes they don’t go back. Or maybe it’s not academics, but smarter people are more likely to find work overseas? And there will be downstream family effects. There will be cultural effects - maybe, for some immigrants from some countries, going back isn’t really an option, so they stay regardless. That’s another selection effect.
All sorts of possibilities. “The test is biased” is one of many. And I’m just speculating. Questions like this require deep investigations to understand.
For those of us following along at home, it makes sense to be wary of news reports of scientific studies. When reading a headline about a nutritional study, my first guess is that it’s probably spurious, because they’re hard to do well and there are many badly done studies, and even if it’s a good one, they’re often badly summarized.
So I don’t really blame anyone for deciding that reading about such things isn’t worth their time.
But I think people go a little far sometimes and assume there is no scientific progress in a field because there have been a lot of blunders. My take on this blog post is that there is progress, that some scientists understand genetics now better than they used to? Including understanding some of the things that can go wrong, and being more hesitant about drawing conclusions?
IQ is a summary statistic, and it does seem like a better understanding will come from not depending so much on summary statistics. It doesn’t seem like there’s much consensus on how to break it down, though?
Yes I have the same takeaway. Progress is incremental. I just felt the need to highlight that this progress is happening in a field where some of the basic premises are inherently flawed by design. It is impossible to have an objective measure of intelligence because the very concept of intelligence itself is based on a combination of personal and cultural values. It's not impossible, however, to have an objective measure of the components which we decide to include in an intelligence quotient, and for something which most hold in high regard, we need to be extremely cautious about what we are saying. IQs have been historically used as arguments for hateful ideologies and atrocities and even when they are used to argue for good it's very easy for them to end up worsening disparities because we select for and promote that which we know and understand which is ultimately a function of what we value.
I think at least from a scientific perspective we've learned to be more hesitant about drawing conclusions, but the problem is that science doesn't exist in a vacuum. This science is used by folks who don't understand science as justification for real world action like in politics.
There's two fields I'd like to highlight as reasons why framing is vitally important and why broader changes are necessary - climate change and AI ethics. In the case of climate change, no amount of scientific rigor helped the world meet the many deadlines we've missed with regards to slowing global warming. In fact, the entire field had to learn to shift its presentation of data drastically as a response to climate change deniers cherry picking the data they wanted to push forward their political agenda. I'd encourage you to read the summary section of modern climate science papers and compare it to those published 20 or 30 years ago and note not just the change in tone, but the change in what is stated in the summary and what words they use to present their findings. In the case of AI ethics, there's clear scientific consensus on the massive harm potential, yet you will no doubt have noticed that there is little deceleration in the rapid adoption of AI in fields where AI ethics researchers have already cast warnings about. It would not surprise me to see a similar shift in tone and wording happen over the next 20 years as humanity continues to ignore researchers in this field.
When it comes to what we can learn from these and how it applies to IQ - I once again urge for a push to change the language we use and for us to focus on specific measures. We need to be extremely cautious about any use of quotients such as IQ and even more wary about ideas such as a general intelligence such as g precisely because science doesn't exist in a vacuum and folks are going to misinterpret scientific findings.
That's correct, but I think the idea is we should focus less on how to define an IQ or a g or any other general factor and instead focus on what we can measure. I personally believe that if we wish to combine some of these more specific measures into something more general, we should avoid the use of the word intelligence at all because of its charged nature.
In the context of this article, I think the author is trying to highlight some of this by focusing in on how 'heritable' it is as a way to attack the historical framing of intelligence being an innate quality and to show just how much it is environmental. I think that the author fails to capture the nuance of how the environmental portion is not just a reflection of the test taker but also of the values of the test creator. Perhaps more importantly, however, I think that the author also fails to spend adequate time or focus breaking apart the social concept of intelligence from the academic one and this is why I highlighted climate science earlier - in certain scientific fields we need to pay much closer attention to this divide between the social and academic when we summarize and talk about findings if we wish for both the science and the world to benefit from this knowledge.
Yes, what we value is often a cultural decision, although sometimes it bottoms out in things that would be valued in any culture, like food and shelter. (The particular kind of food and even what's considered edible is culturally determined, so there is still substantial cultural influence.)
Similarly, I think there is something related to what we call intelligence that is meaningful in respects to who is capable of various jobs more effectively, though again, the specifics are mostly determined by culture, often in ways that are counterproductive. This is similar to how we might say someone is more athletic in general even if they're not the best at all sports.
When you know what you want, asking a more specific question is going to get better results. For example, SAT's are intended to predict academic performance in college and are fairly good at that, but they're just as controversial for similar reasons. Also, using coding interviews for programming jobs is controversial, in part because they're often done badly, and their relationship to the actual work is a bit muddy. It's a more relevant test, but is it relevant enough? Unclear.
So I suspect that the causation goes the other way, too. Some things are controversial because they matter so much? We don't see quite as much controversy around the big five personality traits, even though they're even more suspect. They don't seem to be used for anything important?
I think it would be interesting to read what Gusev has to say about alternatives to IQ.
While I agree with your general premise that we can measure effectiveness in particular roles, we should avoid hyper-reliance on them as the only weighted factor and we should challenge ourselves as to whether we can even measure this. KPIs are a good example of how picking and choosing can have its pros and cons, and you need look no further than service desk culture to see how malicious these can be and how they've contributed to the enshittification of literally every service number ever (if you can even find them nowadays, they're often hidden behind AI chatbots that refuse to connect to you). How many tickets you close per hour is one way to measure efficiency, but it doesn't result in a product that consumers rate highly. Too much customer rating reliance can also result in companies pushing hard to force consumers to leave a review and encourage fake reviews and handouts in response to high values being given.
A relevant anecdote from my own work history which taught me to think more outside the box was a coworker I had at a previous job. It seemed like this guy's whole family worked at the same company - his brother was in the c-suite, he had a sister on another team, he even had cousins in other departments. He had a similar role to the one I held, but I felt like a lot of his work was seriously amateur... I'd be putting together multifunction dashboards and complex analyses and this dude would often roll up with excel spreadsheets and real simple bar graphs and numbers. He seemed to be on the phone, endlessly. He was the kind of guy who greet everyone, and talk with them in depth about their family- you know the kind of person who seemingly treated job like a fun social space. In short, I wondered whether he was in his position due to nepotism and the quality of his work didn't matter. Over the years, however, I noticed that time and time again when we ran into roadblocks and red tape that this guy would silently remove them. He'd get on the phone, drop by the guy's desk, or talk with them at other social places like their church and win them over. By every metric I ever thought to consider important in my job, he was a terrible worker. But his soft skills combined with his deep social ties in the company and the fact that he was just genuinely a nice guy, made him a huge asset. His effectiveness in his position was minimal, but his effectiveness in the organization was unmatched and irreplaceable. My experience with this guy taught me to think a lot more broadly about how everyone fits into the larger picture and how sometimes value can be packaged in unexpected ways.
I mean, I personally have a huge bone to pick with any personality test and have been deeply annoyed by their integration into the workplace in some effectiveness programs, but yes I agree that how controversial something is will be directly influenced on how much it can affect someone's life or humanity. Using IQ to argue for/against resources to groups can mean real-world consequences of opportunity, and is therefore something we need to pay critical attention to.