I interviewed the researcher behind the Misinformation Susceptibility Test
https://youtu.be/vodNabH5qoM
But some important context:
Earlier this month I saw a post regarding a Misinformation Susceptibility Test and was curious how 20 binary questions could be an indicator of someones media biases.
I started digging into the related paper and while the methods and analysis was interesting, there was still a lot of questions. So I reached out to Dr Rakoen Maertens who headed the study and we agreed to a discussion on the assessment and his experiences in social psychology.
The video above is an unlisted, unedited cut of the interview and I'd love to get some feedback:
Firstly: I have offered the Dr a tildes invite and he may engage with any questions or discussion. Time was limited and there were a lot of topics that was only briefly touched on or overlooked. Here is the original paper and supplementary resources if you want to see some of the language model work and bigger 100 question tests.
Secondly: I am going to do a more through edit and posting this on a dedicated channel. Since cutting off reddit, twitter and tiktoc; I've sort of rediscovered a love learning and investigations. I'd like to know if people like this form of engagement and discussions. No fancy production, just simply engaging with the research and academics behind topical and interesting ideas.
I'm already reading into fandom psychology, UV reflective paint, children's TV and CO2 scrubbing technology.
YouTube has an autogenerated transcription now that's pretty decent.
I also jotted down some rough timestamps:
Timestamps
00:00 - Introduction of the Interviewee and of the Field
Discussion about the research
01:05 - What motivated this research
02:35 - Breaking down the technique for analysing the results of a test on misinformation
04:32 - Motivation for making a test tailored for a specific demographic (US)
06:56 - How did you get to the 20 questions presented in the research?
11:54 - Could this test be adapted for other ways information is spread (word of mouth, online group chats, etc...)?
13:51 - Why focus on headlines?
16:20 - Was there any fear that the fake headlines may later become real news?
19:57 - Why doesn't the test provide feedback to the test taker?
21:45 - Would it be possible to adapt the test have higher precision?
23:45 - What has been the response to the research (public, academic, etc...)?
26:25 - What research do you now have in scope now that this work has been published?
29:07 - Is there any complementary research or resources that you recommend?
Discussion on social psychology, the impact of misinformation, and the interviewee
32:09 - History of some clinical terms (e.g. inoculation) used in Social Psychology
36:22 - Combatting misinformation can seem like a losing battle
39:24 - Mental Health on a Societal Scale
41:38 - In what capacity did you do work with the United Nations
Closing
44:38 - What are the healthy ways to address misinformation at the scale of the individual?
48:35 - End
Thanks. Ran it through a few free transcribing tools and the results will need a bit of work. Should figure somthing out for the full vid. I can also hear that my public speaking voice is very out of practice.
Regarding transcription, I don't know if you've tried the Recorder app by Google on Android. It's not designed to transcribe video, but it does a good job transcribing audio over microphone.
Maybe try it out and see?
Thank you for doing this! I remember doing the test and having many questions about why they chose the questions they did.
I would also be interested in other investigations you do. There are not that many people doing this kind of content.
Just like another poster though, I would prefer a transcript to a video interview. But I'll watch the video soon!
The questions really confused me too. There's a lot of cool analytics that went into it but in the end, it only really applies to a certain demographic at this point in time. Would love to see the data quality if the method is used in different cultures and languages.
Wowww the amount of work you went to is amazing! Currently listening to the entire interview.
The researcher himself says that just because you score 18/20 doesn't mean you're worse than someone who scored 20/20 or better than someone who scored 16/20. But you are probably better than someone who scored 11/20. So I think that's a good thing to keep in mind.
I found it funny that he mentions that there were folks giving him articles about the US government manipulating the weather. But I think his answer is pretty interesting. The question of whether or not there is truth in the headline wasn't actually what they were measuring, but rather whether or not you are able to detect "manipulative language" in the headline.
There were a lot of folks in the comments of the last post saying that there was no way to know if something was fake news or not because they couldn't click into an article and read for themselves (and the researcher acknowledges that this is a thing that people do). But then I think about Reddit or Twitter or whichever social media website where people share articles without actually reading the article; or they'll comment on an article based solely off the headline.
I remember reading the comments below a headline about California's reparations task force report in r/moderatepolitics: "California Reparations Task Force calls for elimination of child support debt for black residents" and all of the comments were just about what you'd expect.
The article itself quotes the report (LARGE (1080 pages) PDF warning):
Complete side note... in copy/pasting that, I just noticed the (sic). What is it trying to indicate? "a parent's financial circumstances" seems like a perfectly valid statement?
The comments were all acting like this meant that black parents wouldn't have to pay any child support, and the headline makes no pretense about what it's trying to imply, despite the fact that the quoted parts of the article itself do not support it. The task force recommends that the interest accrued be eliminated. And perhaps most importantly for the racist crew out there: no where in the report does it indicate that this policy only apply to Black people. It is clearly meant to be a universal policy recommendation.
Anyway, I was also skeptical about just testing based on headlines; however, I am at this point persuaded that simply testing on headlines has good enough predictive power.
ETA: the researcher's note that it's more important to maintain trust with someone than to successfully debunk something is really interesting. He says that maintaining that trust makes it more likely to lower their susceptibility to misinformation in the future.
Just wanted to thank you again @SloMoMonday. Really incredible work.
Thanks so much for the kind words. It does feel like I've just scratched the surface and there's so much more to explore. But something I only realized after the fact is that the test isn't some ranking or certification, it's just an indicator of an unconscious bias and that can change moment to moment. So regardless of score, you could have a good/bad day, be faced with a topic you have strong feelings on just need to identify well disguised misinformation.
And it's clear that misinformation tries to override critical thinking with half truthful, emotive and manipulative language that entice you to engage with cherry picked arguments. Or just simply lead you to assume the worst with a practically impenetrable topic; like the child support story. A headline is an excellent tool to do this.
It does lead into many other topics like arbitrating truth with massive online platforms, implications on free speech and improving media literacy.