Article
Off The Record #15: Warning! Fake News!
Misinformation and Geography
By the Busara OTR Team. Special thanks to Tommie Thompson, Kriti Chouhan, Anisha Singh and Aditya Jagati for their input, research and collaboration on this piece.
In 2017 The Shed at Dulwich finally became the top rated restaurant in London on Tripadvisor. The thing is — The Shed did not exist. Journalist Oobah Butler had listed the fake restaurant as “an appointment-only restaurant located in South London, The Shed has been operating privately for years.” He then paid up to 10 pounds per review to get people to leave him good ratings on imaginary meals — a menu he had categorized according to “mood.” Anyone who called for a reservation would be told that the restaurant was fully booked for months — which only boosted its ratings further.
When the story finally broke Oobah opened for one night, serving guests dollar meals on plastic plates after making them go through a rube goldberg machine of an entrance. By this time all Tripadvisor could do is say they had been duped and remind the world that no one’s really interested in opening a fake restaurant, they are more incentivized to raise the ratings of a real one — which is what they check for.
Strange things can happen when people don’t have the right information — and not every instance is just about pulling down a corporate entity’s proverbial pants. Disconnects between public opinions and reality in topics such as vaccine safety, climate change and evolution have existed for a long time (Scheufele & Krause, 2019). If unchecked, false information can undermine policies and civic culture by promoting general mistrust and encouraging sub-optimal behaviors. The urgency to stop misinformation has reemerged over the past few months during the global COVID-19 pandemic as misinformation undermines public health efforts.
The solution, though, is not as simple as removing inaccurate information, since identifying fake news is a difficult and contested process. Perhaps that’s why “demand-side” interventions that increase digital literacy and provide attractive high-quality alternatives may be more effective in curbing the spread of misinformation. A study by Rand et al. found that most people fall for false information because they are cognitively lazy, and it is cognitively “expensive” to critically evaluate everything that we hear. This suggests that nudging people to think more critically of their news may be an effective policy. By understanding these processes, we can develop interventions that (1) encourage people to consume information better and (2) route resources to domains where fake news acceptance is more likely.
In this Off the Record, we explore the link between the geography of a news headline and the ability to ascertain its authenticity. In a globalized world, articles from one country can easily be shared with people in other countries. This can be problematic when consumers in different cultures are not attuned to an information’s source context. For instance, there may be subtle linguistic red flags in Indian English that alert readers to an article’s accuracy, and these cues may not be easily understood by non-Indian readers in the Anglosphere.
We assessed this link by looking at whether the country context of a piece of news affects whether people are able to distinguish its accuracy. News in local contexts have lower barriers to discernment, since critical judgement presumably requires less cognitive resources. People may also be more willing to independently verify fake news taking place in their local context, either because they are more interested in it or because they are more knowledgeable of credible corroborating sources (e.g. a Nigerian may not know what the credible media outlets are in Kenya). Our study addresses the following research question:
Are people better able to classify news as true or false if the subject takes place in their country of residence?
Perhaps that’s why “demand-side” interventions that increase digital literacy and provide attractive high-quality alternatives may be more effective in curbing the spread of misinformation.
What did we do?
To better understand the link between the ability to discern misinformation and geography, we sent a list of news headlines to our staff in Kenya, Nigeria, and India, and to partner organizations, and asked them to judge the accuracy. Our hypothesis was that people would have more accurate judgements for headlines about their country of residence than for headlines about other countries. The headlines were taken from real online articles; true headlines were gathered from credible local sources and false headlines were taken from fact checking websites, such as Africa Check and Snopes. After reading a headline, participants were asked whether they believed it was definitely false, likely false, likely true, or definitely true. We did not manage the discernment process, which meant participants were free to verify the headline on credible outlets by their own volition or to ask their peers for input.
We received 55 Kenyan, 10 Nigerian, 12 Indian, and 8 Other submissions. Since most of our participants were based in Kenya, Nigeria, and India, we focused our headlines on those three countries. In total, all respondents were shown eight headlines — two true headlines and two false headlines from their respective place of residence, and we further sent two true headlines and two false headlines randomly selected from the other two countries . Participants who were not located in those three countries were only analyzed as “international headline” readers, and were randomly sent four true headlines and four false headlines from India, Kenya, and Nigeria. In all, 85 respondents were exposed to 680 headlines, out of which 655 headlines (351 international and 304 local) were judged for their accuracy (14 respondents refused to judge one or more headlines).
We also analysed our participants’ trust in the media in general and the local media. Overall, we found that our respondents were relatively distrustful of the media. When asked about the media in general, only 45.24% of respondents said they were somewhat trustful or very trustful (on a 5-point scale), and only 23.81% reported the same for their local media. Interestingly, those who were less trustful of the media were not any more likely to independently verify headlines. This may be because their distrust of the news extends to the verification sources as well.
What did we find?
We examined two main outcome measures. The first was the percentage of true or false headlines that a respondent judged correctly. If a respondent only answered four headlines and refused to judge the remaining four, we calculated the percentage out of four total headlines. The second outcome was the score assigned to false headlines. We used a four-point scale: definitely false (1), likely false (2), likely true (3), and definitely true (4). This means that a lower score reflected a “better” judgment.
One in two people were not able to accurately classify headlines, even real ones
Geography does not appear to affect the ability to discern between fake and real news. We found a 45.39% and 46.44% accuracy on true or false local and international headlines respectively, and the difference was not statistically significant. Similarly, we did not see a statistical difference in the average score rating of false headlines, where local false headlines received an average score of 2.58 out of 4 and international false headlines received an average score of 2.61.
Geography does affect ability to discern between fake and real headlines for specific countries
We found that even though headline geography did not make a difference for the entire sample, there were significant differences by office location. Those based in India were 21.81 percentage points more accurate in judging local headlines as true or false compared to international headlines, and those based in Nigeria judged local headlines more accurately by 21.75 percentage points compared to international headlines. Kenyan respondents however, had opposite success, in which local true and false headlines had 14.71 percentage point lower accuracy than international headlines.
When looking at the average score rating for fake headlines, only those in India showed a statistically significant higher accuracy in judging local fake versus international fake headlines.
These patterns may mean that the effects of the geography of headlines are very specific to individual countries, as opposed to occurrence of a general effect across all contexts. Specifically, those in India and Nigeria conformed to our main hypothesis that people are more discerning of local news, while those in Kenya saw the opposite. In fact, when we removed Kenyan respondents and re-ran our analysis, we found that respondents were 26% more likely to accurately judge true and false local headlines than international headlines.
These findings may be purely random, since our India and Nigeria samples are relatively small. However, they may also reflect local media perceptions. When we looked at our media trust measures, we found that those in India were substantially more skeptical of the local media. On a scale of 1 -5, Indians rated the media with an average score of 1.25, compared to 2.7 and 2.2 in Kenya and Nigeria respectively. Although the presented local headlines were not necessarily from local sources, respondents may have associated local subjects with local media perceptions.
Social media use and interest in current events negatively affect ability to discern fake headlines
We found that social media use and interest in current events is negatively associated with headline judgements. Those who use social media more than once a week were 11.68% less likely to accurately judge headlines correctly as true or fake, and those who reported being interested in current events were 22.22% less likely to accurately judge headlines.
Takeaways
These findings highlight two possible mechanisms. First, the geographical effect of headlines may depend on local perceptions of the media. In some countries, people are less trustful of local media sources. This distrust may transfer over to their readings of news on local issues while in countries more trustful of their local news, there may be an opposite effect. Future research can explore whether findings still hold if the source of information is provided. In a world where most interventions to improve discernment between true and misinformation rely on bridging the information asymmetry of consumers, it would be worth exploring which form of media can be used as a powerful tool in making people more discerning between fake and real news.
Second, familiarity with social media and more enthusiasm for current events does not appear to improve fake news discernment. In fact, it has the opposite effect. There are many possible cognitive explanations for this. Perhaps those who engage with news more frequently have to “spread” their cognitive resources thin. Frequent readers of online news may also find more thrill in provocative headlines. Either way, the findings tell us that people may not build digital literacy simply from exposure; to some extent, they need to be trained. This sheds doubt on common explanations for why the elderly — who are not as experienced with the internet — are more susceptible to fake news than younger users. On a personal level it reminds us, yet again, to read and understand that article before we share it — or risk giving more fake restaurants top rating.
A final note:
Our goal is to make research accessible in order to start, continue or inform conversations that help us to better understand human behavior.
As a commitment to Open Science, we keep this anonymized data live for all our on-going research efforts.
This blog post is part of our larger “Off The Record” initiative where we share findings from small-scale research projects, designed to collect initial data and kick-start a conversation.
If you would like to learn more about a specific human behavior or have a research idea you think we could explore for a future “Off The Record”, please reach out to us on via email.
