Do violent movies like Probe city leads to aggression? Withdrawn studies have confused the debate.
AF Archive / Alamy Stock Photo
By Cathleen O’Grady
While Samuel West is combing a paper that finds a connection between watching cartoon violence and aggression in children, he notices something strange to the study participants. There were more than 3,000 – extraordinarily large – and they were all ten years old. “It was just too perfect,” says West, a doctor. student in social psychology at Virginia Commonwealth University.
Yet West published the 2019 study, published in Aggressive behavior and led by psychologist Qian Zhang of Southwestern University of Chongqing, to his meta-analysis after a reviewer asked him to cast a wider net. West does not feel that his vague reservations can justify shutting him out of the study. But after that Aggressive behavior West’s meta-analysis published last year, he was apprehensive that the magazine was investigating Zhang’s newspaper while it was its own review.
This is just one of Zhang’s many recent articles that cast doubt on research into the controversial question of whether violent entertainment promotes violent behavior. Zhang denies any wrongdoing, but two papers were withdrawn. Others live on in journals and meta-analyzes – a ‘big problem’ for a field with conflicting results and entrenched camps, says Amy Orben, a cognitive scientist at the University of Cambridge who studies media and behavior. And not just for the ivory tower, she says: the research shapes media warning labels and decisions by parents and health professionals.
The investigation was prompted by Illinois State University psychologist Joe Hilgard, who last month published a blog post cataloging his concerns about Zhang’s work. Hilgard was initially impressed when he came across an article by Zhang in 2018 Youth and society, another study with 3000 subjects. “I was like, holy smoke!” he says. The study found that teens were more aggressive after playing violent video games. Given the enormous sample size, it had the potential to be a ‘powerful piece of evidence’, says Hilgard.
But he found the statistics of the paper mathematically impossible. Zhang and his co-authors reported high levels of statistical significance for their finding, but the reported differences in the effects of violent games versus non-violent games were too small for the high statistical significance. Hilgard warned Zhang and the journal, and Zhang filed a correction. Hilgard says it made the statistics look more plausible, but it was still wrong.
Hilgard says he found problems in other Zhang newspapers, such as nearly identical results reported in three different newspapers. He emailed Zhang and asked to see his data, but he says Zhang refused. Hilgard then contacted Dorothy Espelage, a psychologist at the University of North Carolina, Chapel Hill, and co-author with Zhang on several papers. She told Hilgard that Zhang also refused to send the data to her. Only after Hilgard asked Southwest University to investigate does Zhang send Hilgard data for a Youth and society paper on film violence.
But the data were strange, Hilgard says, and missing features commonly found in similar experiments. He sent his findings to Zhang University, which said it had problems, but not fraud, with Zhang’s data, and that Zhang lacked statistical knowledge and research methods. ‘Dissatisfied with the answer, Hilgard sends his comments to all the magazines involved.
Marc Zimmerman, a psychologist at the University of Michigan, Ann Arbor, and the editor of Youth and society, says Hilgard’s email came a week after another source questioned Zhang’s data. Two months later, in December 2019, Zhang magazine withdrew its two papers. Without both whistleblower reports, Zimmerman says he might not have acted so quickly or decisively. “No magazine editor wants to publish wrong data,” he says, “but it is a serious accusation.”
In an email to Science, Zhang denies any misconduct, writing: “there is never just one ‘right’ way to look at the data.” He wrote that Hilgard, who is skeptical about the link between violent games and aggression, “is trying to establish his name just because he claims that everyone else is doing bad research.” But Jay Hull, a psychologist at Dartmouth College whose work shows a link between violent video games and aggression, agrees with Hilgard’s concerns about Zhang’s work. More generally, Hull is concerned with a ‘science built on shifting sand’.
Other magazines, including Aggressive behavior, did not withdraw any of Zhang’s research, although Espelage withdrew a withdrawal from the Aggressive behavior paper on which his co-author was. Craig Anderson, a psychologist at Iowa State University and editor-in-chief at Aggressive behavior, declined to comment on the record. The newspaper’s results will live on within West’s meta-analysis, which does not make him sure how to correct his own work. It’s ‘very frustrating’, says West.
Zhang’s recent newspapers avoided the red flags that Hilgard complained about. But Hilgard is not satisfied. The delays and lack of action, he says, make him “very concerned about how difficult it will be to detect data that is flawed in more subtle ways.”