M-c Shanahan

A tiny explosion happened in the online science communication world yesterday. Popular Science.com announced that they will be closing off opportunities to post comments on their news stories: no more public comment spaces. Why? They argue that uncivil commenters have an overly negative effect on readers, so negative that it isn’t worth maintaining the comment spaces. They make some scary claims too about a small number of negative commenters poisoning the way readers perceive the stories and about a war waged on expertise. They use an New York Times Op-Ed written by Dominique Brossard and Dietram A. Scheufele to back up those claims.

I must, however, respectfully disagree.

Of course, the site is theirs. And as is true for any publication or online space: your house, your rules. So, really, they can do whatever they like with their comment section. More worrying to me was the series of tweets and facebook posts I saw from friends and fellow science communicators saying that more publications should do the same.

There are two main reasons why I’d like to suggest caution. 1. The evidence for the poison effect of uncivil comments isn’t nearly as damning as their quotes suggest, and 2. There is a lot of potential good in comment sections and removing them ignores those possibilities and sends some fairly negative messages about science communication.

First, it’s important to take a look at the study that they rely on as their justification:

The quotes used in PopSci’s post are from an Op-Ed piece written by two of the four authors of a study called “The ‘‘Nasty Effect:’’ Online Incivility and Risk Perceptions of Emerging Technologies“, forthcoming from the Journal of Computer-Mediated Communication. The authors had 1183 adults read a blog post about risks and benefits related to nanotechnology. Some read a version that had uncivil comments (including, for example, personal attacks and name calling) and some read a version that had only civil comments in the comments section. They measured several characteristics of the participants in relation to the topic, such as their familiarity with nanotechnology, their confidence in their knowledge, and their prior support for the technologies. They also measured other characteristics such as readers’ usual reading behaviours, their religiousness, their age and their gender. The researchers  used all of these variables (including which version of the comments the participants saw) to figure out which ones would explain how the readers would rate the risks of nanotechnology after reading the blog post and comments.  I have a few minor quibbles with the methods but they are on the level of things I’d like to chat with the authors about over coffee, not things that jeopardize the results. The major problem I have is with the large gap between the way the results are presented in the Op-Ed (and then taken up by PopSci) and how they actually appear in the study.

The first glaring issue is that even all of the variables put together (from age to prior beliefs, up to and including the civility of the comments) seem to have a small effect on the readers. All of these things put together only explain 17% of the differences in readers’ responses. So, 83% of what influenced the way readers responded to the article had nothing to do with any of things the researchers measured, including the civility of the comments. Following directly from that, it would be tough to tell from the Op-Ed that the civility of the comments had NO SIGNIFICANT DIRECT EFFECT on readers’ perceptions of nanotechnology. Here it is straight from the paper: “Our findings did not demonstrate a significant direct relationship between exposure to incivility and risk perceptions. Thus, our first hypothesis was not supported.” (p. 8).

The things that did have an impact weren’t too surprising. Readers who were familiar with nanotechnology and who already supported nanotechnology tended to perceive lower risks than those who weren’t familiar and those who went in not supporting the technologies. These factors explained more of the readers’ perceptions than any others, and they support decades of work that prior beliefs are one of the largest factors in how readers (both adults and students) interpret what they read (for example, this study by Stephen Norris and Linda Phillips)

So where in the world do the dire and scary quotes come from? The last piece of the analysis looked for interaction effects. That’s where even if something doesn’t have an effect on the whole group we can sometimes find that it does have an influence on some people in the group in a unique way. They found two very small interaction effects. First, when they looked just at the group who read the uncivil comments, those who already supported nanotechnology expressed even lower risk than they did in the civil comment group and those who already didn’t support it, expressed even higher risks. So among those who already held strong views, the uncivil comments tended to polarize them a bit further. They found a similar relationship around religiousness, although I think it’s harder to explain this one. The authors seem to have come in with the assumption that religious people would generally perceive higher risk than those who are less religious. In the overall sample this didn’t come through though. Risk assessments were evenly spread among people with all religiousness scores. The only difference was that when the comments were uncivil the religiousness factor then (and only then) acted in the way they expected. So highly religious people reading uncivil comments expressed higher risk and vice versa. Both effects were very small though, increasing or decreasing risk perceptions by 1-2%.*

So, in sum, the uncivil comments seemed to slightly heighten the views that people already had, and when we divide them by religion they tended to react slightly differently to the uncivil comments. But both of these effects together explained a whopping 1% of the differences in readers’ risk assessments. So almost all of the factors that influenced of how the readers reacted had nothing to do with the civility of the comments, nothing at all.  I didn’t find any support in the actual study for claims that  people’s views become “much more polarized” when uncivil comments were read.

Does that seem like solid evidence for publications to decide to do away with commenting all together? I don’t think so.

Second, I want to turn my attention to the potential problems with removing comment sections.

In addition to being on shaky justification grounds, I also see a serious problem with the gut and knee-jerk reaction to remove all comments. I know we’ve all been there, reading a perfectly great article about science and then having our faith in humanity shattered by the comments. I totally understand the impulse to say, “Ya, these guys have it right and maybe science communication would be better if more publications did this.”** And I want to clearly say that I’m generally in favour of strong moderating policies. Even if they don’t really change people’s minds about the risks of nanotechnology, I have no problem at all with the idea that uncivil comments may be undesirable for many other reasons. My issue is with the idea of doing away with the incivility by doing away with the comments all together.

A few years ago I completed a study of comments left in response to health stories in the Canadian newspaper The Globe and Mail. My study was about how commenters claimed expertise not about risk perceptions but I think there’s a piece of it that might be valuable to look at. For the study, I gathered all of the comments posted on four health stories one week after the stories had been published. For the analysis, I was only interested in comments where the commenters related their expertise or inexpertise in some way that was related to their scientific experience or their experience with the health condition in question. This basically meant all on-topic and civil comments. So my faithful undergrad assistant that summer had the joyous task of reading all the comments first and removing the off-topic and uncivil ones. You may wonder what was left. It turned out there was a lot left and a lot of important and valuable comments at that. Extensive contributions were made by parents, patients and people with medical expertise. Questions were asked and clear thoughtful answers were often given. The comments including a long discussion of whether it is possible for scent to trigger severe peanut allergies with clear explanatory answers from commenters with medical expertise, for example:

The chemicals that are responsible for the odor and flavor of peanuts are called pyrazines. Pyrazines are volatile organic compounds and have no protein structure, and as far as I know they can not cause allergic reactions, they only cause odors and flavors. For anaphylactic shock to occur (from nuts) there has to protein which triggers the reaction, this, possibly can be air born.

There were patients challenging wrongful assumptions made in the way the reporter described a condition:

While I agree that a more thorough study needs to be done, I am more disturbed by the severe misunderstanding of what ADHD is. It is a neurological disorder where a person is unable to filter out distractions, usually has a lack of inhibitions, exhibits jumpy thoughts, ha s a hard time concentrating and finishing tasks, and may fidgit. It is not a made up reason for drug companies to sell drugs, nor is it due to a lack of “fatherly” affection. …I both have  ADHD and have taught children with ADHD. It is a truly mixed blessing, the worst part being the ignorance and cruelty of people who assume you are just lazy, had bad parents, stupid, and duped.

Commenters also challenged each other for evidence of the claims they made in their comments:

Dr. T, Can you at least post a link or another source that backs up your story? An anonymous quote saying ‘this is just plain wrong’ really doesn’t persuade anyone.”

To which Dr. T responded: “You’re quite right. The British Medical Journal has been running a series of interesting comments under the heading For and Against: Are the dangers of childhood food allergy exaggerated?
http://www.bmj.com/cgi/content/full/333/75”.

Similarly, Esther Laslo from the Israel Institute of Technology and her colleagues studied the comment sections of Israeli newspaper articles that addressed ethical issues in science. Not only did they find the comment sections to be rich and valuable, they noted that the most fruitful discussions and interactions were initiated by topics raised in the comments, not in the articles themselves.

There are often calls in popular science publications for people outside of traditional scientific communities to become more interested and engaged in science. Comment spaces are a real and viable place for that to happen.

At their best they can be a place for different types of people to actually hear from each other and for people who usually don’t have much of a voice in science conversations to actually have one. How often in everyday interaction does a patient get to challenge a doctor for evidence of his claims? Ed Yong even facilitated a collaboration between a scientist and a farmer through the comment section of his blog.

And like any actual place of conversation, they also fall victim to domination by extreme voices and need to be well managed. Town hall meetings and public consultations are a great example, and they’ve often been a focus of research in public understanding of science (see, for example, James Wilsdon and Rebecca Willis’s book See through Science: Why public engagement needs to move upstream). When they’re good, they’re fascinating and offer real insight that the panel members or politicians could never have fully appreciated without opening the floor to members of the public or a particular community. The can provide access and a voice for people to actively influence science and technology as it affects their lives and communities. At their worst they can be reactionary shout-fests of frustration to all involved. But I don’t see many tweets or posts saying that no one should have them any more. The problem isn’t the idea of a town hall or public consultation but a recognition that they really have to be thoughtful and well planned to be successful.

But back to the idea of engagement. How are people supposed to do that if the very venues for that engagement, which are unprecedentedly afforded by online science communication, are closed? The message then becomes “Well we didn’t really mean for people to be engaged, we just want you to listen to us more.” This is a return largely to outdated models of science communication where the sole purpose is to push scientific information out to people for their ready and unquestioning uptake. If science is truly about discussion of evidence and a willingness to be open to new findings, then the public cannot be left out of that process.

What does it also say about people with expertise in scientific topics? Their justification claims that there is a decades long war against expertise, but this choice also contributes that war. A no-commenting policy is also a no-experts-commenting policy. I often comment on news stories related to science education, sometimes to answer questions that I see in the comments and often to try to counter misunderstandings that I think the media pieces sometimes perpetuate. Instead of a place to engage in conversation and even clarify or correct media stories, the message to people with expertise is “Hey now, leave the communication up to us writers, we have it all under control and don’t need your input.” I have no desire to start a scientists vs. science writers thing here, and I think online spaces have gone a long way to helping everyone involved in those debates see how they can work together. I’m just saying that’s another message that’s embedded in saying no comments allowed. That’s especially troubling when it comes to new technologies, where there are serious and evidence-based disagreements and discussions to be had about risk. I’m pretty uncomfortable giving back complete control to how those risks are presented in a forum where no expert has a space to disagree with what PopSci or another venue says. I really want the opportunity to comment on science education stories because I think I have something to contribute to the conversation that may be missing from the stories as written. There are likely scientists and engineers (and science writers!) who feel the same way. What a no-commenting stance like PopSci’s says to me is that they don’t need or want those contributions associated with their articles.

I totally understand the feelings that people have in relation to comments, but I really don’t think the answer is to get rid of them all together. The incivility, first, doesn’t seem to have nearly the dire effect that PopSci seems to think it does in terms of influencing readers’ perceptions. Comments are often annoying and frustrating (sometimes even heartbreaking) but readers are still making up their minds based on other factors. So the benefits PopSci is hoping for are unlikely to be realized. Second, getting rid of comments to get rid of the incivility seems like a serious baby meet bathwater situation. Instead of looking for better ways to manage, guide, moderate or selectively publish comments we lose all of the potential benefits for real engagement.

Yes, comments can be highly uncivil and polarized. Should we encourage popular science publications to find better ways to foster civil discussions? Absolutely. But give up and stop comment spaces all together? Please, I really hope not.

__________________________________

*The authors don’t discuss effect sizes but from the regression coefficients it looks like being 1 pt more religious on a scale of 1-10 and finding yourself reading the uncivil comments seems to add to the risk you perceive by .07/5. In other words, it increases the risk you perceive by 1.4%, a very small amount. Similarly, being 1 pt more in favour of nanotech (also on a scale of 1-10) and then finding yourself reading the uncivil comments seems to decrease the risk you perceive by .09/5, or 1.8%. That hardly supports what is said in the Op-Ed “Those exposed to rude comments, however, ended up with a much more polarized understanding of the risks connected with the technology.”

Also, I might be doing some of the mental math wrong but the table of results actually says there’s negative relationship between religiousness and risk in the uncivil condition, which would actually mean that more religious people see nanotechnology as less risky when they read uncivil comments. It also says there’s a positive relationship between prior support and risk, so people in the uncivil condition actually perceive slightly higher risk than they would have otherwise. I’ve gone with what they’ve reported in the text in case the sign error is mine and not theirs.

** This is a paraphrase of an actual tweet I saw a friend post, but I’d rather not call out anyone specifically. That’s not the point of this post. And, as I said, I completely understand the impulse.