Please don’t blindly follow PopSci’s lead and get rid of comment spaces

A tiny explosion happened in the online science communication world yesterday. Popular Science.com announced that they will be closing off opportunities to post comments on their news stories: no more public comment spaces. Why? They argue that uncivil commenters have an overly negative effect on readers, so negative that it isn’t worth maintaining the comment spaces. They make some scary claims too about a small number of negative commenters poisoning the way readers perceive the stories and about a war waged on expertise. They use an New York Times Op-Ed written by Dominique Brossard and Dietram A. Scheufele to back up those claims.

I must, however, respectfully disagree.

Of course, the site is theirs. And as is true for any publication or online space: your house, your rules. So, really, they can do whatever they like with their comment section. More worrying to me was the series of tweets and facebook posts I saw from friends and fellow science communicators saying that more publications should do the same.

There are two main reasons why I’d like to suggest caution. 1. The evidence for the poison effect of uncivil comments isn’t nearly as damning as their quotes suggest, and 2. There is a lot of potential good in comment sections and removing them ignores those possibilities and sends some fairly negative messages about science communication.

First, it’s important to take a look at the study that they rely on as their justification:

The quotes used in PopSci’s post are from an Op-Ed piece written by two of the four authors of a study called “The ‘‘Nasty Effect:’’ Online Incivility and Risk Perceptions of Emerging Technologies“, forthcoming from the Journal of Computer-Mediated Communication. The authors had 1183 adults read a blog post about risks and benefits related to nanotechnology. Some read a version that had uncivil comments (including, for example, personal attacks and name calling) and some read a version that had only civil comments in the comments section. They measured several characteristics of the participants in relation to the topic, such as their familiarity with nanotechnology, their confidence in their knowledge, and their prior support for the technologies. They also measured other characteristics such as readers’ usual reading behaviours, their religiousness, their age and their gender. The researchers  used all of these variables (including which version of the comments the participants saw) to figure out which ones would explain how the readers would rate the risks of nanotechnology after reading the blog post and comments.  I have a few minor quibbles with the methods but they are on the level of things I’d like to chat with the authors about over coffee, not things that jeopardize the results. The major problem I have is with the large gap between the way the results are presented in the Op-Ed (and then taken up by PopSci) and how they actually appear in the study.

The first glaring issue is that even all of the variables put together (from age to prior beliefs, up to and including the civility of the comments) seem to have a small effect on the readers. All of these things put together only explain 17% of the differences in readers’ responses. So, 83% of what influenced the way readers responded to the article had nothing to do with any of things the researchers measured, including the civility of the comments. Following directly from that, it would be tough to tell from the Op-Ed that the civility of the comments had NO SIGNIFICANT DIRECT EFFECT on readers’ perceptions of nanotechnology. Here it is straight from the paper: “Our findings did not demonstrate a significant direct relationship between exposure to incivility and risk perceptions. Thus, our first hypothesis was not supported.” (p. 8).

The things that did have an impact weren’t too surprising. Readers who were familiar with nanotechnology and who already supported nanotechnology tended to perceive lower risks than those who weren’t familiar and those who went in not supporting the technologies. These factors explained more of the readers’ perceptions than any others, and they support decades of work that prior beliefs are one of the largest factors in how readers (both adults and students) interpret what they read (for example, this study by Stephen Norris and Linda Phillips)

So where in the world do the dire and scary quotes come from? The last piece of the analysis looked for interaction effects. That’s where even if something doesn’t have an effect on the whole group we can sometimes find that it does have an influence on some people in the group in a unique way. They found two very small interaction effects. First, when they looked just at the group who read the uncivil comments, those who already supported nanotechnology expressed even lower risk than they did in the civil comment group and those who already didn’t support it, expressed even higher risks. So among those who already held strong views, the uncivil comments tended to polarize them a bit further. They found a similar relationship around religiousness, although I think it’s harder to explain this one. The authors seem to have come in with the assumption that religious people would generally perceive higher risk than those who are less religious. In the overall sample this didn’t come through though. Risk assessments were evenly spread among people with all religiousness scores. The only difference was that when the comments were uncivil the religiousness factor then (and only then) acted in the way they expected. So highly religious people reading uncivil comments expressed higher risk and vice versa. Both effects were very small though, increasing or decreasing risk perceptions by 1-2%.*

So, in sum, the uncivil comments seemed to slightly heighten the views that people already had, and when we divide them by religion they tended to react slightly differently to the uncivil comments. But both of these effects together explained a whopping 1% of the differences in readers’ risk assessments. So almost all of the factors that influenced of how the readers reacted had nothing to do with the civility of the comments, nothing at all.  I didn’t find any support in the actual study for claims that  people’s views become “much more polarized” when uncivil comments were read.

Does that seem like solid evidence for publications to decide to do away with commenting all together? I don’t think so.

Second, I want to turn my attention to the potential problems with removing comment sections.

In addition to being on shaky justification grounds, I also see a serious problem with the gut and knee-jerk reaction to remove all comments. I know we’ve all been there, reading a perfectly great article about science and then having our faith in humanity shattered by the comments. I totally understand the impulse to say, “Ya, these guys have it right and maybe science communication would be better if more publications did this.”** And I want to clearly say that I’m generally in favour of strong moderating policies. Even if they don’t really change people’s minds about the risks of nanotechnology, I have no problem at all with the idea that uncivil comments may be undesirable for many other reasons. My issue is with the idea of doing away with the incivility by doing away with the comments all together.

A few years ago I completed a study of comments left in response to health stories in the Canadian newspaper The Globe and Mail. My study was about how commenters claimed expertise not about risk perceptions but I think there’s a piece of it that might be valuable to look at. For the study, I gathered all of the comments posted on four health stories one week after the stories had been published. For the analysis, I was only interested in comments where the commenters related their expertise or inexpertise in some way that was related to their scientific experience or their experience with the health condition in question. This basically meant all on-topic and civil comments. So my faithful undergrad assistant that summer had the joyous task of reading all the comments first and removing the off-topic and uncivil ones. You may wonder what was left. It turned out there was a lot left and a lot of important and valuable comments at that. Extensive contributions were made by parents, patients and people with medical expertise. Questions were asked and clear thoughtful answers were often given. The comments including a long discussion of whether it is possible for scent to trigger severe peanut allergies with clear explanatory answers from commenters with medical expertise, for example:

The chemicals that are responsible for the odor and flavor of peanuts are called pyrazines. Pyrazines are volatile organic compounds and have no protein structure, and as far as I know they can not cause allergic reactions, they only cause odors and flavors. For anaphylactic shock to occur (from nuts) there has to protein which triggers the reaction, this, possibly can be air born.

There were patients challenging wrongful assumptions made in the way the reporter described a condition:

While I agree that a more thorough study needs to be done, I am more disturbed by the severe misunderstanding of what ADHD is. It is a neurological disorder where a person is unable to filter out distractions, usually has a lack of inhibitions, exhibits jumpy thoughts, ha s a hard time concentrating and finishing tasks, and may fidgit. It is not a made up reason for drug companies to sell drugs, nor is it due to a lack of “fatherly” affection. …I both have  ADHD and have taught children with ADHD. It is a truly mixed blessing, the worst part being the ignorance and cruelty of people who assume you are just lazy, had bad parents, stupid, and duped.

Commenters also challenged each other for evidence of the claims they made in their comments:

Dr. T, Can you at least post a link or another source that backs up your story? An anonymous quote saying ‘this is just plain wrong’ really doesn’t persuade anyone.”

To which Dr. T responded: “You’re quite right. The British Medical Journal has been running a series of interesting comments under the heading For and Against: Are the dangers of childhood food allergy exaggerated?
http://www.bmj.com/cgi/content/full/333/75”.

Similarly, Esther Laslo from the Israel Institute of Technology and her colleagues studied the comment sections of Israeli newspaper articles that addressed ethical issues in science. Not only did they find the comment sections to be rich and valuable, they noted that the most fruitful discussions and interactions were initiated by topics raised in the comments, not in the articles themselves.

There are often calls in popular science publications for people outside of traditional scientific communities to become more interested and engaged in science. Comment spaces are a real and viable place for that to happen.

At their best they can be a place for different types of people to actually hear from each other and for people who usually don’t have much of a voice in science conversations to actually have one. How often in everyday interaction does a patient get to challenge a doctor for evidence of his claims? Ed Yong even facilitated a collaboration between a scientist and a farmer through the comment section of his blog.

And like any actual place of conversation, they also fall victim to domination by extreme voices and need to be well managed. Town hall meetings and public consultations are a great example, and they’ve often been a focus of research in public understanding of science (see, for example, James Wilsdon and Rebecca Willis’s book See through Science: Why public engagement needs to move upstream). When they’re good, they’re fascinating and offer real insight that the panel members or politicians could never have fully appreciated without opening the floor to members of the public or a particular community. The can provide access and a voice for people to actively influence science and technology as it affects their lives and communities. At their worst they can be reactionary shout-fests of frustration to all involved. But I don’t see many tweets or posts saying that no one should have them any more. The problem isn’t the idea of a town hall or public consultation but a recognition that they really have to be thoughtful and well planned to be successful.

But back to the idea of engagement. How are people supposed to do that if the very venues for that engagement, which are unprecedentedly afforded by online science communication, are closed? The message then becomes “Well we didn’t really mean for people to be engaged, we just want you to listen to us more.” This is a return largely to outdated models of science communication where the sole purpose is to push scientific information out to people for their ready and unquestioning uptake. If science is truly about discussion of evidence and a willingness to be open to new findings, then the public cannot be left out of that process.

What does it also say about people with expertise in scientific topics? Their justification claims that there is a decades long war against expertise, but this choice also contributes that war. A no-commenting policy is also a no-experts-commenting policy. I often comment on news stories related to science education, sometimes to answer questions that I see in the comments and often to try to counter misunderstandings that I think the media pieces sometimes perpetuate. Instead of a place to engage in conversation and even clarify or correct media stories, the message to people with expertise is “Hey now, leave the communication up to us writers, we have it all under control and don’t need your input.” I have no desire to start a scientists vs. science writers thing here, and I think online spaces have gone a long way to helping everyone involved in those debates see how they can work together. I’m just saying that’s another message that’s embedded in saying no comments allowed. That’s especially troubling when it comes to new technologies, where there are serious and evidence-based disagreements and discussions to be had about risk. I’m pretty uncomfortable giving back complete control to how those risks are presented in a forum where no expert has a space to disagree with what PopSci or another venue says. I really want the opportunity to comment on science education stories because I think I have something to contribute to the conversation that may be missing from the stories as written. There are likely scientists and engineers (and science writers!) who feel the same way. What a no-commenting stance like PopSci’s says to me is that they don’t need or want those contributions associated with their articles.

I totally understand the feelings that people have in relation to comments, but I really don’t think the answer is to get rid of them all together. The incivility, first, doesn’t seem to have nearly the dire effect that PopSci seems to think it does in terms of influencing readers’ perceptions. Comments are often annoying and frustrating (sometimes even heartbreaking) but readers are still making up their minds based on other factors. So the benefits PopSci is hoping for are unlikely to be realized. Second, getting rid of comments to get rid of the incivility seems like a serious baby meet bathwater situation. Instead of looking for better ways to manage, guide, moderate or selectively publish comments we lose all of the potential benefits for real engagement.

Yes, comments can be highly uncivil and polarized. Should we encourage popular science publications to find better ways to foster civil discussions? Absolutely. But give up and stop comment spaces all together? Please, I really hope not.

__________________________________

*The authors don’t discuss effect sizes but from the regression coefficients it looks like being 1 pt more religious on a scale of 1-10 and finding yourself reading the uncivil comments seems to add to the risk you perceive by .07/5. In other words, it increases the risk you perceive by 1.4%, a very small amount. Similarly, being 1 pt more in favour of nanotech (also on a scale of 1-10) and then finding yourself reading the uncivil comments seems to decrease the risk you perceive by .09/5, or 1.8%. That hardly supports what is said in the Op-Ed “Those exposed to rude comments, however, ended up with a much more polarized understanding of the risks connected with the technology.”

Also, I might be doing some of the mental math wrong but the table of results actually says there’s negative relationship between religiousness and risk in the uncivil condition, which would actually mean that more religious people see nanotechnology as less risky when they read uncivil comments. It also says there’s a positive relationship between prior support and risk, so people in the uncivil condition actually perceive slightly higher risk than they would have otherwise. I’ve gone with what they’ve reported in the text in case the sign error is mine and not theirs.

** This is a paraphrase of an actual tweet I saw a friend post, but I’d rather not call out anyone specifically. That’s not the point of this post. And, as I said, I completely understand the impulse.

About these ads
Leave a comment

15 Comments

  1. Steve

     /  September 25, 2013

    your a moron

    Reply
  2. Seems like research that explicitly includes more context would help. Some topics may be too important to risk a poison effect, no matter how small, for example. Or, some topics may be too, shall we say, socially potent, to elicit helpful comments. Or perhaps some articles, at certain outlets, are too inundated by professional astroturfers and public relations folks. It is interesting to me that you have not mentioned climate change: this issue seemed to be at the centre of the Popular Science decision.

    Reply
    • Yes, I agree. I think more examinations of what comments do in particular contexts would be really helpful. There may well be situations where there is strong evidence to support closing comments on certain issues or topics.
      And to be honest, I didn’t mention climate change because the policy they’re talking about seems to be for all (or at least most) news stories and it didn’t figure prominently in the justification that they posted. It may be a key issue in their own reasoning, but I was most interested in the overall policy and the reactions that I saw from friends and colleagues that more science publications should do this.

      Reply
      • Climate science was one of two examples of how a “decades long” public relations battles fought in comments can erode public trust and result in rejecting a consensus: “the cynical work of undermining bedrock scientific doctrine is now being done beneath our own stories.”

        Ha ha, your reflection on your friends and colleagues makes me think of other interesting research: survey scientists’ and other experts on the perceived value of comments, paying special attention to how the expert publishes/reads/participates in populist publications and comments (and what field they’re in). Some may simply be jaded by their personal experience.

        You know, I’m surprised there’s not better research on the impacts of commenting from marketing or communications studies or business communications. Hmm, I suppose that kind of research is proprietary.

    • No topic is too sacred or important for public discussion. In fact, the need for public comment is directly proportional to a topic’s importance. Topics with political, religious or social implications certainly tend to attract cranks from all parts of the spectrum, but judicious moderation is feasible even if it is challenging.

      I suspect we might be underestimating the ability of the average reader to identify and disregard incendiary and inaccurate remarks. Are readers who are attracted to scientific topics hopelessly suggestible? Are they pathologically sensitive to uncivil remarks? I see no reason to think so.

      Reply
      • I hope that you’re right. But the concern looks something like this:
        1. Science reporting is done in the public interest.
        2. *If* some kinds of comments undermine the goals of science reporting, then those comments need to be stopped.
        3. *If* we don’t know which comments are negatively impacting science reporting, then all of the comments need to be stopped.

        I, for one, don’t want to rely on my intuitions for this kind of thing. I want the insights of actual research. Actually, the more I think about it, the more I think the burden of proof may actually be on the pro-comment side. Taking the goal of science reporting seriously, and taking the preliminary results on the risks of comments seriously, I think there is a really interesting precautionary case for just stopping the practice.

  3. Very insightful, well-thought-out post.

    I’m a (new) science writer, and I welcome the comments on my articles when they’re published, not just for the helpful watchdogs but for the community around topics that arises. I mean, we want people to get involved in science, we needn’t take away such a common, key example of that. Sure,the trolls drive us crazy, and they’ll find a way to troll no matter what.

    My doubts, honestly, were raised when PopSci quoted an op ed and not a study. And I’m concerned as to why scientists felt the need to go to an op ed to express claims they didn’t find in their research. That’s not the good ol’ inductive reasoning scientists are supposed to use!

    Reply
  4. I couldn’t agree more. I value the feedback I get in comments on my blog immensely, it’s taught me a lot.

    I for one will be using “Rbutr” (google it) to post replies to articles on Popular Science from now on.

    Reply
  5. tim

     /  September 28, 2013

    Have you ever read the comments on popsci and/or its Facebook page? The misinformation (anti-science/anti-gmo/anti-vax/anti-agw) and banal comments far outweigh constructively critical and informative comments. Just check it out for yourself. It’s tedious and I’m not surprised at their decision.

    Reply
  6. I found it troublesome that PopSci was basing its decision primarily on a single study in a field that is notoriously difficult to study. Replication is a key element of confidence in any set of results.

    Reply
  7. Shutting off comments could function to further disconnect scientists from the public, as we risk becoming “elitists” – the idea that our word is final, so why should anyone in the general public speak up to argue or ask for clarification?

    Reply
  1. Popular Science has turned off comments. Here's why that's a bad idea. | Designolics
  2. Please don’t blindly follow PopSci’s lead and get rid of comment spaces | The Finch and Pea
  3. Turning Off Comments? Effect Size and Why You Should Always Read the Primary Literature | Mike the Mad Biologist
  4. Should You Allow Comments on Your Law Blog?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,735 other followers

%d bloggers like this: