Wednesday, January 2, 2013

Science, well sort of...me forcing me to answer my own question


In one of my recent posts I had some fun with the topic “…the six things I want our kids to know about science … of course there are more, but it was a good start.  Among the comments I received on the post, one from a friend posed a troubling question, which could be restated as, “OK wise guy, how DO we help our kids figure out what to believe with all this seemingly conflicting and confusing sciency stuff?”  In re-reading my rants on this subject I realize that while I have done a passable job of laying out the challenge of figuring out how to decide what to believe, I’ve done a crap job in providing any answers to the question how.  This led to some rapid self- reflection, asking myself the question, “OK wise guy, how do YOU figure out what to believe with all this seemingly conflicting and confusing sciency stuff?”  …fie on self-reflection. But the effort did allow me to see that I do have a bit of method to my madness, outlined here.

My first step in assessing the validity of a claim is pretty much always the same… I take a really, really close look at the source. I feel this is the best place to start, and while taking a bit of time and effort it can often yield immediate results. If the source does not pass the smell test, then move on.  This exercise breaks down into two broad categories, qualification and motivation.

Part one: Assess if the source is qualified to make the claim

For me, this one is pretty straight forward... Do they have the expertise, track record and just plain legitimacy to make claims? If I don’t immediately recognize the source of a claim, such as a major university, research institute, etc, a quick internet check can be very revealing.  A couple of my favorite examples here are when I Google mapped the location of a very sciency sounding institute that was putting out climate change denial lies, and it turns out they are housed in a small out-building on a dirt road in rural Oregon; or when a legitimate sounding, oft quoted source on the danger of GMOs turned out to be a group of osteopaths in Kansas who did no research at all, but just passed along bogus information.  I’ve done these searches many times and it’s really amazing how much you can learn about the sources of the nonsense that gets put out, and then picked up, by the media, bloggers, interest groups or your Facebook friends.  You don’t have to be a scientist to figure this out…just consider the source and use your judgment.

BONUS Tactic - If you still have lingering doubts about a source, look into how they get their funding, who is on their Board of Directors and, if they are a nonprofit, have a quick look at their tax returns (not hard, go here http://www.guidestar.org) … lots of insight to be gained doing this.

Part Two: Assess the motivation of the source making the claim
This one seems straight forward, but you have to be careful. When examining the motivation of a source, it is easy to fall into the trap of, “Well, of course they would say that, it is in their self-interest”.  Self-interest can take many forms, such as a financial stake in the outcome, academic rewards, pressure to conform to an ideological perspective and just plain status/ego/hubris.  The tricky problem here though is that many perfectly valid sources have self-interest in the claims they make, or at least can be accused of having so.  Therefore, self-interest should never be an immediate disqualification of the validity of a claim.  But it does play a role, and needs to be viewed in the context of the other factors.  In the recent effort to label GMOs in California, the claims made by the large food industry were attacked in part for their self-interested motivation to maintain the status quo…fair enough.  But you had to also understand that many pro labeling folks also had significant self-interest at stake, in this case the growth of non GMO food markets. So yes, once again you’ll have to exercise some judgment, and weigh all the factors.  My next stop is the data itself.

This is probably the hardest part for us nonscientists, because if you ever try to look at the actual data underlying any claim, unless you are an expert in the field, you quickly realize that you really don’t fully understand what is being presented…in fact, that’s why we rely on trusted sources to interpret and explain the meaning of the data. So my approach is to assess the preponderance and longevity of the data from qualified sources supporting the claim.  Examples…the data to support the claims of human caused climate change is overwhelming and longstanding, while that to support the risks of GMOs is essentially nonexistent.  I could go on, but the point is that we need to weigh a number of factors in assessing claims.  Sadly, there is no easy or simple approach, but it’s certainly worth the effort.

Yikes, I’ve reached my self-imposed word limit, having no doubt lost most readers already (…sorry gang).   I’ll close for now, but to keep you on the edge of your seats, next time I’ll weave in systemic causation, epistemic closure and the limits of science on how they factor in to our beliefs…consider this part one of two…nothing but wacky fun on this blog… [Sic].

So, what do you believe?

10 comments:

  1. How about asking someone you know who is more qualified to understand the data what she/he thinks? This is second order, but to the extent you trust the person you ask, it can be quite solid.

    Also, watch any claim that is based on statistical analysis. The misuse (deliberate and not) is quite rampant especially in medical claims. On that topic, you can ask how many people were involved in a medical test or similar activity. Small sample sizes are always a red flag that the data don't support the claim very well.

    ReplyDelete
  2. Thanks David..totally agree with both points...would only caution that the expert friends need to be vetted just like any other expert...and remember, just because someone may be a scientist, does not mean they have any expertise outside their own field...as to statistics...don't get me started!

    ReplyDelete
  3. Love following these blogs...looking forward to next installment.

    ReplyDelete
  4. Thanks for following through on your promise, Alex. I agree with your suggested methods, though they sound quite laborious for you average "controversial" issue. Most of the time, we need heuristics. I would add to your list that research published in peer-reviewed journals can usually be considered vetted by experts, and so it's helpful to be familiar with some of the more well-respected scientific journals in a given field. When something is published in Science or Nature, I don't usually feel compelled to Google the address of the researchers. However, I'll admit that I don't EVER read Science or Nature, but usually come across articles about what's been published there in the NYT--which was the source of the information that led to my eventual epiphany about climate change. I happen to trust the NYT, but know that others might question the "motivation of the source." Is reading about research in a reputable newspaper (though one with an acknowledged political bias), an acceptable heuristic? Are there less biased secondary sources or "filters" for your average person like me who a) want to stay informed of scientific research, and b) want to be able to do some quick research when something like "GMOs" or "Smartmeters" or "Moth Spray" or "vaccinations" becomes the cause du jour in our community and everyone is up in arms about it? Thanks again. --Chloe

    ReplyDelete
    Replies
    1. Thanks Chloe...I don't really think there is such a thing as a blanket acceptable heuristic, as even the very best sources can get it very wrong ( See my friend Jon Kooney's blog on the topic here http://www.koomey.com/post/39543658454?0a9efbd8). The point I'm making is that any one data point or source is insufficient to draw a conclusion...that's why I lean towards the preponderance of evidence...all sources have their biases, shortcomings and ability to be just pain wrong...but by digging just a little you can get a clearer picture...bottom line, there really may be no short cut to this stuff.

      Delete
  5. Very insightful, Mr. Zwissler. I too encounter uninformed "deniers" when I conduct classroom lectures and/or book-tour talks on overpopulation. Usually such blow-back comes from theologians (whose self-interest lies in boosting their numbers and, hence, power) or their flock, who believe God's bounty is limitless... in spite of the scientific data proving otherwise. Another uniformed faction -- who might more accurately be termed "population skeptics" rather than "population deniers" -- are those who mistakenly believe there is a racist agenda behind anyone trying to raise awareness to the fact that human overpopulation is the single greatest threat to our environment and, ergo, to humanity itself. And there's even a third faction -- the "last piece of the pie-ers" -- who believe curtailing population growth means curtailing economic growth. Even if this were true, we risk something akin to a Pyrrhic victory in pursuing the "It's the economy, stupid" mindset in light of the fact that, ultimately, "It's the environment, stupid!" It is frustrating to say the least. Thanks, and keep up the good work!

    certainly,

    Robert P. Johnson
    author of "the momentum of folly" and "Thirteen Moons: A Year in the Wilderness"

    ReplyDelete
  6. Thank you for your comment Robert...the funny thing about deniers of science of all stripes is that they do not feel themselves to be uninformed...in fact, many are very informed, but have simply drawn the wrong conclusions, or used dubious data to support their positions...and so it goes...hey, don't you still owe me some money?

    ReplyDelete
  7. Alex,

    Great topic! One observation: it seems that if we are asking the question, "What should I believe?" we are already at the tail end of the dogma bell curve. If we are willing to go to the trouble of investigating resource qualification as part of this objective question, we are truly outside the mainstream, exhibiting the characteristics of scientists. If we are investigating resource qualification simply to confirm or deny our existing beliefs we are likely to fall prey to typical anchor bias that is rooted deeply in our personal histories.

    I drive a carpool of neighborhood high-schoolers every morning and get to listen in on vigorous debates about politics, climate change, economic policy, and more from the perspective of very bright 16-year olds. And they back up their positions with volumes of news reports, book readings, internet research, and Fox/MSNBC interviews. The debates grow more and more heated as the kids fling their "research" at each other without being willing to consider their "opponent's" data. As an objective observer who doesn't participate in the debates but sits back and chuckles a lot, it's interesting to note that each teen inevitably takes the position of their parent (The ultimate qualified authority figure). Children of Libertarian parents are spouting Libertarian spun data. Children of Liberals are spouting Liberal spun research. Parental-inspired anchor bias seems to root us in the muck and mire of intransigent beliefs from an early age making it very difficult to even consider search objectively for data. The next generation of kids will be far more facile at internet research, but, it seems that there is a big gap in their education around objectivity and awareness of their own cognitive biases.

    Might be a good idea to create a Facebook simulation to help kids learn objectivity and become more self-aware. We could call it, "Think Like Alex!" Scary!

    ReplyDelete
    Replies
    1. ...Scary indeed! But great points about anchor biases, and you're so right that few of us will ever take the time to do as I suggest...and of course my process does not necessarily mitigate for any biases of my own...it always seems to come back to this is simply how we are wired...where I remain intrigued is around the idea of giving kids some early exposure and self awareness to how our beliefs are actually formed...do a test with the girls in the car...Ask them how they form their beliefs...I bet they will all insist they are being completely rational, scientific and unbiased in forming their opinions. Thanks buddy!

      Delete
  8. I read your post. Really it's a interesting post by you.Thanks for all the reviews you write about science. This design looks really beautiful.

    Arginase Assay Kit

    ReplyDelete